JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 10.11.2020, 18:27
nathan1 nathan1 is offline
Black Hole
 
Join Date: Apr 2012
Posts: 274
Default Problems to decrypt links for metalarea.org

I have add basic authentication in JD2 for metalarea.org

I paste these links on JD2 but it can't crawl nothing. But inside these pages I have several hosts

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Maybe you can't see links inside metalarea pages because you need credentials to see hidden content (host links are also hidden in spoilers)
Reply With Quote
  #2  
Old 10.11.2020, 18:30
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,809
Default

Hi,

I don't think that basic-authentication will work for this forum style website.
You will probably have to extract the cookies of that website and add a link crawler rule.
Without your login credentials I won't be able to help you with that.
If you need an example, see your older thread HERE.

-psp-
EDIT

I'll be offline soon and back again tomorrow.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 10.11.2020 at 18:40.
Reply With Quote
  #3  
Old 10.11.2020, 18:55
nathan1 nathan1 is offline
Black Hole
 
Join Date: Apr 2012
Posts: 274
Default

I sent you metalarea credentials.

Just another thing if I can.
When I copy information from JD2 I see like that

Link;2010 - Crusted.rar;**External links are only visible to Support Staff****External links are only visible to Support Staff**
Link;1996 - Incomplete Minds.7z;**External links are only visible to Support Staff****External links are only visible to Support Staff**

Is possible to see also (not only) the original link of forum from which I copied the link? I mean in this way:

Link;2010 - Crusted.rar;**External links are only visible to Support Staff****External links are only visible to Support Staff**
Link;1996 - Incomplete Minds.7z;**External links are only visible to Support Staff****External links are only visible to Support Staff**

Because I need to have also forum link in informations from which that link was taken.
Reply With Quote
  #4  
Old 10.11.2020, 19:00
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

You can customize the CopyToClipboard Action via rightclick->context menu-> menu editor and there you can modify what is copied to clipboard
additional tags are
Quote:
{url.container}
{url.origin}
{url.content}
{url.referrer}
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 10.11.2020, 19:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

Also check rightclick->context menu->properties->show url and double click into the url to see all known urls
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 10.11.2020, 19:02
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,809
Default

The procedure is basically the same as described in your older forum thread but this time you need the "masession_id" cookie:
Code:
[ {
  "enabled" : true,
  "cookies" : [ ["masession_id", "CENSORED"] ],
  "updateCookies" : true,
  "logging" : false,
  "maxDecryptDepth" : 1,
  "name" : "metalarea.org example rule with cookie-login",
  "pattern" : "https?://metalarea\\.org/forum/index\\.php\\?showtopic=\\d+",
  "rule" : "DEEPDECRYPT",
  "packageNamePattern" : null,
  "passwordPattern" : null,
  "formPattern" : null,
  "deepPattern" : "Download from <a href=\"(https?://[^\"]+)\"",
  "rewriteReplaceWith" : null
} ]
Rule as plaintext for easier copy & paste:
pastebin.com/JNC85fCH
EDIT

Please keep in mind that while we always try to help, the creation of custom LinkCrawler Rules is something that you should learn!
We won't provide example rules for another 100 websites for you.
In order to learn how to do this, you need to learn how to use regular expressions first - you can use webtools such as this to practice: regex101.com


Regarding your 2nd question:
Go to Settings -> User Interface -> Downloadlink address display
Move "Data" to the top and/or deselect all others.

This will only work if you add "uncrypted" downloadlinks.
If you e.g. all .DLC containers, JD won't ever display the direct-URLs to you.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 10.11.2020 at 19:05.
Reply With Quote
  #7  
Old 11.11.2020, 02:32
nathan1 nathan1 is offline
Black Hole
 
Join Date: Apr 2012
Posts: 274
Default

@Jiaz
@pspzockerscene

Thanks a lot!
Reply With Quote
  #8  
Old 11.11.2020, 11:22
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

@nathan1: also see my comment here https://board.jdownloader.org/showthread.php?t=85914
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 11.11.2020, 15:08
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,809
Default

Thanks for your feedback.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #10  
Old 11.11.2020, 23:50
nathan1 nathan1 is offline
Black Hole
 
Join Date: Apr 2012
Posts: 274
Default

@Jiaz
@pspzockerscene

I add in this way

{type};{name};{url};{url.container};{url.origin};{packagename}

I try to test some links but after update it has some problems to recognize URLs and set <title> or don't copy <title> with CopyInformation Action

For example in this URL has mediafire link
**External links are only visible to Support Staff****External links are only visible to Support Staff**

but it don't crawl

also for this links have difficult
**External links are only visible to Support Staff****External links are only visible to Support Staff**

LOG
Code:
11.11.20 22.33.02 <--> 11.11.20 22.28.29 jdlog://6033425302851/
Reply With Quote
  #11  
Old 12.11.2020, 00:58
nathan1 nathan1 is offline
Black Hole
 
Join Date: Apr 2012
Posts: 274
Default

Just another example

For example, this link is not crawled
**External links are only visible to Support Staff****External links are only visible to Support Staff**

The <title> of this URL is Carach Angren - Franckensteina Strataemontanus (2020), Symphonic Black Metal

But JD2 after update don't works

1. don't crawl links inside it
2. don't generates <title> of **External links are only visible to Support Staff****External links are only visible to Support Staff** for packet where host links are crawled or however don't copy <title> from CopyInformation (also if I set up {type};{name};{url};{url.container};{url.origin};{packagename})
Reply With Quote
  #12  
Old 12.11.2020, 01:09
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,809
Default

All working fine here!
1. Working fine.
If it doesn't for you, logout in browser and login again --> Grab the new value of the cookie and put that in your rule.
Cookies can expire - yours might have expired.
Mine also expired in my test-rule and I had to renew the cookie to make it work again.
2. If you want the rule to set a package title, you'd have to define that in the rule ("packageNamePattern").
Again I'm uring you to learn how to use regular expressions but I've modified the rule once again for you to grab- and set the title:
Code:
[ {
  "enabled" : true,
  "cookies" : [ [ "masession_id", "CENSORED" ] ],
  "updateCookies" : true,
  "logging" : false,
  "maxDecryptDepth" : 1,
  "id" : 1605027636498,
  "name" : "metalarea.org example rule with cookie-login",
  "pattern" : "https?://metalarea\\.org/forum/index\\.php\\?showtopic=\\d+",
  "rule" : "DEEPDECRYPT",
  "packageNamePattern" : "<title>(.*?)</title>",
  "passwordPattern" : null,
  "formPattern" : null,
  "deepPattern" : "Download from <a href=\"(https?://[^\"]+)\"",
  "rewriteReplaceWith" : null
} ]
Rule on pastebin:
pastebin.com/Cfv2Zgv0

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #13  
Old 12.11.2020, 02:12
nathan1 nathan1 is offline
Black Hole
 
Join Date: Apr 2012
Posts: 274
Default

Ok, I refresh browser and add new cookie and now works better but not perfectly.
I give you an examples. I copy these links (I use copy selected links extension of firefox to copy URLs)

**External links are only visible to Support Staff****External links are only visible to Support Staff**
javascript:multi_page_jump('**External links are only visible to Support Staff**, 38, 30 );
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

JD2 crawls these links


I copy information and I see this

Link;Demoniac 1993 Satanas 666 (Rehearsal 11-1993).rar;**External links are only visible to Support Staff****External links are only visible to Support Staff** 1993 Satanas 666 (Rehearsal 11-1993);
Link;Moonblood-The Winter Falls Over The Land [Remastered](CD 2015).rar;**External links are only visible to Support Staff****External links are only visible to Support Staff** Winter Falls Over The Land [Remastered](CD 2015);
Link;st.rar;**External links are only visible to Support Staff****External links are only visible to Support Staff**

What is problem?
For st.rar "packageNamePattern" : "<title>(.*?)</title>" doesn't works.
Information that it returns to me are

Link;st.rar;**External links are only visible to Support Staff****External links are only visible to Support Staff**

At the end of line I see ;st; and not ;Moonblood-The Winter Falls Over The Land [Remastered](CD 2015);

And I don't understand why then JD2 requires also Login for metalarea if I add correctly cookie login. When I copy several links JD2 ask me login.



I give you LOG
Code:
11.11.20 23.48.47 <--> 12.11.20 00.11.04 jdlog://5333425302851/
Reply With Quote
  #14  
Old 12.11.2020, 02:36
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,809
Default

Oh lol we actually had a crawler for this website from 2016.
I'll check that with Jiaz tomorrow.
The old crawler only listens for "http" URLs which is why you triggered it.

Please wait for us to re-check this tomorrow/later ...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #15  
Old 12.11.2020, 17:11
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,809
Default

I've fixed our old metalarea plugin from 2016.
We're usually not making plugins for such simple websites anymore but you're lucky that this one still exists and didn't require a lot of work to fix.
You do not need the above linkcrawler rule anymore after the next update.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #16  
Old 13.11.2020, 00:39
nathan1 nathan1 is offline
Black Hole
 
Join Date: Apr 2012
Posts: 274
Default

Quote:
"packageNamePattern" : "<title>(.*?)</title>",
But is still possible to use "packageNamePattern" : "<title>(.*?)</title>", rule? Or just add {packagename} for CopyToClipboard Action?

Last edited by nathan1; 13.11.2020 at 00:41.
Reply With Quote
  #17  
Old 13.11.2020, 01:19
nathan1 nathan1 is offline
Black Hole
 
Join Date: Apr 2012
Posts: 274
Default

In your package naming I see this

Quote:
Amenra - Mass VI (2017) - Metal Area - Extreme Music Portal
Amenra - Prayers 9 10 [ep] (2004) - Metal Area - Extreme Music Portal
Amenra - Mass IIII (2008) - Metal Area - Extreme Music Portal


I see that for every package is added Metal Area - Extreme Music Portal. Is possible to retrieve the real name of packet? Real names have its music genre tag in the <title> name. Look why, please

Quote:
Amenra - Mass VI (2017) - Doom/Sludge/Hardcore/Drone (D)
Amenra - Prayers 9 10 [ep] (2004) - Doom/Sludge/Hardcore/Drone
Amenra - Mass Iii (2006) - Doom/Sludge/Hardcore
I ask this because when you crawl a metalarea link in JD2, for example
**External links are only visible to Support Staff****External links are only visible to Support Staff**

you can see the real <title> is this

Amenra - Mass Iii (2006), Doom/Sludge/Hardcore



and not Amenra - Mass Iii (2006) - Metal Area - Extreme Music Portal
This because relative URLs are inside that page from where JD2 crawls links.

I would like his to refer to that title

Last edited by nathan1; 13.11.2020 at 01:30.
Reply With Quote
  #18  
Old 13.11.2020, 13:10
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,809
Default

Quote:
Originally Posted by nathan1 View Post
But is still possible to use "packageNamePattern" : "<title>(.*?)</title>", rule? Or just add {packagename} for CopyToClipboard Action?
No.
Also, our plugin auto-does this to get the packagename.

Quote:
Originally Posted by nathan1 View Post
In your package naming I see this


...

I see that for every package is added Metal Area - Extreme Music Portal. Is possible to retrieve the real name of packet? Real names have its music genre tag in the <title> name. Look why, please
I've updated this for the next plugin-update.
In the future, please use packagizer rules to correct package titles if you need to remove parts of them.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #19  
Old 14.11.2020, 04:31
nathan1 nathan1 is offline
Black Hole
 
Join Date: Apr 2012
Posts: 274
Default

I try to change <tag> to retrieve correct title name but something doesn't work.

I use your packagizer rules to fix package titles and instead of

Code:
"packageNamePattern" : "<title>(.*?)</title>",
I use this

Code:
"packageNamePattern" : "<td style=\"word-wrap:break-word;\" width=\"99%\">(.*?)</td>",

Code:
[ {
  "enabled" : true,
  "cookies" : [ [ "masession_id", "CENSORED" ] ],
  "updateCookies" : true,
  "logging" : false,
  "maxDecryptDepth" : 1,
  "id" : 1605027636498,
  "name" : "metalarea.org example rule with cookie-login",
  "pattern" : "https?://metalarea\\.org/forum/index\\.php\\?showtopic=\\d+",
  "rule" : "DEEPDECRYPT",
  "packageNamePattern" : "<td style="word-wrap:break-word;" width="99%">(.*?)</td>",
  "passwordPattern" : null,
  "formPattern" : null,
  "deepPattern" : "Download from <a href="(https?://[^"]+)"",
  "rewriteReplaceWith" : null
} ]

In your update you just remove "Metal Area.. " but real name is inside this tag
Code:
<td style="word-wrap:break-word;" width="99%"> </td>

For example, in this URL
**External links are only visible to Support Staff****External links are only visible to Support Staff**


this is text that I try to set up with your packagizer rules to correct package titles

Reply With Quote
  #20  
Old 16.11.2020, 18:00
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,809
Default

I've updated it once again in our plugin.

Again:
If ay plugin for a website is available, the plugin will be used and LinkCrawler Rules for the same website will be ignored.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 16:26.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.