JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 07.09.2024, 13:12
egamal egamal is offline
Super Loader
 
Join Date: Jan 2017
Posts: 27
Default Link Crawler cookies gets ignored

I have a basic Link Crawler rule and I am not sure why it is not working properly

I have the following rule:

{
"cookies" : [ ["name1", "very_long_value_208_characters"], ["name2", "value2"], ..., ["name17", "value17"]],
"deepPattern" : null,
"formPattern" : null,
"id" : [somenumber],
"name" : "fancy_rule_name",
"packageNamePattern" : null,
"passwordPattern" : null,
"pattern" : "**External links are only visible to Support Staff**,
"rewriteReplaceWith" : null,
"rule" : "DIRECTHTTP",
"enabled" : true,
"logging" : true,
"updateCookies" : true
}


I know the cookies are correct because in python I can successfully download a file using a requests.session after setting the session's cookies.

Based on the logs the Link Crawler Rule gets activated but not all the cookies set in rule gets propagated,
  1. Is there an upper limit on the number of cookies that could be set in the rule?
  2. Is there a limit on the cookie value length? The most important cookie which is a login session hash does not get propagated from the rule, which is the main issue I am facing.
Reply With Quote
  #2  
Old 07.09.2024, 13:23
egamal egamal is offline
Super Loader
 
Join Date: Jan 2017
Posts: 27
Default

Please ignore my question, I found an old rule with the same pattern that was triggered before my new rule was considered. All good folks!
Reply With Quote
  #3  
Old 07.09.2024, 13:31
egamal egamal is offline
Super Loader
 
Join Date: Jan 2017
Posts: 27
Default

Another slightly related question. Is there a way to update the rules, e.g. override a file or something?

It seems that I can override `jd.controlling.linkcrawler.LinkCrawlerConfig.linkcrawlerrules.json` is there a way to avoid restarting JDownloader for the changes to take effect?

Last edited by egamal; 07.09.2024 at 13:35.
Reply With Quote
  #4  
Old 09.09.2024, 12:28
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,143
Default

Quote:
Originally Posted by egamal View Post
Another slightly related question. Is there a way to update the rules, e.g. override a file or something?
How you mean this?
You want to update rules from outside JDownloader?
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #5  
Old 10.09.2024, 01:36
egamal egamal is offline
Super Loader
 
Join Date: Jan 2017
Posts: 27
Default

> You want to update rules from outside JDownloader?
Correct. After some searching, I found a way via MyJDownloader API to set the 'LinkCrawlerRules' key of the 'jd.controlling.linkcrawler.LinkCrawlerConfig' interface.


I am still facing an issue with the `LinkCrawlerRules` and I am not sure how to debug it.

The rule gets activated (the request includes the rule cookies) for a link like this one
[prefix_that_matches_the_pattern]/filename.zip

but it does not get activated (the rule cookies are not part of the request) for a link like this one
[prefix_that_matches_the_pattern]/1080p/filename.mp4
Reply With Quote
  #6  
Old 10.09.2024, 12:12
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,143
Default

Quote:
Originally Posted by egamal View Post
The rule gets activated (the request includes the rule cookies) for a link like this one
[prefix_that_matches_the_pattern]/filename.zip

but it does not get activated (the rule cookies are not part of the request) for a link like this one
[prefix_that_matches_the_pattern]/1080p/filename.mp4
Two possibilities:
- Wrong pattern in rule
- or: the 2nd link is handled by a plugin and thus cannot be handled by the rule

Without real life example URLs and the real/full rule I will not be able to provide further help.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #7  
Old 10.09.2024, 17:20
egamal egamal is offline
Super Loader
 
Join Date: Jan 2017
Posts: 27
Default

> Without real life example URLs and the real/full rule I will not be able to provide further help.


Happy to share the rule, links and the full log. How can I securely share that?
Reply With Quote
  #8  
Old 10.09.2024, 17:47
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,143
Default

You can send that to me via PN or to psp@jdownloader.org.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #9  
Old 11.09.2024, 02:43
egamal egamal is offline
Super Loader
 
Join Date: Jan 2017
Posts: 27
Default

Quote:
Originally Posted by pspzockerscene View Post
You can send that to me via PN or to psp@jdownloader.org.
Thanks pspzockerscene before I bug you with logs. I checked all the rules before mine and there is one fairly generic yet broad rule. That rule gets triggered before the link is tested against my rule pattern.

Everything works as expected if I placed my rule before the generic broad one.

This is the generic broad rule I am referring to



{
"cookies" : null,
"deepPattern" : null,
"formPattern" : null,
"id" : 1628106034293,
"maxDecryptDepth" : 0,
"name" : "Learned file extension: mp4",
"packageNamePattern" : null,
"passwordPattern" : null,
"pattern" : "(?i)https?://.*\\.mp4($|\\?.*$)",
"rewriteReplaceWith" : null,
"rule" : "DIRECTHTTP",
"enabled" : true,
"logging" : false,
"updateCookies" : true
}


I don't recall creating that rule but based on the name "Learned file extension" it is probably auto-generated. If this is the case, where can I find more info on automatically learnt rules (I have a quite a few of them)?
Reply With Quote
  #10  
Old 11.09.2024, 12:37
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,143
Default

Quote:
Originally Posted by egamal View Post
If this is the case, where can I find more info on automatically learnt rules (I have a quite a few of them)?
There is not much more info I can provide now but if you don't want JD to add such rules, you can disable this like here:
Settings -> Advanced Settings:
Code:
LinkCrawler.autolearnextensionorigins
Change value to an empty array:
Code:
[]
The basic idea behind this feature is to make JD learn unknown file extensions so next time you copy a URL with a filename that ends with an "auto learnt file extension", the clipboard observation will just eat it as if it was supported via plugin.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #11  
Old 11.09.2024, 13:18
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,671
Default

Far as I understand the mp4 is already listened to by the DIRECTHTTP plugin, there is no need for such learned rule, unless I'm missing something here.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #12  
Old 11.09.2024, 13:25
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,143
Default

This might have been a super old entry from when the "auto learn file extensions" feature has been added for the first time.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 07:08.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.