[egamal]

I have a basic Link Crawler rule and I am not sure why it is not working properly

I have the following rule:

{
"cookies" : [ ["name1", "very_long_value_208_characters"], ["name2", "value2"], ..., ["name17", "value17"]],
"deepPattern" : null,
"formPattern" : null,
"id" : [somenumber],
"name" : "fancy_rule_name",
"packageNamePattern" : null,
"passwordPattern" : null,
"pattern" : "**External links are only visible to Support Staff**,
"rewriteReplaceWith" : null,
"rule" : "DIRECTHTTP",
"enabled" : true,
"logging" : true,
"updateCookies" : true
}


I know the cookies are correct because in python I can successfully download a file using a requests.session after setting the session's cookies.

Based on the logs the Link Crawler Rule gets activated but not all the cookies set in rule gets propagated,

1. Is there an upper limit on the number of cookies that could be set in the rule?
   
2. Is there a limit on the cookie value length? The most important cookie which is a login session hash does not get propagated from the rule, which is the main issue I am facing.

[egamal]

Please ignore my question, I found an old rule with the same pattern that was triggered before my new rule was considered. All good folks!

[egamal]

Another slightly related question. Is there a way to update the rules, e.g. override a file or something?

It seems that I can override `jd.controlling.linkcrawler.LinkCrawlerConfig.linkcrawlerrules.json` is there a way to avoid restarting JDownloader for the changes to take effect?

[pspzockerscene]

Originally Posted by egamal

Another slightly related question. Is there a way to update the rules, e.g. override a file or something?

How you mean this?
You want to update rules from outside JDownloader?

[egamal]

> You want to update rules from outside JDownloader?
Correct. After some searching, I found a way via MyJDownloader API to set the 'LinkCrawlerRules' key of the 'jd.controlling.linkcrawler.LinkCrawlerConfig' interface.


I am still facing an issue with the `LinkCrawlerRules` and I am not sure how to debug it.

The rule gets activated (the request includes the rule cookies) for a link like this one
[prefix_that_matches_the_pattern]/filename.zip

but it does not get activated (the rule cookies are not part of the request) for a link like this one
[prefix_that_matches_the_pattern]/1080p/filename.mp4

[pspzockerscene]

Originally Posted by egamal

The rule gets activated (the request includes the rule cookies) for a link like this one
[prefix_that_matches_the_pattern]/filename.zip

but it does not get activated (the rule cookies are not part of the request) for a link like this one
[prefix_that_matches_the_pattern]/1080p/filename.mp4

Two possibilities:
- Wrong pattern in rule
- or: the 2nd link is handled by a plugin and thus cannot be handled by the rule

Without real life example URLs and the real/full rule I will not be able to provide further help.

[egamal]

> Without real life example URLs and the real/full rule I will not be able to provide further help.


Happy to share the rule, links and the full log. How can I securely share that?

[pspzockerscene]

You can send that to me via PN or to psp@jdownloader.org.

[egamal]

Originally Posted by pspzockerscene

You can send that to me via PN or to psp@jdownloader.org.

Thanks pspzockerscene before I bug you with logs. I checked all the rules before mine and there is one fairly generic yet broad rule. That rule gets triggered before the link is tested against my rule pattern.

Everything works as expected if I placed my rule before the generic broad one.

This is the generic broad rule I am referring to



{
"cookies" : null,
"deepPattern" : null,
"formPattern" : null,
"id" : 1628106034293,
"maxDecryptDepth" : 0,
"name" : "Learned file extension: mp4",
"packageNamePattern" : null,
"passwordPattern" : null,
"pattern" : "(?i)https?://.*\\.mp4($|\\?.*$)",
"rewriteReplaceWith" : null,
"rule" : "DIRECTHTTP",
"enabled" : true,
"logging" : false,
"updateCookies" : true
}


I don't recall creating that rule but based on the name "Learned file extension" it is probably auto-generated. If this is the case, where can I find more info on automatically learnt rules (I have a quite a few of them)?

[pspzockerscene]

Originally Posted by egamal

If this is the case, where can I find more info on automatically learnt rules (I have a quite a few of them)?

There is not much more info I can provide now but if you don't want JD to add such rules, you can disable this like here:
Settings -> Advanced Settings:

LinkCrawler.autolearnextensionorigins

Change value to an empty array:

[]

The basic idea behind this feature is to make JD learn unknown file extensions so next time you copy a URL with a filename that ends with an "auto learnt file extension", the clipboard observation will just eat it as if it was supported via plugin.

[raztoki]

Far as I understand the mp4 is already listened to by the DIRECTHTTP plugin, there is no need for such learned rule, unless I'm missing something here.

[pspzockerscene]

This might have been a super old entry from when the "auto learn file extensions" feature has been added for the first time.