JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 11.05.2022, 03:45
academyoffetishes academyoffetishes is offline
JD Fan
 
Join Date: Feb 2018
Posts: 73
Default A few miscellaneous questions about LinkCrawler rules

I'm trying to create linkcrawler rules without asking staff for help, but there are some things that confuse me in the documentation.
  1. If you scroll down to "Basic example of the structure of a LinkCrawler Rule:", you'll see that example looks like this:

    Code:
    [
    {
    "enabled": true,
    "cookies": [
    [
    "key",
    "value"
    ]
    ],
    "updateCookies": true,
    "logging": false,
    "maxDecryptDepth": 1,
    "id": 1000001540111,
    "name": "example rule",
    "pattern": "**External links are only visible to Support Staff**,
    "rule": "DEEPDECRYPT",
    "packageNamePattern": "<title>(.*?)</title>",
    "passwordPattern": null,
    "formPattern": null,
    "deepPattern": null,
    "rewriteReplaceWith": "**External links are only visible to Support Staff**
    }
    ]
    So the rule is "DEEPDECRYPT", but it has a "rewriteReplaceWith". According to the table below it, this is only allowed in "rule": "REWRITE". So what does this do? Where does the field get its $1 from? I know what $1 means, but I don't know which field it's coming from.
  2. Using the same example, "packageNamePattern": "<title>(.*?)</title>" But in the table below, that field uses this example: **External links are only visible to Support Staff** So does this regex look at the URL or the html, or both? I have the same question about all the other "pattern" fields.
  3. How do you set the name of the file in a rule? e.g., the URL //example.com/video/-210372192_456239067 should download as the page's <h1> value on the page it came from.
  4. What kind of rule would I use for this example? **External links are only visible to Support Staff****External links are only visible to Support Staff** I want the name of the file to be "Smokie - Needles and Pins 1977.<whatever>". If you look at the <source src= on that page, you will see the url of the file to download (it has no extension).
Reply With Quote
  #2  
Old 11.05.2022, 10:11
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,793
Default

@academyoffetishes:
Quote:
Originally Posted by academyoffetishes View Post
So the rule is "DEEPDECRYPT", but it has a "rewriteReplaceWith". According to the table below it, this is only allowed in "rule": "REWRITE". So what does this do? Where does the field get its $1 from? I know what $1 means, but I don't know which field it's coming from.
it depends on the rule type (eg DEEPDECRYPT) on which fields are used for this rule. There are not different type of objects/jsons for different types of rules and thus you can set any field you want but only those used by that specific type of rule will actually get used.

Quote:
Originally Posted by academyoffetishes View Post
Using the same example, "packageNamePattern": "<title>(.*?)</title>" But in the table below, that field uses this example: **External links are only visible to Support Staff** So does this regex look at the URL or the html, or both? I have the same question about all the other "pattern" fields.
pattern(URL) specifies the pattern on which the rule matches on, rewriteReplaceWith references to pattern(URL), all other pattern match on HTML, see https://support.jdownloader.org/Know...kcrawler-rules

Quote:
Originally Posted by academyoffetishes View Post
How do you set the name of the file in a rule? e.g., the URL //example.com/video/-210372192_456239067 should download as the page's <h1> value on the page it came from.
That's not possible from within a linkcrawler rule, please use packagizer for this, see https://support.jdownloader.org/Know...paketverwalter

Quote:
Originally Posted by academyoffetishes View Post
What kind of rule would I use for this example?
wait for next plugin update, this site matches existing plugin template and videos are hosted on vk.com
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 11.05.2022 at 10:20.
Reply With Quote
  #3  
Old 12.05.2022, 00:26
academyoffetishes academyoffetishes is offline
JD Fan
 
Join Date: Feb 2018
Posts: 73
Default

Thank you for the help!

Quote:
Originally Posted by Jiaz View Post
@academyoffetishes:

it depends on the rule type (eg DEEPDECRYPT) on which fields are used for this rule. There are not different type of objects/jsons for different types of rules and thus you can set any field you want but only those used by that specific type of rule will actually get used.
Okay, so what I think that example is, is just showing off the structure, but the example wouldn't really work.

Quote:
Originally Posted by Jiaz View Post
pattern(URL) specifies the pattern on which the rule matches on, rewriteReplaceWith references to pattern(URL), all other pattern match on HTML, see **External links are only visible to Support Staff**...
Okay that's really useful to know. Is that said anywhere in the documentation, because it's news to me. I don't know if you have any interest in updating the documentation, but I do notice a lot of people asking questions about link crawler rules. If you would have any interest, I could give some suggestions on some edits that I think would reduce forum questions. For example, I think multiple examples for each rule would help me a lot.

Regardless, thank you!
Reply With Quote
  #4  
Old 12.05.2022, 10:43
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,793
Default

Quote:
Originally Posted by academyoffetishes View Post
Okay, so what I think that example is, is just showing off the structure, but the example wouldn't really work.
I'm sorry but I don't understand?! The examples are all good, see https://support.jdownloader.org/Know...kcrawler-rules
What example do you mean?

Quote:
Originally Posted by academyoffetishes View Post
Okay that's really useful to know. Is that said anywhere in the documentation
Yes, see here, https://support.jdownloader.org/Know...kcrawler-rules
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 12.05.2022, 10:55
academyoffetishes academyoffetishes is offline
JD Fan
 
Join Date: Feb 2018
Posts: 73
Default

Quote:
Originally Posted by Jiaz View Post
The examples are all good, see **External links are only visible to Support Staff**...
What example do you mean?
The examples are good, but more examples would be more helpful. I guess I am requesting more examples of different scenarios for each rule.
Reply With Quote
  #6  
Old 12.05.2022, 11:01
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 62,518
Default

It depends.
In a lot of cases you probably want more examples of regular expressions -> You should learn that separately (use e.g. regex101.com).
Also, you will find a lot more examples in our forum, just search for "link crawler rule".

If you do not find answers for your questions, you can still ask here.
At this moment I don't really see the need for more examples as we got one example for each rule-type and also a general explanation of all types and fields.
However, feedback is always welcome!
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 21:53.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.