JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 16.09.2022, 11:00
kingxerjsaeg kingxerjsaeg is offline
I will play nice!
 
Join Date: Jun 2022
Posts: 3
Arrow JDownloader2 crawler rule json schemas

I've made two json schemas for validating JD2's crawler rule format.
They're available at github.com/sergxerj/jdownloader2-crawler-rule-json-schema including a config folder for working locally with them in visual studio code. Further instructions and a few tips are in the repo's README.md.

One scheme is for validating single rules, the other is for validating an array of rules. The latter is what you can "Ctrl+A, Ctrl+C; Ctrl+A, Ctrl+V" from the editor into JD2's "Link Crawler: Link Crawler Rules".

I'll make a PR to add it to schemastore.org.

Last edited by Jiaz; 21.09.2022 at 19:48.
Reply With Quote
  #2  
Old 16.09.2022, 12:33
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

@kingxerjsaeg: VERY NICE Thumbs up! Also like the good explanations!
Maybe add info to where copy/paste (Settings->Advanced Settings->LinkCrawler.linkcrawlerrules) those rules here
Quote:
This is the format that you can "Ctrl+A, Ctrl+V" in the current JD2 version as of latest commit.
maybe change
Quote:
with the Java 8 engine
to
Quote:
with the Java 8 FLAVOR
as it matches the wording on the website
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 16.09.2022 at 12:36.
Reply With Quote
  #3  
Old 17.09.2022, 00:29
kingxerjsaeg kingxerjsaeg is offline
I will play nice!
 
Join Date: Jun 2022
Posts: 3
Default

Thanks! Suggestions applied. Also fixed an extension misplacement in the Usage section.
Reply With Quote
  #4  
Old 18.09.2022, 14:54
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

@kingxerjsaeg: would be great if you chould mix schemes, so eg also define value xy must be valid regex pattern
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 19.09.2022, 06:49
kingxerjsaeg kingxerjsaeg is offline
I will play nice!
 
Join Date: Jun 2022
Posts: 3
Default

@Jiaz
Quote:
would be great if you chould mix schemes
You mean for instance "Have different types of rules (DEEPCRYPT, DIRECTHTTP, etc.) in the same multi-rule array"? Because if it's that, then it's already accounted for.

The single-rule scheme validates a rule object only to internal consistency. The multi rule-scheme just reuses this as the array-item definition and validates the listed rule objects to their individual internal consistency.
Unless you mean some other scenario(?)

Quote:
define value xy must be valid regex pattern
Reading this I remembered that I tried to do it in the definition I was working on, but "format" doesn't seem to be configured as an assertion expression for all possible values stated to be 'builtin' according to spec for the case of VScode's implementation of draft07-onwards. "format":"uri" or "format":"email" work as assertion, "format":"regex" appears to be among the annotation ones. Which is was a strong motivator for me including the "- Type: HTML RegEx" bit in the "description" properties of each rule property. On the "it's not that relevant at this point of user workflow" side, the likely fact that if there was incorrect behaviour from the rule, users would end up putting the regex against the url in the online regex validator itself, which would tell them if the expression was off anyway. So I just left it out.
A small aside is that "format":"regex" validates explicitly against the ES262 regex dialect, I'm not too knowledgeable of the Java flavor but there's probably some construct out there that doesn't translate to the point of breaking the expresion from ES262's viewpoint, hopefully nothing but edge cases that will be caught by regex101.

However, right now, mid-explanation, it prompted my curiosity to see if the online validators had it in their implementation... And lo! They have it. So I'm gonna add a bunch of "format":"regex" to the definition right away.
Reply With Quote
  #6  
Old 21.09.2022, 16:21
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,713
Default

Nice one!
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #7  
Old 21.09.2022, 19:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

@kingxerjsaeg
VERY nice to see it being available on jsonschemavalidator.net
Very cool to see the validator in action!

you could optimize to avoid having to set enabled, because enable=true is default value. so you can set it but also leave away for shorter rules
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 21.09.2022, 20:00
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,713
Default

I've also noticed some possible mistakes in the scheme checking:

Rule:
Code:
[
  {
    "deepPattern": "(https?://mega\\.nz/[^\"<>]+)",
    "formPattern": null,
    "maxDecryptDepth": 1,
    "name": "test",
    "packageNamePattern": "<title>(.*?)</title>",
    "passwordPattern": null,
    "pattern": "https?://(www\\.)?test\\.org/topic/[0-9]+-[a-z0-9\\-]+.*",
    "rewriteReplaceWith": null,
    "rule": "DEEPDECRYPT",
    "enabled": true,
    "logging": true,
    "updateCookies": true
  }
]
Errors:
Code:
Found 3 error(s)
Message:
Invalid type. Expected String but got Null.
Schema path:
**External links are only visible to Support Staff**
Message:
Invalid type. Expected String but got Null.
Schema path:
**External links are only visible to Support Staff**
Message:
Invalid type. Expected String but got Null.
Schema path:
**External links are only visible to Support Staff**
I'd say:
- formPattern can be null
- passwordPattern can be null
- rewriteReplaceWith can be null / is only needed for rule type "REWRITE"

Please let me know if I made a mistake.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #9  
Old 22.09.2022, 14:46
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,713
Default

I've linked jsonschemavalidator.net in our LinkCrawler rules article.
I've included credits to you @kingxerjsaeg.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 22.09.2022 at 14:47. Reason: Credits
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 01:23.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.