|
#1
|
|||
|
|||
JDownloader2 crawler rule json schemas
I've made two json schemas for validating JD2's crawler rule format.
They're available at github.com/sergxerj/jdownloader2-crawler-rule-json-schema including a config folder for working locally with them in visual studio code. Further instructions and a few tips are in the repo's README.md. One scheme is for validating single rules, the other is for validating an array of rules. The latter is what you can "Ctrl+A, Ctrl+C; Ctrl+A, Ctrl+V" from the editor into JD2's "Link Crawler: Link Crawler Rules". I'll make a PR to add it to schemastore.org. Last edited by Jiaz; 21.09.2022 at 19:48. |
#2
|
||||
|
||||
@kingxerjsaeg: VERY NICE Thumbs up! Also like the good explanations!
Maybe add info to where copy/paste (Settings->Advanced Settings->LinkCrawler.linkcrawlerrules) those rules here Quote:
Quote:
Quote:
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 16.09.2022 at 12:36. |
#3
|
|||
|
|||
Thanks! Suggestions applied. Also fixed an extension misplacement in the Usage section.
|
#4
|
||||
|
||||
@kingxerjsaeg: would be great if you chould mix schemes, so eg also define value xy must be valid regex pattern
__________________
JD-Dev & Server-Admin |
#5
|
|||
|
|||
@Jiaz
Quote:
The single-rule scheme validates a rule object only to internal consistency. The multi rule-scheme just reuses this as the array-item definition and validates the listed rule objects to their individual internal consistency. Unless you mean some other scenario(?) Quote:
A small aside is that "format":"regex" validates explicitly against the ES262 regex dialect, I'm not too knowledgeable of the Java flavor but there's probably some construct out there that doesn't translate to the point of breaking the expresion from ES262's viewpoint, hopefully nothing but edge cases that will be caught by regex101. However, right now, mid-explanation, it prompted my curiosity to see if the online validators had it in their implementation... And lo! They have it. So I'm gonna add a bunch of "format":"regex" to the definition right away. |
#6
|
||||
|
||||
Nice one!
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#7
|
||||
|
||||
@kingxerjsaeg
VERY nice to see it being available on jsonschemavalidator.net Very cool to see the validator in action! you could optimize to avoid having to set enabled, because enable=true is default value. so you can set it but also leave away for shorter rules
__________________
JD-Dev & Server-Admin |
#8
|
||||
|
||||
I've also noticed some possible mistakes in the scheme checking:
Rule: Code:
[ { "deepPattern": "(https?://mega\\.nz/[^\"<>]+)", "formPattern": null, "maxDecryptDepth": 1, "name": "test", "packageNamePattern": "<title>(.*?)</title>", "passwordPattern": null, "pattern": "https?://(www\\.)?test\\.org/topic/[0-9]+-[a-z0-9\\-]+.*", "rewriteReplaceWith": null, "rule": "DEEPDECRYPT", "enabled": true, "logging": true, "updateCookies": true } ] Code:
Found 3 error(s) Message: Invalid type. Expected String but got Null. Schema path: **External links are only visible to Support Staff** Message: Invalid type. Expected String but got Null. Schema path: **External links are only visible to Support Staff** Message: Invalid type. Expected String but got Null. Schema path: **External links are only visible to Support Staff** - formPattern can be null - passwordPattern can be null - rewriteReplaceWith can be null / is only needed for rule type "REWRITE" Please let me know if I made a mistake.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#9
|
||||
|
||||
I've linked jsonschemavalidator.net in our LinkCrawler rules article.
I've included credits to you @kingxerjsaeg.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 22.09.2022 at 14:47. Reason: Credits |
Thread Tools | |
Display Modes | |
|
|