JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 17.06.2021, 05:21
inzzzomnia inzzzomnia is offline
JD Fan
 
Join Date: Apr 2017
Posts: 72
Default Specific regex version to use for linkcrawler?

I'm new with the whole regex thing and is trying to figure out how to use the REWRITE crawler rules but I'm pulling my hair since I can't get it to work.

In your knowledge base you link to regex101 for checking your code. So I create something that works there but when trying to use it in JD I just get this error
Code:
is not valid 'java.util.List<jd.controlling.linkcrawler.LinkCrawlerRuleStorable>'
And when trying the example pattern from https://support.jdownloader.org/Know...xample-rewrite on regex101 I can't get it to pass with an OK so I'm kind of lost.

What settings should I use on regex101 so it's compatible with JD?

Where can I find a reference to the regex that's used in your examples because the links you provide in the knowledgebase and sites I found myself doesn't seem to cover whatever version is used in JD or I'm missreading something completely.

Reply With Quote
  #2  
Old 17.06.2021, 10:45
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,126
Default

@inzzzomnia: The rules are JSON strings and therefore must be encoded properly.
Most important is to escape the \ and " character. On regex101, you should change to java flavor.
Just post example/your rule here and we will explain/help with JSON escaping, eg freeformatter.com/json-escape.html
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 17.06.2021 at 17:28.
Reply With Quote
  #3  
Old 17.06.2021, 17:10
inzzzomnia inzzzomnia is offline
JD Fan
 
Join Date: Apr 2017
Posts: 72
Default

The double backslashes is was confuses me the most in your examples, can't figure out what purpose they serve since there's no single backslash to escape at that place in the URL.

This gets a green light with java 8 selected on regex101

(https:\/\/img.nbc.com\/sites\/[a-z]*\/files\/images\/\d*\/\d*\/\d*\/NUP_\d*_\d*.JPG)\?impolicy=.*&imwidth=\d*&imdensity=\d*

and match what I try to fix

**External links are only visible to Support Staff****External links are only visible to Support Staff**

But JD still wont accept

Code:
[{
	"enabled": true,
	"name": "nbc",
	"pattern": "(https:\/\/img.nbc.com\/sites\/[a-z]*\/files\/images\/\d*\/\d*\/\d*\/NUP_\d*_\d*.JPG)\?impolicy=.*&imwidth=\d*&imdensity=\d*",
	"rule": "REWRITE",
	"rewriteReplaceWith": "$1"
}]
Why not?
Reply With Quote
  #4  
Old 17.06.2021, 17:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,827
Default

Quote:
Originally Posted by inzzzomnia View Post
Why not?
Stop escaping single normal (forward) slashes in JD

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 17.06.2021 at 17:27.
Reply With Quote
  #5  
Old 17.06.2021, 17:28
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,126
Default

see pastebin.com/PxQYAW13
You don't have to escape /, you have to make sure to proper json escaping
see
freeformatter.com/json-escape.html
jsonformatter.org/json-pretty-print
__________________
JD-Dev & Server-Admin

Last edited by pspzockerscene; 18.06.2021 at 12:04. Reason: fixed reeformatter.com --> freeformatter.com
Reply With Quote
  #6  
Old 17.06.2021, 17:31
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,827
Default

Also you should escape dots (see the meaning of a dot in regex101.com) though in this case no escaping them will still do the job.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #7  
Old 17.06.2021, 23:45
inzzzomnia inzzzomnia is offline
JD Fan
 
Join Date: Apr 2017
Posts: 72
Default

Quote:
Originally Posted by pspzockerscene View Post
Stop escaping single normal (forward) slashes in JD

-psp-
That wasn't the issue though, was it? Seems to work either way.
Quote:
Originally Posted by pspzockerscene View Post
Also you should escape dots (see the meaning of a dot in regex101.com) though in this case no escaping them will still do the job.
I know, I'm past the point of the meaning of the dot in this adventure But as you say I didn't see the need for doing it here.

Quote:
Originally Posted by Jiaz View Post
see pastebin.com/PxQYAW13
You don't have to escape /, you have to make sure to proper json escaping
see
freeformatter.com/json-escape.html
jsonformatter.org/json-pretty-print
Thanks for that, you should link those two sites in the knowledge base instead of the Wikipedia page as it doesn't cover the json version at all and just makes it confusing. It was thanks to that I thought you should use the ordinary regex and then got confused by your double backslashes and stuff
The check on regex101 was marking the slashes red in my first attempts at this but with escaping they was marked OK so I went with it.
So with that said regex101 really can't be used since it OKs versions that isn't working in JD, even with the "proper" java setting. Any other tool that has the same explaining thingy but actually help you with a proper version that will work with JD?

Last edited by pspzockerscene; 18.06.2021 at 12:04. Reason: fixed reeformatter.com --> freeformatter.com
Reply With Quote
  #8  
Old 18.06.2021, 12:35
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,827
Default

Quote:
Originally Posted by inzzzomnia View Post
I know, I'm past the point of the meaning of the dot in this adventure But as you say I didn't see the need for doing it here.
I understand - I'm lazy too :D

Quote:
Originally Posted by inzzzomnia View Post
Thanks for that, you should link those two sites in the knowledge base instead of the Wikipedia page as it doesn't cover the json version at all and just makes it confusing. It was thanks to that I thought you should use the ordinary regex and then got confused by your double backslashes and stuff
I've just linked two json editors and one json escaping webtool in our LinkCrawler Rules knowledgebase article.

Quote:
Originally Posted by inzzzomnia View Post
So with that said regex101 really can't be used since it OKs versions that isn't working in JD, even with the "proper" java setting. Any other tool that has the same explaining thingy but actually help you with a proper version that will work with JD?
Just use Regex101 and then do the proper replacements to make it fit for JD.
Really the only thing that bothers me is the double backslash escaping but as said you can easily fix this via search and replace.
You could probably easily make a macro/script for an advanced text editor such as "Notepad++" that can auto-handle these replacements.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #9  
Old 18.06.2021, 13:16
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,314
Default

Quote:
Originally Posted by inzzzomnia View Post
So with that said regex101 really can't be used since it OKs versions that isn't working in JD, even with the "proper" java setting. Any other tool that has the same explaining thingy but actually help you with a proper version that will work with JD?
Pattern generated on that site with 'java 8' flavor should work just fine directly in packagizer, linkfilter etc. Additional escaping is required only for settings like linkcrawler/domain rules which are stored in JSON format. You can run the 'Code Generator' found at the bottom on the same page on that site and copy just the escaped pattern (final String regex) from the generated code and use it directly in JSON objects in JD.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 03:23.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.