JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #21  
Old 23.02.2020, 13:55
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,379
Default

The plugin only supports /release/[a-z0-9\\-]+
__________________
FAQ: How to upload a Log

Last edited by raztoki; 23.02.2020 at 14:17.
Reply With Quote
  #22  
Old 24.02.2020, 00:51
RPNet-user RPNet-user is offline
JD Addict
 
Join Date: Apr 2017
Posts: 171
Default

Is there anyway around this?
Reply With Quote
  #23  
Old 24.02.2020, 03:04
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,195
Default

by creating a linkcrawler rules for the unsupported patterns, or edit the decrypter plugin to support additional website function.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #24  
Old 24.02.2020, 03:55
RPNet-user RPNet-user is offline
JD Addict
 
Join Date: Apr 2017
Posts: 171
Default

Quote:
Originally Posted by raztoki View Post
by creating a linkcrawler rules for the unsupported patterns, or edit the decrypter plugin to support additional website function.
How can a rule possibly work since the plugin will only crawl/support the given domain? I know it will not work in the pattern alone so unless it works if/when it is included in the deepPattern then I don't see how this will make it crawl/grab from it.

Where is the decrypter plugin for this domain and what is it named because I have yet to find it?
Reply With Quote
  #25  
Old 24.02.2020, 05:07
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,195
Default

Quote:
Originally Posted by RPNet-user View Post
How can a rule possibly work since the plugin will only crawl/support the given domain? I know it will not work in the pattern alone so unless it works if/when it is included in the deepPattern then I don't see how this will make it crawl/grab from it.
linkcrawler rules will create support for a url pattern that you specify, it will then scan the page for all supported links. OR return with all links within a field that regex search for. it has limited function compared to plugins, but should be able todo what you want, to return supported links.

Quote:
Originally Posted by RPNet-user View Post
Where is the decrypter plugin for this domain and what is it named because I have yet to find it?
Quote:
Originally Posted by raztoki View Post
search source for the domain name, or class name is rpdmvzcm
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #26  
Old 24.02.2020, 05:11
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,379
Default

@RPNet-user
I merged these so you can see post #6 for your second question.
Domain is the same, rmz.cr.
The rule should crawl /l/m to find /release/ that is supported by plugin.
__________________
FAQ: How to upload a Log
Reply With Quote
  #27  
Old 24.02.2020, 06:16
RPNet-user RPNet-user is offline
JD Addict
 
Join Date: Apr 2017
Posts: 171
Default

@raztoki
It will only create support for a url pattern that originate from the top level domain(rmz.cr) which is in the plugin and not rmz.cr/l/m. The pattern I modified should have worked even without the numbered pages so unless I have to add/edit the deepPattern it will not work provided that the domain in the rule over-rides the domain supported by the plugin.
All file links originate from the top level domain plus the '/release' sub-path even if I get there from rmz.cr/l/m.

Last edited by RPNet-user; 24.02.2020 at 06:47.
Reply With Quote
  #28  
Old 24.02.2020, 08:28
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,195
Default

Quote:
Originally Posted by RPNet-user View Post
@raztoki
It will only create support for a url pattern that originate from the top level domain(rmz.cr) which is in the plugin
incorrect, your regex pattern creates the support, its protocol://domain/path(s).

Quote:
Originally Posted by RPNet-user View Post
@raztoki, thanks, using python without the extra backlash does match all five of my crawler test strings, unfortunately, JD/Java must have it so testing with other languages will not benefit the test strings.
I don't agree with your statement. helps you write and match in real time. You then adapt for the situation, either java code with the extra escaping (as it will show up as error). Inside JD graphical interface you shouldn't need the extra escaping. Link crawler rules you might as its JSON
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #29  
Old 24.02.2020, 12:00
RPNet-user RPNet-user is offline
JD Addict
 
Join Date: Apr 2017
Posts: 171
Default

Yes it will create the pattern but it will not grab any links from posts/titles where the url is rmz.cr/l/m because when you go to any pages or posts originating from the newly created pattern it will always be rmz.cr/release. The rmz.cr/l/m path on their site is simply for displaying different categories (/l/m for movies, /l/s for series, and /l/b for both) it does not provide url displayable access via those paths, therefore, how will the pattern know to grab a link that contains that path.
Reply With Quote
  #30  
Old 24.02.2020, 13:06
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,379
Default

Create another rule where deepPattern contains /l/m for example.
__________________
FAQ: How to upload a Log
Reply With Quote
  #31  
Old 24.02.2020, 14:15
RPNet-user RPNet-user is offline
JD Addict
 
Join Date: Apr 2017
Posts: 171
Default

Quote:
Originally Posted by tony2long View Post
Create another rule where deepPattern contains /l/m for example.
I already did and it will not work.
Check the screenshot for a further explanation.
The left side is rmz.cr and the right side is rmz.cr/l/m but at the bottom you will see that they are in the same location--->rmz.cr/release/movietitle
Therefore, no pattern will be able to differentiate between the location of the links since the path to the actual title/post will be the same.
Attached Images
File Type: png rmz.cr.file.path.png (181.5 KB, 2 views)
Reply With Quote
  #32  
Old 24.02.2020, 15:06
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,195
Default

it wont matter, as dedicated plugin doesn't scan this either.
so you need first "pattern": to listen to /l/m etc, and then "deepPattern" : within the html body (maybe <table>) to just return /release links then dedicated plugin will do the rest.
I wouldn't personally follow multiple pages deep, just keep the links you want to scan on txt file and copy them all. Then they are all single tasks.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #33  
Old 24.02.2020, 15:09
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,379
Default

In the page source html you can find /l/m, so first rule will get that /l/m page, then the second rule grab /l/m page and find /release/.
__________________
FAQ: How to upload a Log
Reply With Quote
  #34  
Old 25.02.2020, 01:46
RPNet-user RPNet-user is offline
JD Addict
 
Join Date: Apr 2017
Posts: 171
Default

Ok, so I will just go with the first page in /l/m only.

This is what is currently working properly but from the main rmz.cr page only.

[ {
"enabled" : true,
"updateCookies" : true,
"logging" : false,
"maxDecryptDepth" : 1,
"id" : 1422443765154,
"name" : "1080p rarbg and vxt",
"pattern" : "https?://rmz\\.cr/",
"rule" : "DEEPDECRYPT",
"packageNamePattern" : null,
"passwordPattern" : null,
"formPattern" : null,
"deepPattern" : "(/release/[a-z0-9\\-]+1080p[a-z0-9\\-]+rarbg)|(/release/[a-z0-9\\-]+1080p[a-z0-9\\-]+vxt)",
"rewriteReplaceWith" : null
} ]

These patterns will not work as it just grabs everything instead of the keyword links as the above regex:
"pattern" : "https?://rmz\\.cr/l/m/[0-5]",
"pattern" : "https?://rmz\\.cr/l/m/",
Reply With Quote
  #35  
Old 25.02.2020, 02:05
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,317
Default

Quote:
Originally Posted by RPNet-user View Post
These patterns will not work as it just grabs everything instead of the keyword links as the above regex:
"pattern" : "https?://rmz\\.cr/l/m/[0-5]",
"pattern" : "https?://rmz\\.cr/l/m/",
Yes this happened because the RegEx of our own plugin was very wide.
I've limited it to the "release" URLs only for the next update which means this would be possible and would cover adding URLs to desired pages (without number = first page):
Code:
[ {
"enabled" : true,
"updateCookies" : true,
"logging" : false,
"maxDecryptDepth" : 1,
"id" : 1422443765154,
"name" : "1080p rarbg and vxt",
"pattern" : "https?://rmz\\.cr/l/b/[0-9]*?",
"rule" : "DEEPDECRYPT",
"packageNamePattern" : null,
"passwordPattern" : null,
"formPattern" : null,
"deepPattern" : "(/release/[a-z0-9\\-]+1080p[a-z0-9\\-]+rarbg)|(/release/[a-z0-9\\-]+1080p[a-z0-9\\-]+vxt)",
"rewriteReplaceWith" : null
} ]
-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #36  
Old 25.02.2020, 02:46
RPNet-user RPNet-user is offline
JD Addict
 
Join Date: Apr 2017
Posts: 171
Default

So as of this moment it is not possible to crawl and grab from /l/m due to the plugin correct?
The next update would make it possible?

Last edited by RPNet-user; 25.02.2020 at 02:53.
Reply With Quote
  #37  
Old 25.02.2020, 02:56
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,317
Default

Yes.
Plugins always have priority which is good and makes sense ... usually.
This is just an edge-case and soon not anymore.

We have a ticket about creating link crawler rules with higher priority than plugins but again in this case, your rules would then override our plugin completely and you'd have to add another rule to manually handle "/release" URLs ...
This is the ticket:



Plugin updates have just been released - you can now test the above mentioned rule!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #38  
Old 25.02.2020, 03:19
RPNet-user RPNet-user is offline
JD Addict
 
Join Date: Apr 2017
Posts: 171
Default

Thanks psp,
I was confused earlier as to which one over rides the other: rules>plugin or plugin>rules.
When trying to modify the pattern earlier I suspected that the plugin had the higher priority due to the wide regex regardless of what I specified. I will update, test, and feedback.
Reply With Quote
  #39  
Old 25.02.2020, 03:29
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,317
Default

Yeah basically you were really really unlucky.
This is our last RegEx:
Code:
https?://(?:www\\.)?rmz\\.cr/(?:release/)?(?!l/)[^/]+
This is the new one:
Code:
https?://(?:www\\.)?rmz\\.cr/release/[^/]+
I did not remove it earlier because I suspected that rmz.cr has URLs without containing "release" which then redirect to "release" URLs but I was unable to find such so I've modified our RegEx.
The other alternative would have been to block "l/" in our RegEx but the current solution is the nicer one^^

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #40  
Old 25.02.2020, 06:14
RPNet-user RPNet-user is offline
JD Addict
 
Join Date: Apr 2017
Posts: 171
Default

Thanks psp, it is working perfectly now.

I set the pattern to:
"pattern" : "https?://rmz\\.cr/l/m/*?",
I set the crawl for ----> rmz.cr/l/m

The event scripter is working perfectly as well as I just test it with rmz.cr/l/m.

I'm having problems with the logic for the linkgrabber filter 'views' rule as I'm trying to exclude 'srt' files during the grab so I tried setting a simple rule with only the file type to 'is not' 'srt'. So the rule is: Allow Links if, File isn't a 'srt'-File!
However, when I test the rule it still adds the srt files along with the video files.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 06:03.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.