JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 12.01.2022, 23:23
karizzan karizzan is offline
Baby Loader
 
Join Date: Jan 2022
Posts: 7
Default Can't get Crawler link to work for the life of me

I am trying to download from cyberdrop dot to. Each image in the album is hosted on a media domain, fs-0 followed by a number, 1, 2, 3 etc, like so:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

I use the following crawler rule:
Code:
[ {
  "enabled" : true,
  "cookies" : null,
  "updateCookies" : true,
  "logging" : true,
  "maxDecryptDepth" : 0,
  "id" : 1640504858132,
  "name" : "test",
  "pattern" : "(https?://fs-04.cyberdrop.to/)(.*)",
  "rule" : "REWRITE",
  "packageNamePattern" : null,
  "passwordPattern" : null,
  "formPattern" : null,
  "deepPattern" : null,
  "rewriteReplaceWith" : "$1test/$2"
} ]
Now I copy the album url and JDownloader picks it up, but it does not want to change the url for each of the images that match the regex.

I right click on each image > Change Url > and it still shows the original url, no changes, when it should show sth like "**External links are only visible to Support Staff**
I also enable the crawler rule checkbox right above. What am I doing wrong?
Reply With Quote
  #2  
Old 13.01.2022, 10:43
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,874
Default

@karizzan: cyberdrop.me is already supported out of the box. You don't have to open the url in browser and copy the image urls.
In case the album URL does not work, please provide example links, then we can check it.



The linkcrawler rule won't work because your pattern does only match on the actual image urls but not on the album url.
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 13.01.2022 at 10:49.
Reply With Quote
  #3  
Old 13.01.2022, 12:22
karizzan karizzan is offline
Baby Loader
 
Join Date: Jan 2022
Posts: 7
Default

Quote:
Originally Posted by Jiaz View Post
@karizzan: cyberdrop.me is already supported out of the box. You don't have to open the url in browser and copy the image urls.
In case the album URL does not work, please provide example links, then we can check it.



The linkcrawler rule won't work because your pattern does only match on the actual image urls but not on the album url.
Their domain for fs-0 was down most of the time, however you could get around it by replacing "fs-0(number)" with "img-01". That is why I wanted to use the crawler rule.

There are 2 cases:
1- The image redirects to img-01 on its own, like this one below (Which JD does not download, says File Not Found, probably cause of the redirect):
**External links are only visible to Support Staff****External links are only visible to Support Staff**

2- The image does not redirect to img-01, and the link returns Invalid Response, in which case I need to manually change fs-01 to img-01, like the one below:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Also yes I thought about it not matching because it only checks for the album url, not the individual images. Is it possible to make it match with the individual images?

Last edited by karizzan; 13.01.2022 at 12:30.
Reply With Quote
  #4  
Old 13.01.2022, 12:34
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,874
Default

@karizzan: why add the images when you can just add the whole album with one click?
cyperdrop.me is supported

@karizzan: both links work fine without need of any rules, for me. looks like server issues or sort of protection that maybe it does not autoredirect.
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 13.01.2022 at 12:42.
Reply With Quote
  #5  
Old 13.01.2022, 12:51
karizzan karizzan is offline
Baby Loader
 
Join Date: Jan 2022
Posts: 7
Default

Quote:
Originally Posted by Jiaz View Post
@karizzan: why add the images when you can just add the whole album with one click?
cyperdrop.me is supported

@karizzan: both links work fine without need of any rules, for me. looks like server issues or sort of protection that maybe it does not autoredirect.
I am not adding the images separately. I copy the album's link and JD picks it up, parses all the images and begins downloading. But as I mentioned, some of those images show up as File Not Found, and when I check the individual link for those File Not Found images, they fall in either of the 2 cases I mentioned (Redirecting or Invalid Response).

Is there a way to automatically replace all the individual links for the album images so that they start with img-01 instead of fs-01? Or perhaps another smart solution?
Reply With Quote
  #6  
Old 13.01.2022, 13:09
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,874
Default

@karizzan: Please provide example links, then I can check/work in this. You should not have to do anything by yourself, especially not workaround those temp. issues.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 13.01.2022, 13:09
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,874
Default

Quote:
Originally Posted by karizzan View Post
Is there a way to automatically replace all the individual links for the album images so that they start with img-01 instead of fs-01? Or perhaps another smart solution?
Please provide example links, then I will check and make plugin work reliable so you don't have to mess with the direct urls
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 13.01.2022, 13:23
karizzan karizzan is offline
Baby Loader
 
Join Date: Jan 2022
Posts: 7
Default

Quote:
Originally Posted by Jiaz View Post
Please provide example links, then I will check and make plugin work reliable so you don't have to mess with the direct urls
**External links are only visible to Support Staff****External links are only visible to Support Staff**

This is what I get from downloading the album: **External links are only visible to Support Staff****External links are only visible to Support Staff**

And after I click on one of the images > Change url > replace fs-01 with img-01 > then press Force Download Start > It works just fine > **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #9  
Old 13.01.2022, 13:29
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,874
Default

Thanks, will check it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 13.01.2022, 16:17
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,874
Default

@karizzan: can you please create a debug log? enable Settings->Advanced Settings->FolderWatch.debugenabled and restart JDownloader, now remove/add the links again and once the download fails, create a log, see https://support.jdownloader.org/Know...d-session-logs
and post logID here
I was not able to reproduce any issues
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 13.01.2022, 16:45
karizzan karizzan is offline
Baby Loader
 
Join Date: Jan 2022
Posts: 7
Default

Quote:
Originally Posted by Jiaz View Post
@karizzan: can you please create a debug log? enable Settings->Advanced Settings->FolderWatch.debugenabled and restart JDownloader, now remove/add the links again and once the download fails, create a log, see **External links are only visible to Support Staff**...
and post logID here
I was not able to reproduce any issues
**External links are only visible to Support Staff**13.01.22 17.43.38 <--> 13.01.22 17.43.32 jdlog://8653925302851/

Is this it?
Reply With Quote
  #12  
Old 13.01.2022, 17:30
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,874
Default

@karizzan: can you open those easily in browser? looks like sort of rate limit or crawl protection.
I mean do the albums show correct in browser?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 13.01.2022, 17:35
karizzan karizzan is offline
Baby Loader
 
Join Date: Jan 2022
Posts: 7
Default

Quote:
Originally Posted by Jiaz View Post
@karizzan: can you open those easily in browser? looks like sort of rate limit or crawl protection.
I mean do the albums show correct in browser?
The album itself shows the pictures no problem, but if I open one of the pictures in a tab, I get the following
HTML Code:
This site can’t provide a secure connectionfs-01.cyberdrop.cc sent an invalid response.
ERR_SSL_PROTOCOL_ERROR
I can only bypass that by changing the domain to img-01.cyberdrop.to, instead of fs-01.cyberdrop.cc
Reply With Quote
  #14  
Old 13.01.2022, 18:53
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,874
Default

The following rule will rewrite all fs subdomains to img-01
Code:
[ {
  "enabled" : true, 
  "name" : "Cyberdrop Images",
  "pattern" : "(https?://fs-\\d+.cyberdrop.to/)(.*)",
  "rule" : "REWRITE",
  "rewriteReplaceWith" : "**External links are only visible to Support Staff**
} ]
Please know that this will only be applied when adding the links

@karizzan: What browser are you using? The error message sounds like old browser?
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 13.01.2022 at 18:59.
Reply With Quote
  #15  
Old 13.01.2022, 20:29
karizzan karizzan is offline
Baby Loader
 
Join Date: Jan 2022
Posts: 7
Default

Quote:
Originally Posted by Jiaz View Post
The following rule will rewrite all fs subdomains to img-01
Code:
[ {
  "enabled" : true, 
  "name" : "Cyberdrop Images",
  "pattern" : "(https?://fs-\\d+.cyberdrop.to/)(.*)",
  "rule" : "REWRITE",
  "rewriteReplaceWith" : "**External links are only visible to Support Staff**
} ]
Please know that this will only be applied when adding the links

@karizzan: What browser are you using? The error message sounds like old browser?
This rule works unlike the one I provided above. Huh, guess we now know why one of us is a dev and the other is a noob

I am using the latest Chrome, so no problems there. If the links are working for you, then it is probably an availability issue for my ISP or the like.

Again, very much appreciate your attentiveness and helpfulness. I'll be sure to donate to support the project.
Reply With Quote
  #16  
Old 14.01.2022, 10:30
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,874
Default

@karizzan: Thanks for the feedback and you're welcome!
Your rule has wrong rewriteReplaceWith you just added a "test", see comparison to my rule
__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 27.04.2022, 10:48
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,874
Default

Here is the rule on pastebin
pastebin.com/xMk4EUE5
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 18:11.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.