JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 08.06.2021, 15:06
ES9038PRO ES9038PRO is offline
Ultra Loader
 
Join Date: Mar 2017
Posts: 49
Default [Cloudflare] Spankbang.com some videos unable to download

t

Last edited by ES9038PRO; 10.06.2022 at 02:02.
Reply With Quote
  #2  
Old 08.06.2021, 15:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,288
Default

Could not reproduce any issue, worked fine here
Please create a log, see https://support.jdownloader.org/Know...d-session-logs and post shown logID here
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 28.06.2021, 21:54
ES9038PRO ES9038PRO is offline
Ultra Loader
 
Join Date: Mar 2017
Posts: 49
Default

t

Last edited by ES9038PRO; 10.06.2022 at 02:02.
Reply With Quote
  #4  
Old 29.06.2021, 09:07
mero4711 mero4711 is offline
Tornado
 
Join Date: Sep 2010
Posts: 225
Default

Spankbang works absolutely fine for me.

While link-grabbing JD doesn't know the filesizes yet but this is no problem at all.
Reply With Quote
  #5  
Old 29.06.2021, 10:34
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,288
Default

@ES9038PRO: log doesn't contain any download activity, nor any error.
Please create a debug log. Enable Settings->Advanced Settings->Log.debugmodeenabled
and restart JDownloader. Now try to add the link again and create new log
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 29.06.2021, 17:41
woolf woolf is offline
Wind Gust
 
Join Date: Jun 2021
Posts: 41
Default

I can confirm, there is a problem with spankbang.com. When I copy a lot of links (100+) only some of them are found or some other weird stuff happens.
I just copied 180 links and 92 JD classified as offline and created one package with them. The other time only 100 were found and also weird stuff with offline packages happened.

You can try yourself with these:
Spoiler:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #7  
Old 29.06.2021, 17:43
woolf woolf is offline
Wind Gust
 
Join Date: Jun 2021
Posts: 41
Default

It probably is a new hcaptcha issue, because when I try open these links in a browser it gets me an hcaptcha page.
Reply With Quote
  #8  
Old 29.06.2021, 19:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,288
Default

Can you please provide a debug log. Enable Settings->Advanced Settings->Log.debugmodeenabled
restart JDownloader and wait for issue to happen again and create a log, see https://support.jdownloader.org/Know...d-session-logs
and post shown logID here
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 29.06.2021, 21:27
ES9038PRO ES9038PRO is offline
Ultra Loader
 
Join Date: Mar 2017
Posts: 49
Default

g

Last edited by ES9038PRO; 09.06.2022 at 21:44.
Reply With Quote
  #10  
Old 30.06.2021, 11:37
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Hi,

English / Englisch
the issue you've reported has been caused by our current Cloudflare issues.
Please read the first post of the linked thread and post in this thread if you have further questions!
In case you've posted a new thread and it will either get merged with the linked Cloudflare thread or we will lock it.
Please post Cloudflare related questions in the above linked thread only!
Attention: The first post of that linked thread contains useful information for you AND website-owners and also the hint that website-owners can solve this issue on their side which would also make our plugin work again!
There is no ETA for a Cloudflare fix from our side so it might be faster to ask website admins! Please read the first post of the linked thread completely and consider sending it to the admins of the website you are having issues with!

Deutsch / German
Das von dir gemeldete Problem wurde durch die aktuellen Cloudflare Probleme verursacht.
Bitte lies den ersten Post des Cloudflare Threads.
Falls du einen neuen Thread dazu geöffnet hast wurde dieser entweder in den bestehenden Cloudflare Thread verschoben oder mit diesem Post von mir beantwortet und geschlossen.
Weitere Fragen bitte im oben verlinkten Cloudflare Thread posten!
Achtung: Im ersten Post ist u.a. beschrieben, dass auch Webseiten-Admins das Problem auf ihrer Seite lösen können --> Unser Plugin würde dann sofort wieder funktionieren - das bitte beachten, den ersten Post vollständig lesen und ggf. an den Support der betroffenen Webseite weiterleiten!
Wie können derzeit nicht sagen, wann es von unserer Seite aus eine Lösung für das Cloudflare Problem geben wird!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #11  
Old 30.06.2021, 17:28
ES9038PRO ES9038PRO is offline
Ultra Loader
 
Join Date: Mar 2017
Posts: 49
Default

6

Last edited by ES9038PRO; 10.06.2022 at 02:02.
Reply With Quote
  #12  
Old 30.06.2021, 18:54
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,288
Default

@ES9038PRO: It depends on many different factors if/when Cloudflare will kick in. It can be caused by your IP or simply by the number of links you are trying to add/download
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 01.07.2021, 12:03
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Quote:
Originally Posted by ES9038PRO View Post
Why is the plug-in working for some people but not me? Is the cloudflare issue applied unevenly based on the IP address source? Would it be possible to use a VPN to get around this?
All of these questions are answered in the linked Cloudflare thread/FAQ...
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #14  
Old 24.07.2021, 22:02
ES9038PRO ES9038PRO is offline
Ultra Loader
 
Join Date: Mar 2017
Posts: 49
Default

Duplicate post

Last edited by ES9038PRO; 26.07.2021 at 18:00. Reason: Duplicate post
Reply With Quote
  #15  
Old 24.07.2021, 22:38
ES9038PRO ES9038PRO is offline
Ultra Loader
 
Join Date: Mar 2017
Posts: 49
Default spankbang.com linkgrabber fails on large number of links

f

Last edited by ES9038PRO; 09.06.2022 at 21:44.
Reply With Quote
  #16  
Old 26.07.2021, 15:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Merged duplicated posts.
No reason to open up a separate thread and sending a duplicated post.
We also watch threads marked as "Declined" or "Solved"!

Quote:
Originally Posted by ES9038PRO View Post
1. "Do link check" is turned off in Advanced. JD should add the links without trying to verify online status. There's no reasons the URL should not show up in linkgrabber, cloudflare or not.
"Do link check" won't have any effect for spankbang.com because in this case the URLs get processed by a crawler first (to e.g. find all available qualities).
For most of all other plugins however, "do link check" will make a difference.

Quote:
Originally Posted by ES9038PRO View Post
2. I dug through my backups for an older config of JD. It was able to load spankbang links into linkgrabber while my most recent installs always fail.
Sounds like maybe you got older or newer URLs with a so far unsupported structure.
Please provide example URLs so we can check this.

Quote:
Originally Posted by ES9038PRO View Post
I would suggest at minimum to add a longer time between linkgrabber fetching spankbang URLs. Right now it's adding them rather quickly and I think that is hitting an internal timeout somewhere and causing the attempt to fail.
Not required and/or again:
Please provide a log.

Quote:
Originally Posted by ES9038PRO View Post
I will also note that I've been able to launch spankbang many times on my browser without once seeing a cloudflare protection screen, sometimes opening many SB links at once in a browser.
As described in the linked Cloudflare thread, Cloudflare can even stop our plugins from working without that screen ever appearing in your browser.
Cloudflare also got "completely invisible" Java Script challenges which can also break our plugins.


EDIT

With your huge list of links I was indeed able to reproduce the issue with URLs appearing as "offline".
Again the issue is Cloudflare which seems to kick in after X requests.
I wasn't able to make it work without getting a new IP...
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by Jiaz; 26.07.2021 at 15:45.
Reply With Quote
  #17  
Old 26.07.2021, 15:44
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,288
Default

Quote:
Originally Posted by ES9038PRO View Post
Also, "Do link check" is turned off in Advanced. JD should add the links without trying to verify online status. There's no reasons the URL should not show up in linkgrabber, cloudflare or not. With 'do link check' turned off, sites like xhamster and xvideo properly add the URLs without checking in linkgrabber. I can add 100k XH or XV links into linkgrabber in seconds, while plugins like spankbang and womanless attempt to add each URL one at a time and take much longer, despite 'do link check' being unchecked.
I'm sorry but that's not how it works. It depends if the link is picked up by crawler or hoster plugin. For spankbang, the links are picked up/processed by crawler plugin. And in case this plugin fails, eg due to cloudflare, it does not return any found content and thus nothing shows up in linkcrawler.See previous response from pspzockerscene

Spankbang crawler plugin handles the quality selection and can return multiple qualities whereas other plugins, eg xhamster, does not provide this feature because no crawler plugin is being involved in process. The hoster plugin can only download in certain quality but cannot provide/list all qualities
__________________
JD-Dev & Server-Admin
Reply With Quote
  #18  
Old 26.07.2021, 23:36
ES9038PRO ES9038PRO is offline
Ultra Loader
 
Join Date: Mar 2017
Posts: 49
Default

y

Last edited by ES9038PRO; 10.06.2022 at 02:01.
Reply With Quote
  #19  
Old 27.07.2021, 13:36
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Quote:
Originally Posted by ES9038PRO View Post
This is inconsistent behavior and should be fixed.
No.
We have different plugins handling this differently.
A lot of websites will block powerusers - this case is no different.

Quote:
Originally Posted by ES9038PRO View Post
The plug-in should attempt to download the highest quality version of the video available after being moved to the downloads tab, just like XH and XV.
Not planned at this moment.

Quote:
Originally Posted by ES9038PRO View Post
It should not be an impossible struggle to merely add links to linkgrabber with "do link check" disabled, especially considering that cloudflare and similar protections will only become more common as time passes.
Impossible!
A lot of crawlers are designed to find multiple items e.g. photo albums.
There is no way to process those without accessing the webpage first

Quote:
Originally Posted by ES9038PRO View Post
Cloudflares protection kicks in with X requests over Y amount of time, not just "after X requests".
Not only this. There are a lot of other factors


Quote:
Originally Posted by ES9038PRO View Post
This is the exact same thing that happened with the bandcamp plug-in with too many requests in a too short of a time by the linkgrabber, which is an issue you have acknowledged and fixed already.
Not the exact same but a different, "standard" rate-limiting.

Quote:
Originally Posted by ES9038PRO View Post
If you just slowed down how often linkgrabber is sending requests to the spankbang server, the cloudflare protection would not kick in at all, and your program would do what it was designed for instead of breaking.
Quote:
Originally Posted by ES9038PRO View Post
If you just slowed down how often linkgrabber is sending requests to the spankbang server, the cloudflare protection would not kick in at all, and your program would do what it was designed for instead of breaking.
You're unable to tell without testing this.

Quote:
Originally Posted by ES9038PRO View Post
With X number of requests over Y amount of time protection being implemented on more and more websites, it's past time to give users more control over the frequency of requests by linkgrabber trying to fetch or verify links.
You can already do this from outside using 3rd party tools.
It is not planned to add extra request-limit plugin settings for such/all of our supported websites...
Most of all should work just fine out of the box.

Quote:
Originally Posted by ES9038PRO View Post
JD is a bot, and cloudflare is an anti-bot script. It's not voodoo black magic, it can be worked around by slowing down the number of requests over time.
No.
Only for a hand full of websites. For all others, the "real" Cloudflare protection kicks in and JD will not be able to handle this at all at this moment!
See our Cloudflare thread...

Quote:
Originally Posted by ES9038PRO View Post
It does the program a disservice by throwing your hands up the moment you see cloudflare in the user debug logs when there are workable solutions available.
Again wrong - there are no widespread/free/open source automated Cloudflare solutions available at this moment.
See our Cloudflare thread.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #20  
Old 27.07.2021, 15:10
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

I don't have unlimited time available to test this but here are my results so far:
Test with 1000ms request limit:
Block after ~260 URLs

This tells me that there is either no way around the block or it will maybe work with 500ms although this would slow down the crawl process significantly ... also for non-powerusers.

I'll run some more tests once I find the time but I can't promise anything.
I'm sorry to say this but as a poweruser it might sometimes be better to use tools specifically designed to download from specific websites.
I'm unsure if there are any available for spankbang.

I also want to mention that we're open source so you're free to contribute code and improve what you think can be improved.

Please don't take this as an insult but we just don't have the time to optimize every single crawler for every single website.
As long as it works for the majority of our users we're usually also happy.

There are more examples for such rate-limiting websites where we did not optimize our plugins e.g. twitter.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #21  
Old 27.07.2021, 16:23
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

2nd Test:
Request limit: 3000ms

I was able to successfully add all 594 of your 752 URLs.
125 of them are offline.

I've set that request-limit for the next update.
The sweet-spot might be somewhere in between but it's better to wait longer than not enough.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #22  
Old 28.07.2021, 03:25
ES9038PRO ES9038PRO is offline
Ultra Loader
 
Join Date: Mar 2017
Posts: 49
Default

a

Last edited by ES9038PRO; 10.06.2022 at 02:00.
Reply With Quote
  #23  
Old 28.07.2021, 12:52
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Quote:
Originally Posted by ES9038PRO View Post
I already have tested it and have a working external solution with AHK scripts that are designed to paste one line at a time from the clipboard or a pre-defined text file, with a configurable delay.
It's always nice to see custom solutions

Quote:
Originally Posted by ES9038PRO View Post
With a delay of 2500 MS, it's able to add thousands of SB links to linkgrabber without triggering cloudflare.
As said I've added a delay in our plugin to 3000ms.
If you find a better/smaller value e.g. 2500 I can use that instead.

Quote:
Originally Posted by ES9038PRO View Post
This is not an ideal solution because the active window needs to be fixated on JD so the script can keep pasting into linkgrabber, and it's not as simple as pasting a bunch of links directly into JD.
We also got APIs you can use to send stuff to JD directory e.g. via CNL2:
https://jdownloader.org/knowledge/wiki/glossary/cnl2
... or via myjd API:
https://my.jdownloader.org/developers/

Quote:
Originally Posted by ES9038PRO View Post
IMO this is going to become a larger and larger problem for JD and the list of broken plug-ins will continue to add up. You can't avoid it forever.
That's what I've thought at first too but most of all items in that list are working fine most times for most of our users.
They're only on that list to prevent more tickets from ppl asking again and again.
Sometimes Cloudflare is only used to e.g. block VPN users or all users of a specified country.

Quote:
Originally Posted by ES9038PRO View Post
If you don't have the time to tweak the linkgrabber delay on each plug-in, at least give the user the ability to do so.
Still not planned.
Again: You can use external tools to set individual request-limits for specified hosts which will work just fine.

Also again:
The plugin update I've mentioned in my last post will be released with our next CORE-update:
Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 21:45.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.