JDownloader Community - Appwork GmbH

Thread Tools Display Modes
Prev Previous Post   Next Post Next
Old 16.03.2022, 19:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
Join Date: Mar 2009
Location: Deutschland
Posts: 69,691

I'm currently working on this.
I'll try to implement an auto-retry for whenever the rate-limit is reached but for now I've simply added two settings so you can customize the "request interval limit" for "api.twitter.com" (e.g. profile crawling) and "twimg.com" (downloading)".
Setting the api.twitter.com request interval limit to 2000-3000ms might already help to completely avoid ever reaching the API rate-limit.

I've also noticed that twitter seems to be very generous regarding such limits:
I was easily able to do 3000 requests with only 500ms of wait time in between without running into a limit but this may vary based on a lot of favtors such as logged in state (account/no twitter account), IP, country, ...

Once the limit is reached, twitter will return http-headers contaning a timestamp that defines when the limit will be over and the complete required wait time in seconds.
In my tests this was always 180 seconds.
I will try to modify future versions of our crawler in a way that auto-resume is possible.

JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download

A users' JD crashes and the first thing to ask is:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 17.03.2022 at 15:45. Reason: Improved readability
Reply With Quote

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 20:40.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.