[pspzockerscene]

I'm currently working on this.
I'll try to implement an auto-retry for whenever the rate-limit is reached but for now I've simply added two settings so you can customize the "request interval limit" for "api.twitter.com" (e.g. profile crawling) and "twimg.com" (downloading)".
Setting the api.twitter.com request interval limit to 2000-3000ms might already help to completely avoid ever reaching the API rate-limit.

I've also noticed that twitter seems to be very generous regarding such limits:
I was easily able to do 3000 requests with only 500ms of wait time in between without running into a limit but this may vary based on a lot of favtors such as logged in state (account/no twitter account), IP, country, ...

Once the limit is reached, twitter will return http-headers contaning a timestamp that defines when the limit will be over and the complete required wait time in seconds.
In my tests this was always 180 seconds.
I will try to modify future versions of our crawler in a way that auto-resume is possible.

-psp-

[pspzockerscene]

The settions I've mentioned in my last post are now available in the current JDownloader version.

Next update will include one more change:
Whenever something in a profile-crawl process goes wrong, the dummy-item that is returned contains all parameters needed to continue the crawl process at the same place where it stopped e.g.:

twitter.com/username/likes?page=178&totalCrawledTweetsCount=3656&nextCursor=xxxyyy

Now if you re-add this URL later the crawler will resume where it stopped and you should be able to crawl big profiles even without a proper 100% automated rate-limit handling.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[pspzockerscene]

We didn't get any further reports about hitting twitters rate-limits.
Therefore I will mark this thread as "Solved" for now.

Generally if any of our users runs into trouble because of twitter rate-limiting, you should be able to avoid running into those limits as explained here.

It will remain open for further reports.