JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 22.08.2020, 02:16
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default Twitter.com - Downloading Likes rate limit exceeded

Downloading the media section of a user works flawlessly as of now. However it's not possible to download a users likes. Can you implement a likes download as well?

Example link: **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #2  
Old 24.08.2020, 15:55
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Can you give feedback for this?

Thanks.
Reply With Quote
  #3  
Old 24.08.2020, 16:03
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

Hi,
Quote:
Originally Posted by accelheim View Post
Can you give feedback for this?
please stop pushing - especially if you see that I'm already around in the forum answering other threads of you.
EDIT (Also read our forum rules regarding pushing threads - that is not allowed!)

Also, you've posted your thread in the middly of the weekend where usually none of us is around ... please be patient.

Regarding the answer:
Later ...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 24.08.2020 at 16:05.
Reply With Quote
  #4  
Old 24.08.2020, 17:47
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

Added twitter "/likes" support.

Please keep in mind that a twitter account is needed in JD in order to view/download "likes" of twitter users.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #5  
Old 18.10.2020, 16:03
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Hey,

for some reason both likes and profiles stopped working from one minute to the next. The grabber doesn't grab anything right now. Maybe Twitter changed something?

Example link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #6  
Old 19.10.2020, 15:34
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Works fine here. In case you stil have issues, please create a log, see https://support.jdownloader.org/Know...d-session-logs
Please enable Debug Mode first, see Settings->Advanced Settings->Log.debugmodeenabled
restart JDownloader. Wait for error to happen and then create log and post shown logID here
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 19.10.2020, 18:14
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Quote:
Originally Posted by Jiaz View Post
Works fine here. In case you stil have issues, please create a log, see **External links are only visible to Support Staff**...
Please enable Debug Mode first, see Settings->Advanced Settings->Log.debugmodeenabled
restart JDownloader. Wait for error to happen and then create log and post shown logID here
Here is the log:

19.10.20 17.14.13 <--> 19.10.20 17.14.38 jdlog://9136325302851/
Reply With Quote
  #8  
Old 19.10.2020, 18:21
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

"To protect our users from spam and other malicious activity, this account is temporarily locked. Please log in to **External links are only visible to Support Staff****External links are only visible to Support Staff** to unlock your account."
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 19.10.2020, 18:22
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

try to disable/remove the account from JDownloader
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 19.10.2020, 18:25
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Thanks, that helped!
Reply With Quote
  #11  
Old 19.10.2020, 18:32
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Thanks for the feedback! Maybe try again with account tomorrow/later this week and check if it works
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 23.10.2020, 17:00
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

I've removed an unnecessary token in twitter loggedin mode which may or may not have caused these temporary account blocks.

However, I think that an account block can always happen if a user is mass-crawling twitter profiles so my recommendstions would be:
- Only use your twitter account to crawl profiles which you cannot see without being logged in (or use a 2nd twitter account therefore)
and/or
- Do not mass crawl stuff that you might not need any time later - only crawl what you really want to keep

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #13  
Old 03.04.2021, 13:50
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Pasting the likes link of a user in JD2 does not work for me right now. I paste it but JD2 doesn't load anything. Help is appreciated.

03.04.21 12.48.28 <--> 03.04.21 12.50.13 jdlog://0100825302851/
Reply With Quote
  #14  
Old 06.04.2021, 20:12
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

Please post a DEBUG-log and example URLs.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #15  
Old 06.04.2021, 23:02
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Quote:
Originally Posted by pspzockerscene View Post
Please post a DEBUG-log and example URLs.

-psp-
Log:
06.04.21 21.56.04 <--> 06.04.21 21.57.25 jdlog://6020825302851/

URL's:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Both profiles and likes are not working.
Reply With Quote
  #16  
Old 07.04.2021, 14:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

{"errors":[{"message":"Rate limit exceeded","code":88}]}
developer.twitter.com/en/docs/twitter-api/rate-limits
Only wait will help at the moment
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 07.04.2021 at 14:11.
Reply With Quote
  #17  
Old 07.04.2021, 22:20
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Quote:
Originally Posted by Jiaz View Post
{"errors":[{"message":"Rate limit exceeded","code":88}]}
developer.twitter.com/en/docs/twitter-api/rate-limits
Only wait will help at the moment
Thanks for the info. Is this a issue for all Twitter users? How long will I/we have to wait to download again?
Reply With Quote
  #18  
Old 08.04.2021, 14:09
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

Quote:
Originally Posted by accelheim View Post
Is this a issue for all Twitter users?
Well basically yes but it will probably also depend on your country/IP/account and other factors.
E.g. it is working fine here.
I guess especially VPN users will run into issues quickly.


Quote:
Originally Posted by pspzockerscene View Post
How long will I/we have to wait to download again?
See here:
developer.twitter.com/en/docs/twitter-api/rate-limits
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #19  
Old 08.04.2021, 18:13
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Quote:
Originally Posted by pspzockerscene View Post
Well basically yes but it will probably also depend on your country/IP/account and other factors.
E.g. it is working fine here.
I guess especially VPN users will run into issues quickly.



See here:
developer.twitter.com/en/docs/twitter-api/rate-limits
Thanks for the update. I've issues understanding what the Twitter article is saying about how long it will take to start downloading again. Could you translate it in simple terms? And what I can do to start downloading from Twitter again?
Reply With Quote
  #20  
Old 08.04.2021, 19:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@accelheim: you could try a different IP/reconnect or VPN/Proxy
here your example links did work fine, so most likely you've hit IP limits
__________________
JD-Dev & Server-Admin
Reply With Quote
  #21  
Old 16.03.2022, 19:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

I'm currently working on this.
I'll try to implement an auto-retry for whenever the rate-limit is reached but for now I've simply added two settings so you can customize the "request interval limit" for "api.twitter.com" (e.g. profile crawling) and "twimg.com" (downloading)".
Setting the api.twitter.com request interval limit to 2000-3000ms might already help to completely avoid ever reaching the API rate-limit.

I've also noticed that twitter seems to be very generous regarding such limits:
I was easily able to do 3000 requests with only 500ms of wait time in between without running into a limit but this may vary based on a lot of favtors such as logged in state (account/no twitter account), IP, country, ...

Once the limit is reached, twitter will return http-headers contaning a timestamp that defines when the limit will be over and the complete required wait time in seconds.
In my tests this was always 180 seconds.
I will try to modify future versions of our crawler in a way that auto-resume is possible.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 17.03.2022 at 15:45. Reason: Improved readability
Reply With Quote
  #22  
Old 17.03.2022, 13:38
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

The settions I've mentioned in my last post are now available in the current JDownloader version.

Next update will include one more change:
Whenever something in a profile-crawl process goes wrong, the dummy-item that is returned contains all parameters needed to continue the crawl process at the same place where it stopped e.g.:
Code:
twitter.com/username/likes?page=178&totalCrawledTweetsCount=3656&nextCursor=xxxyyy
Now if you re-add this URL later the crawler will resume where it stopped and you should be able to crawl big profiles even without a proper 100% automated rate-limit handling.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 01.09.2022 at 14:28. Reason: Fixed typo
Reply With Quote
  #23  
Old 12.01.2023, 18:33
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

We didn't get any further reports about hitting twitters rate-limits.
Therefore I will mark this thread as "Solved" for now.

Generally if any of our users runs into trouble because of twitter rate-limiting, you should be able to avoid running into those limits as explained here.

It will remain open for further reports.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 19:13.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.