JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 24.06.2021, 18:13
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Exclamation [Rate limiting] Problems downloading pictures in Twitter

Hi

With the new update there are problems downloading pictures in Twitter.


From time to time at the moment a picture is going to be downloaded it appears off line, after several tries it appears again on line and I'm able to download it.

The last time I had this problem was with this picture:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Log:
24.06.21 10.00.40 <--> 24.06.21 10.11.48 jdlog://4593825302851/
__________________
- Ramón V M
Reply With Quote
  #2  
Old 24.06.2021, 18:18
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Fixed.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #3  
Old 24.06.2021, 18:20
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

I created a new log after I was able to download the pictures after several tries of testing if the file was online:
24.06.21 10.00.40 <--> 24.06.21 10.20.13 jdlog://5593825302851/
__________________
- Ramón V M
Reply With Quote
  #4  
Old 24.06.2021, 18:24
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

Quote:
Originally Posted by pspzockerscene View Post
Fixed.

-psp-

Sorry I posted my last log without knowing it was an answer

Thank you! I will wait for the next update!
__________________
- Ramón V M
Reply With Quote
  #5  
Old 26.06.2021, 03:49
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

With the new update, now I get "Server Error 503"

**External links are only visible to Support Staff****External links are only visible to Support Staff**

25.06.21 19.14.06 <--> 25.06.21 19.49.16 jdlog://5104825302851/
__________________
- Ramón V M
Reply With Quote
  #6  
Old 28.06.2021, 14:59
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Hm that is really really strange as the same URL is working fine here.
My guess is that your IP is blocked for some reason.

Either try again later or change your IP (reconnect or use a VPN/proxy) and try again.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #7  
Old 28.06.2021, 15:23
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@RedViper: it failed for me as well in browser, took me 2-3 refresh before they worked. I guess sort of temp. issue at twitter
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 28.06.2021, 18:58
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

As I download a lot of content from Twitter, that error has not stopped appearing :(
__________________
- Ramón V M
Reply With Quote
  #9  
Old 28.06.2021, 19:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Please provide a new log
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 28.06.2021, 20:35
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

Yes

Link:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Log:

28.06.21 12.29.29 <--> 28.06.21 12.36.14 jdlog://2314825302851/
__________________
- Ramón V M
Reply With Quote
  #11  
Old 28.06.2021, 20:56
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

Now besides I'm getting the error that the: plugin is outdated

With the same link as the last post [I still could not download the image]

28.06.21 12.29.29 <--> 28.06.21 12.55.17 jdlog://6314825302851/
__________________
- Ramón V M
Reply With Quote
  #12  
Old 29.06.2021, 15:24
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Fixed.
(Though the 503 issue is a serverside issue which we cannot fix!)

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #13  
Old 29.06.2021, 15:39
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@RedViper: The plugin issue will be fixed with next update but still it looks like you've been rate limited by Twitter. The final image download responds with 503 error
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 30.06.2021, 03:21
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

With the last update now I can't download any picture

"Try restarting this link (plugin outdated)" like that with all the picture links.

**External links are only visible to Support Staff****External links are only visible to Support Staff**


29.06.21 19.12.30 <--> 29.06.21 19.20.22 jdlog://0024825302851/
__________________
- Ramón V M
Reply With Quote
  #15  
Old 30.06.2021, 11:16
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Fixed.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #16  
Old 30.06.2021, 15:44
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

Thanks a lot!

With this new update, no error appears anymore.


Only the 503 issue continues, which is very strange because I already realized that JDownloader is not the problem as you said. Because as soon as I flush my IP, this problem is solved at least temporarily
__________________
- Ramón V M
Reply With Quote
  #17  
Old 30.06.2021, 15:56
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

Sorry, I meant. Not flush my IP but flush my DNS!

As soon as I flush my DNS all the pictures with the problem 503 start to download.
__________________
- Ramón V M
Reply With Quote
  #18  
Old 30.06.2021, 18:48
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@RedViper: flushing DNS -> most likely next time JDownloader tries to download the image another server/IP will be used. It will work until you've hit rate/request limit on that server again.

Quote:
Originally Posted by RedViper View Post
As I download a lot of content from Twitter, that error has not stopped appearing :(
how many? rough estimated number? maybe you can send us example links so we can try to reproduce the issue.
we might try to set a request limit to avoid this issue
__________________
JD-Dev & Server-Admin
Reply With Quote
  #19  
Old 30.06.2021, 18:51
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

I'm not so sure flushing my DNS is really working as I said before

But for sure this has to be with Twitter blocking IPs or something.

I haven't try performing a reconnection with a new dynamic IP but I think that it might work.

Any way thank you again for assisting me!
__________________
- Ramón V M
Reply With Quote
  #20  
Old 30.06.2021, 18:51
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

With next plugin update, I've set a request limit of 4 requests/second
__________________
JD-Dev & Server-Admin
Reply With Quote
  #21  
Old 30.06.2021, 18:56
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

Quote:
Originally Posted by Jiaz View Post
@RedViper: flushing DNS -> most likely next time JDownloader tries to download the image another server/IP will be used. It will work until you've hit rate/request limit on that server again.


how many? rough estimated number? maybe you can send us example links so we can try to reproduce the issue.
we might try to set a request limit to avoid this issue
After my last reply I saw that you had already answered me again.

Apparently it is when I download too many pictures, it only happens with the pictures. After about 10 minutes or so the error disappears and all the images start to download again.

But it happens with any type of link image with twitter, after downloading several more than 20 at the same time [one by one].
__________________
- Ramón V M
Reply With Quote
  #22  
Old 30.06.2021, 18:58
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

Quote:
Originally Posted by Jiaz View Post
With next plugin update, I've set a request limit of 4 requests/second
It must have something to do with it, but it still seems very strange to me because it is something that had not appeared before this week.



Thanks again!
__________________
- Ramón V M
Reply With Quote
  #23  
Old 30.06.2021, 19:04
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Wait for next update and check if it helps
__________________
JD-Dev & Server-Admin
Reply With Quote
  #24  
Old 30.06.2021, 19:13
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

Quote:
Originally Posted by Jiaz View Post
Wait for next update and check if it helps
Excellent! Really, thank you very much!
__________________
- Ramón V M
Reply With Quote
  #25  
Old 30.06.2021, 19:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Update is live. Hit update button and restart JDownloader
__________________
JD-Dev & Server-Admin
Reply With Quote
  #26  
Old 30.06.2021, 20:20
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

Quote:
Originally Posted by Jiaz View Post
Update is live. Hit update button and restart JDownloader
As soon as I tried with a bunch of pictures all were stop by the error

Then I tried again with only 3 and the same thing happened

I waited and deleted the links

I tried again with this picture

**External links are only visible to Support Staff****External links are only visible to Support Staff**

30.06.21 11.58.49 <--> 30.06.21 12.09.23 jdlog://5324825302851/

And after some minutes I was able to downloaded it

I tried with more images and nothing bad happened

What makes me think that on my first attempt the error came up and then it is no longer that it must be a problem with the Twitter servers

The first attempt with the new update list of links I could not download them, I came out with the error and better I deleted them. And then mysteriously no longer even if there were many images.

Try again with a list of images and failed. I deleted the links and immediately try again and suddenly I was able

**External links are only visible to Support Staff****External links are only visible to Support Staff**

30.06.21 11.58.49 <--> 30.06.21 12.18.20 jdlog://6324825302851/

:confused:

I don't think it has to do with JDownloader but with Twitter, because this doesn't make much sense
__________________
- Ramón V M
Reply With Quote
  #27  
Old 01.07.2021, 19:59
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@RedViper: Try again with next update. There was a misstake in my rate limiter.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #28  
Old 02.07.2021, 07:38
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Default

Thank you so much, Jiaz!

At the beginning I tried all these links and everything was perfect:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**


But I really wanted to know if the error no longer appeared and I tried these:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

And then the error appeared

But this time, unlike the other times, the error did not last more than 5 minutes

01.07.21 23.20.40 <--> 01.07.21 23.37.55 jdlog://4134825302851/
__________________
- Ramón V M
Reply With Quote
  #29  
Old 02.07.2021, 10:21
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

I'm sorry but we won't be able to prohibit their rate-limiting (error 503) 100% from kicking in.

For me e.g. it does not happen at all...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #30  
Old 02.07.2021, 11:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@RedViper: with next plugin update I tried to further reduce the requests being made
__________________
JD-Dev & Server-Admin
Reply With Quote
  #31  
Old 03.07.2021, 08:54
RedViper RedViper is offline
Fibre Channel User
 
Join Date: Apr 2016
Location: Mexico
Posts: 129
Cool

With the last update I have been downloading images for more than an hour and the error has not appeared once!

Thanks so much, Jiaz! Now JDownloader has returned to be the best software also to download content on Twitter
__________________
- Ramón V M
Reply With Quote
  #32  
Old 05.07.2021, 06:22
Terrowin Terrowin is offline
Modem User
 
Join Date: Jul 2021
Posts: 2
Default

Sorry for posting in a thread marked solved but I believe I'm experiencing the same issue; however for me it's still ongoing even with the latest updates. As of the latest update Twitter has stopped working entirely for me.

Add links using link grabber, use packagerizer to format the filename and save directory, hit start downloads, download errors out "An error occurred! (twitter.com) File not found"

04.07.21 23.04.06 <--> 04.07.21 23.04.05 jdlog://1444825302851/

If this is a separate issue or you need additional troubleshooting let me know what I can do to help.

Edit: I should also note my issue was similar to previous posts; when downloading too many it would error or if I waited a while it would work again. Right now I've tried with one link after ten hours and it still fails.

Edit 2: Tried again out of habit and found that it did work, sort of. When adding 9 links the 4th in the set had the same error "An error occurred! (twitter.com) File already exists", the other 8 added at the same time through linkgrabber downloaded without issue. I then reset my 200ish failed links and tried those again, but all of those failed instantly.

04.07.21 23.04.06 <--> 05.07.21 01.05.38 jdlog://5444825302851/

Edit 3: I tried removing the packages completely instead of resetting them which has mostly good results but still mixed. Out of 276 links 29 had the same error.

Last edited by Terrowin; 05.07.2021 at 08:20. Reason: Additional info
Reply With Quote
  #33  
Old 05.07.2021, 12:34
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

@Terrowin
Please post some example URLs that you are unable to download and please post a debug log (enable debug mode first, see log help article).

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #34  
Old 06.07.2021, 17:10
Terrowin Terrowin is offline
Modem User
 
Join Date: Jul 2021
Posts: 2
Default

Seems like I cant replicate the issue anymore, sorry for wasting your time.

Resetting the packages would get the same error, but completely removing the packages and recreating them through link grabber got those links working again for the bulk of those 276 twitter URLs. For the remainder there were a mix of some that I had downloaded manually, some that were unavailable (author deleted), and only a few that seemed to fail for no reason. Those last few I got to work by deleting their packages and retrying them from scratch again.
Reply With Quote
  #35  
Old 06.07.2021, 17:28
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@Terrowin: You've added the failed Links (LinkCrawlerRetry) to the download list. Those links are not working and you cannot make them work, I'm sorry but you will have to readd those. sorry for late response about that
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 13:31.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.