#1
|
|||
|
|||
[Rate limiting] Problems downloading pictures in Twitter
Hi
With the new update there are problems downloading pictures in Twitter. From time to time at the moment a picture is going to be downloaded it appears off line, after several tries it appears again on line and I'm able to download it. The last time I had this problem was with this picture: **External links are only visible to Support Staff****External links are only visible to Support Staff** Log: 24.06.21 10.00.40 <--> 24.06.21 10.11.48 jdlog://4593825302851/
__________________
- Ramón V M |
#2
|
||||
|
||||
Fixed.
Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#3
|
|||
|
|||
I created a new log after I was able to download the pictures after several tries of testing if the file was online:
24.06.21 10.00.40 <--> 24.06.21 10.20.13 jdlog://5593825302851/
__________________
- Ramón V M |
#4
|
|||
|
|||
Sorry I posted my last log without knowing it was an answer Thank you! I will wait for the next update!
__________________
- Ramón V M |
#5
|
|||
|
|||
With the new update, now I get "Server Error 503"
**External links are only visible to Support Staff****External links are only visible to Support Staff** 25.06.21 19.14.06 <--> 25.06.21 19.49.16 jdlog://5104825302851/
__________________
- Ramón V M |
#6
|
||||
|
||||
Hm that is really really strange as the same URL is working fine here.
My guess is that your IP is blocked for some reason. Either try again later or change your IP (reconnect or use a VPN/proxy) and try again. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#7
|
||||
|
||||
@RedViper: it failed for me as well in browser, took me 2-3 refresh before they worked. I guess sort of temp. issue at twitter
__________________
JD-Dev & Server-Admin |
#8
|
|||
|
|||
As I download a lot of content from Twitter, that error has not stopped appearing :(
__________________
- Ramón V M |
#9
|
||||
|
||||
Please provide a new log
__________________
JD-Dev & Server-Admin |
#10
|
|||
|
|||
Yes
Link: **External links are only visible to Support Staff****External links are only visible to Support Staff** Log: 28.06.21 12.29.29 <--> 28.06.21 12.36.14 jdlog://2314825302851/
__________________
- Ramón V M |
#11
|
|||
|
|||
Now besides I'm getting the error that the: plugin is outdated
With the same link as the last post [I still could not download the image] 28.06.21 12.29.29 <--> 28.06.21 12.55.17 jdlog://6314825302851/
__________________
- Ramón V M |
#12
|
||||
|
||||
Fixed.
(Though the 503 issue is a serverside issue which we cannot fix!) Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#13
|
||||
|
||||
@RedViper: The plugin issue will be fixed with next update but still it looks like you've been rate limited by Twitter. The final image download responds with 503 error
__________________
JD-Dev & Server-Admin |
#14
|
|||
|
|||
With the last update now I can't download any picture
"Try restarting this link (plugin outdated)" like that with all the picture links. **External links are only visible to Support Staff****External links are only visible to Support Staff** 29.06.21 19.12.30 <--> 29.06.21 19.20.22 jdlog://0024825302851/
__________________
- Ramón V M |
#15
|
||||
|
||||
Fixed.
Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#16
|
|||
|
|||
Thanks a lot!
With this new update, no error appears anymore. Only the 503 issue continues, which is very strange because I already realized that JDownloader is not the problem as you said. Because as soon as I flush my IP, this problem is solved at least temporarily
__________________
- Ramón V M |
#17
|
|||
|
|||
Sorry, I meant. Not flush my IP but flush my DNS!
As soon as I flush my DNS all the pictures with the problem 503 start to download.
__________________
- Ramón V M |
#18
|
||||
|
||||
@RedViper: flushing DNS -> most likely next time JDownloader tries to download the image another server/IP will be used. It will work until you've hit rate/request limit on that server again.
Quote:
we might try to set a request limit to avoid this issue
__________________
JD-Dev & Server-Admin |
#19
|
|||
|
|||
I'm not so sure flushing my DNS is really working as I said before
But for sure this has to be with Twitter blocking IPs or something. I haven't try performing a reconnection with a new dynamic IP but I think that it might work. Any way thank you again for assisting me!
__________________
- Ramón V M |
#20
|
||||
|
||||
With next plugin update, I've set a request limit of 4 requests/second
__________________
JD-Dev & Server-Admin |
#21
|
|||
|
|||
Quote:
Apparently it is when I download too many pictures, it only happens with the pictures. After about 10 minutes or so the error disappears and all the images start to download again. But it happens with any type of link image with twitter, after downloading several more than 20 at the same time [one by one].
__________________
- Ramón V M |
#22
|
|||
|
|||
Quote:
Thanks again!
__________________
- Ramón V M |
#23
|
||||
|
||||
Wait for next update and check if it helps
__________________
JD-Dev & Server-Admin |
#24
|
|||
|
|||
Excellent! Really, thank you very much!
__________________
- Ramón V M |
#25
|
||||
|
||||
Update is live. Hit update button and restart JDownloader
__________________
JD-Dev & Server-Admin |
#26
|
|||
|
|||
As soon as I tried with a bunch of pictures all were stop by the error
Then I tried again with only 3 and the same thing happened I waited and deleted the links I tried again with this picture **External links are only visible to Support Staff****External links are only visible to Support Staff** 30.06.21 11.58.49 <--> 30.06.21 12.09.23 jdlog://5324825302851/ And after some minutes I was able to downloaded it I tried with more images and nothing bad happened What makes me think that on my first attempt the error came up and then it is no longer that it must be a problem with the Twitter servers The first attempt with the new update list of links I could not download them, I came out with the error and better I deleted them. And then mysteriously no longer even if there were many images. Try again with a list of images and failed. I deleted the links and immediately try again and suddenly I was able **External links are only visible to Support Staff****External links are only visible to Support Staff** 30.06.21 11.58.49 <--> 30.06.21 12.18.20 jdlog://6324825302851/ :confused: I don't think it has to do with JDownloader but with Twitter, because this doesn't make much sense
__________________
- Ramón V M |
#27
|
||||
|
||||
@RedViper: Try again with next update. There was a misstake in my rate limiter.
__________________
JD-Dev & Server-Admin |
#28
|
|||
|
|||
Thank you so much, Jiaz!
At the beginning I tried all these links and everything was perfect: **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** But I really wanted to know if the error no longer appeared and I tried these: **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** And then the error appeared But this time, unlike the other times, the error did not last more than 5 minutes 01.07.21 23.20.40 <--> 01.07.21 23.37.55 jdlog://4134825302851/
__________________
- Ramón V M |
#29
|
||||
|
||||
I'm sorry but we won't be able to prohibit their rate-limiting (error 503) 100% from kicking in.
For me e.g. it does not happen at all... -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#30
|
||||
|
||||
@RedViper: with next plugin update I tried to further reduce the requests being made
__________________
JD-Dev & Server-Admin |
#31
|
|||
|
|||
With the last update I have been downloading images for more than an hour and the error has not appeared once!
Thanks so much, Jiaz! Now JDownloader has returned to be the best software also to download content on Twitter
__________________
- Ramón V M |
#32
|
|||
|
|||
Sorry for posting in a thread marked solved but I believe I'm experiencing the same issue; however for me it's still ongoing even with the latest updates. As of the latest update Twitter has stopped working entirely for me.
Add links using link grabber, use packagerizer to format the filename and save directory, hit start downloads, download errors out "An error occurred! (twitter.com) File not found" 04.07.21 23.04.06 <--> 04.07.21 23.04.05 jdlog://1444825302851/ If this is a separate issue or you need additional troubleshooting let me know what I can do to help. Edit: I should also note my issue was similar to previous posts; when downloading too many it would error or if I waited a while it would work again. Right now I've tried with one link after ten hours and it still fails. Edit 2: Tried again out of habit and found that it did work, sort of. When adding 9 links the 4th in the set had the same error "An error occurred! (twitter.com) File already exists", the other 8 added at the same time through linkgrabber downloaded without issue. I then reset my 200ish failed links and tried those again, but all of those failed instantly. 04.07.21 23.04.06 <--> 05.07.21 01.05.38 jdlog://5444825302851/ Edit 3: I tried removing the packages completely instead of resetting them which has mostly good results but still mixed. Out of 276 links 29 had the same error. Last edited by Terrowin; 05.07.2021 at 07:20. Reason: Additional info |
#33
|
||||
|
||||
@Terrowin
Please post some example URLs that you are unable to download and please post a debug log (enable debug mode first, see log help article). -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#34
|
|||
|
|||
Seems like I cant replicate the issue anymore, sorry for wasting your time.
Resetting the packages would get the same error, but completely removing the packages and recreating them through link grabber got those links working again for the bulk of those 276 twitter URLs. For the remainder there were a mix of some that I had downloaded manually, some that were unavailable (author deleted), and only a few that seemed to fail for no reason. Those last few I got to work by deleting their packages and retrying them from scratch again. |
#35
|
||||
|
||||
@Terrowin: You've added the failed Links (LinkCrawlerRetry) to the download list. Those links are not working and you cannot make them work, I'm sorry but you will have to readd those. sorry for late response about that
__________________
JD-Dev & Server-Admin |
Thread Tools | |
Display Modes | |
|
|