JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 18.08.2015, 01:42
247yak
Guest
 
Posts: n/a
Default Can't download multiple files: AWS Errors?

Hi

When trying to download multiple files (3000, with each file being around 1-3mb), I get: "An Error Occured! (aws hosting code). While every file was previously online during the linkgrabber, when I try and download it then shows the availability to be offline.

When I break it down to 500 file blocks, everything works fine. Any ideas on why this is happening? Is Amazon throttling my downloads?

thanks!
Reply With Quote
  #2  
Old 18.08.2015, 03:52
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,199
Default

probably 'some connection limit / some time period',

what sort of max sim dls and max chunks are you using?

if you could provide the list of links (paste into pastebin service http://jdownloader.org/pastebin/)
and your logs for that session it should help identify time parameters, and the error response.

http://jdownloader.org/knowledge/wik...d-log?s=lng_en (scroll down for JDownloader 2.0 guide)


raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #3  
Old 18.08.2015, 05:22
247yak
Guest
 
Posts: n/a
Default

Hi

Thanks for looking into this:

Max Chunks per download = 1
Max sim downloads = 2

http://jdownloader.org/pastebin/134264

Last edited by 247yak; 18.08.2015 at 05:52.
Reply With Quote
  #4  
Old 18.08.2015, 05:23
247yak
Guest
 
Posts: n/a
Default

deleted log file

Last edited by 247yak; 18.08.2015 at 05:51.
Reply With Quote
  #5  
Old 18.08.2015, 05:26
247yak
Guest
 
Posts: n/a
Default

Couldn't upload to the pastebin; they asked to find a supporter?
Reply With Quote
  #6  
Old 18.08.2015, 05:37
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,199
Default

Upload to our pastebin, then copy the url (it will say .org/pastebin/number123). Only supporters can view this.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #7  
Old 18.08.2015, 05:50
247yak
Guest
 
Posts: n/a
Default

ahh....got it.

here's the bin link:
http://jdownloader.org/pastebin/134264

thanks!
Reply With Quote
  #8  
Old 18.08.2015, 05:55
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,199
Default

directhttp plugin took the download, it failed I assume because the link expires after time or IP change

did you just add these links yourself or from deepanalyse?

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #9  
Old 18.08.2015, 07:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 65,456
Default


we should add a plugin to refresh token/session of the image
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 18.08.2015, 14:35
247yak
Guest
 
Posts: n/a
Default

Hi

I just copied the links (webpage URL) and run them through 'deep analyse' to see if it can find the download file. When you do it for an individual link (or up to 500 links) it works fine; that is it finds the download for each URL. Then if you try and download 500 files, it also works fine.

Issue begins when I try and download 1000+ files; the first 500 or so start of fine (i.e download is found from URL by deep analyse + I successfully get the download). Then after 500 files or so all the remaining files begin to "Error" out and it shows the file going offline.

The URL doesn't expire (its the webpage) and the also filename doesn't change over time. Not sure what you mean by IP change outside of perhaps Amazon blocking my IP from downloading more than 500+ files at a time?

thanks!
Reply With Quote
  #11  
Old 19.08.2015, 02:13
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,199
Default

in the image url, has a expire token and session id both can be dynamic. Hence why I'm of the opinion that dedicated plugin is required to handle these link types. The volume of downloads probably relates to triggering time in which older links expire before download.

No as in your IP address changes between the time the session links are made for your IP. Changing IP could result in all the generated links not been valid and getting the 403 forbidden response code.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #12  
Old 27.09.2015, 21:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 48,696
Default

Added support for tookapic.com.
---
[Please read]=>How/when do I get updates?
Read it carefully BEFORE complaining that something would still be not working!
Update cycles depend on your installed JD version!!

---
[Bitte lesen]=>Wie/wann bekomme ich Updates?
Bitte vollständig lesen bevor ihr euch beschwert, dass etwas immernoch nicht funktionieren würde!
Die Updatezyklen variieren je nach installierter JD Version!!


GreeZ pspzockerscene
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 21:45.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.