JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 28.12.2016, 23:52
zrato zrato is offline
Giga Loader
 
Join Date: Oct 2011
Posts: 93
Default JD Constantly rewriting cfg\downloadlist__.zip and big CPU load

See attached picture.
The downloads running are ~500kB/s and the static you can barely see. The big humps are JD writing with ~8MB/s the files cfg\downloadlistXXXXXXX.zip - then a little pause and JD will write the next file. Each of those are ~140MB zipped.
The CPU load at the same time is >25% (4 core CPU).
JD is pretty much ripping my SSD apart with over 5 GB writes every 15 Minutes.
Attached Images
File Type: png Unbenannt.png (45.5 KB, 17 views)

Last edited by zrato; 29.12.2016 at 00:00.
Reply With Quote
  #2  
Old 29.12.2016, 04:02
zrato zrato is offline
Giga Loader
 
Join Date: Oct 2011
Posts: 93
Default

Seems I have found the problem thanks to jd.controlling.downloadcontroller.DownloadWatchDog.log.0

> --ID:119TS:1482976127383-29.12.16 02:48:47 - [jd.controlling.downloadcontroller.DownloadWatchDog(setFinalLinkStatus)] -> DownloadCandidate:xxx.zip@fileflyer.com|Host fileflyer.com|Account:Plugin:fileflyer.com|Version:35367|Type:NONE|Account:null|Proxy:Direkt->FILE_UNAVAILABLE

It was a download in my list that was unavailable from fileflyer.com. It was tried like every 30s. If I see it correctly, you save the download list directly into the zip, yes? Every time the download attempt started the package size/loaded MB changed, so I guess this is why an updated file was written?
Also had one similar download from sendmyway that I deleted from the list prior to deleting the fileflyer download, but that one did not change the behavior (probably because the fileflyer download was still in the list?). Now with both those downloads removed, not only did the giant copy orgies stop, the CPU usage dropped significantly and even my RAM usage dropped very significantly (2,6 GB -> 1,9 GB and still falling. EDIT: 1,7 GB now).
Attached Images
File Type: png Unbenannt2.png (45.0 KB, 14 views)

Last edited by zrato; 29.12.2016 at 04:12.
Reply With Quote
  #3  
Old 29.12.2016, 21:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,908
Default

Ticket:


GreeZ pspzockerscene
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #4  
Old 27.03.2017, 14:53
zrato zrato is offline
Giga Loader
 
Join Date: Oct 2011
Posts: 93
Default Please stop writing downloadList__.zip all the time

JD2 backups the download list in the cfg folder to some zip file.
This file for me is ~175MB in size.
Not only does this shred my SSD with giant amounts of writes (compared to what it does normally) but it also uses a lot of CPU as you can see in the screenshot - Windows says 25% CPU for ~25s (on a 4 core CPU).

The problem gets worse (or starts happening), when there are some stray downloads trying to re-run (for example if the host does not respond or there is some problem with the plugin that cannot get the file or such).

Please give the user the option (expert options) to have a hard minimum backup time limit where I can set the minutes between two backups.
Attached Images
File Type: png Unbenannt.png (33.0 KB, 14 views)
Reply With Quote
  #5  
Old 14.04.2017, 23:38
zrato zrato is offline
Giga Loader
 
Join Date: Oct 2011
Posts: 93
Default

Next example, downloads running at 800KB/s, JD writing constantly 7MB/s pretty much without any pause. And giving me a constant 25% CPU on a 4 core i5.
Attached Images
File Type: png Unbenannt.png (47.8 KB, 12 views)
Reply With Quote
  #6  
Old 15.04.2017, 02:02
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,208
Default

please refrain from making new threads about something you have commented on previously.

I've merged your duplicated threads .

Maybe look at finding solution/work around, for example adjust the saving of configs settings in the mean time (Waiting for SVN ticket to get resolution).

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #7  
Old 03.06.2017, 19:30
travestree travestree is offline
Giga Loader
 
Join Date: Oct 2013
Posts: 91
Default

I just noticed the writes to my ssd, this is still a problem?
Reply With Quote
  #8  
Old 04.06.2017, 01:36
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,208
Default

nothing has changed in respects to saving to disk.
there are advanced config settings to delay the writing to disk (existing for years), but longer the delay the higher the consequence when it comes to data recovery in a not so nice event.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #9  
Old 04.06.2017, 04:06
travestree travestree is offline
Giga Loader
 
Join Date: Oct 2013
Posts: 91
Default

jdlog://7810804015941/
02.06.17 20.00.13 <--> 03.06.17 16.19.31 jdlog://7810804015941/

The condition seems to occur when there is ram starvation? When I open a bajillion chrome tabs it starts to happen, even after things calm down and I have free memory again.

What is the setting for delaying the saving of the download list?



It seems to resave the download list even if the files being downloaded have not changed status, aka the same file is still being downloaded.

*ok the system has calmed down it isn't doing it as frequently but still every few minutes even though the same file is being downloaded, maybe its io piling up? Also my download list is quite large.

Last edited by travestree; 04.06.2017 at 04:22.
Reply With Quote
  #10  
Old 04.06.2017, 04:32
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,208
Default

it only ever saves configs if there has been a change

there is checks to only save every x seconds between save.

seems to be only linkcollector (linkgrabber) preference (Settings > advanced settings > filter: 'save')

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #11  
Old 11.06.2017, 00:21
travestree travestree is offline
Giga Loader
 
Join Date: Oct 2013
Posts: 91
Default

Quote:
Originally Posted by raztoki View Post
it only ever saves configs if there has been a change

there is checks to only save every x seconds between save.

seems to be only linkcollector (linkgrabber) preference (Settings > advanced settings > filter: 'save')

raztoki

So does it write excessively if downloading small files like packages filled with jpgs? I've seen the strange behavior where it is writing 3 different download lists at the same time.
Reply With Quote
  #12  
Old 16.06.2017, 15:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 67,541
Default

I will add more fine tuning settings for this next week.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 07.08.2017, 13:31
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 67,541
Default

Sorry for long waiting, I will try to find time for it this week
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 16:14.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.