JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 28.12.2016, 22:52
zrato zrato is offline
Giga Loader
 
Join Date: Oct 2011
Posts: 95
Default JD Constantly rewriting cfg\downloadlist__.zip and big CPU load

See attached picture.
The downloads running are ~500kB/s and the static you can barely see. The big humps are JD writing with ~8MB/s the files cfg\downloadlistXXXXXXX.zip - then a little pause and JD will write the next file. Each of those are ~140MB zipped.
The CPU load at the same time is >25% (4 core CPU).
JD is pretty much ripping my SSD apart with over 5 GB writes every 15 Minutes.
Attached Images
File Type: png Unbenannt.png (45.5 KB, 17 views)

Last edited by zrato; 28.12.2016 at 23:00.
Reply With Quote
  #2  
Old 29.12.2016, 03:02
zrato zrato is offline
Giga Loader
 
Join Date: Oct 2011
Posts: 95
Default

Seems I have found the problem thanks to jd.controlling.downloadcontroller.DownloadWatchDog.log.0

> --ID:119TS:1482976127383-29.12.16 02:48:47 - [jd.controlling.downloadcontroller.DownloadWatchDog(setFinalLinkStatus)] -> DownloadCandidate:xxx.zip@fileflyer.com|Host fileflyer.com|Account:Plugin:fileflyer.com|Version:35367|Type:NONE|Account:null|Proxy:Direkt->FILE_UNAVAILABLE

It was a download in my list that was unavailable from fileflyer.com. It was tried like every 30s. If I see it correctly, you save the download list directly into the zip, yes? Every time the download attempt started the package size/loaded MB changed, so I guess this is why an updated file was written?
Also had one similar download from sendmyway that I deleted from the list prior to deleting the fileflyer download, but that one did not change the behavior (probably because the fileflyer download was still in the list?). Now with both those downloads removed, not only did the giant copy orgies stop, the CPU usage dropped significantly and even my RAM usage dropped very significantly (2,6 GB -> 1,9 GB and still falling. EDIT: 1,7 GB now).
Attached Images
File Type: png Unbenannt2.png (45.0 KB, 14 views)

Last edited by zrato; 29.12.2016 at 03:12.
Reply With Quote
  #3  
Old 29.12.2016, 20:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,285
Default

Ticket:


GreeZ pspzockerscene
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #4  
Old 27.03.2017, 13:53
zrato zrato is offline
Giga Loader
 
Join Date: Oct 2011
Posts: 95
Default Please stop writing downloadList__.zip all the time

JD2 backups the download list in the cfg folder to some zip file.
This file for me is ~175MB in size.
Not only does this shred my SSD with giant amounts of writes (compared to what it does normally) but it also uses a lot of CPU as you can see in the screenshot - Windows says 25% CPU for ~25s (on a 4 core CPU).

The problem gets worse (or starts happening), when there are some stray downloads trying to re-run (for example if the host does not respond or there is some problem with the plugin that cannot get the file or such).

Please give the user the option (expert options) to have a hard minimum backup time limit where I can set the minutes between two backups.
Attached Images
File Type: png Unbenannt.png (33.0 KB, 14 views)
Reply With Quote
  #5  
Old 14.04.2017, 22:38
zrato zrato is offline
Giga Loader
 
Join Date: Oct 2011
Posts: 95
Default

Next example, downloads running at 800KB/s, JD writing constantly 7MB/s pretty much without any pause. And giving me a constant 25% CPU on a 4 core i5.
Attached Images
File Type: png Unbenannt.png (47.8 KB, 12 views)
Reply With Quote
  #6  
Old 15.04.2017, 01:02
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,375
Default

please refrain from making new threads about something you have commented on previously.

I've merged your duplicated threads .

Maybe look at finding solution/work around, for example adjust the saving of configs settings in the mean time (Waiting for SVN ticket to get resolution).

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #7  
Old 03.06.2017, 18:30
travestree travestree is offline
Giga Loader
 
Join Date: Oct 2013
Posts: 92
Default

I just noticed the writes to my ssd, this is still a problem?
Reply With Quote
  #8  
Old 04.06.2017, 00:36
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,375
Default

nothing has changed in respects to saving to disk.
there are advanced config settings to delay the writing to disk (existing for years), but longer the delay the higher the consequence when it comes to data recovery in a not so nice event.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #9  
Old 04.06.2017, 03:06
travestree travestree is offline
Giga Loader
 
Join Date: Oct 2013
Posts: 92
Default

jdlog://7810804015941/
02.06.17 20.00.13 <--> 03.06.17 16.19.31 jdlog://7810804015941/

The condition seems to occur when there is ram starvation? When I open a bajillion chrome tabs it starts to happen, even after things calm down and I have free memory again.

What is the setting for delaying the saving of the download list?



It seems to resave the download list even if the files being downloaded have not changed status, aka the same file is still being downloaded.

*ok the system has calmed down it isn't doing it as frequently but still every few minutes even though the same file is being downloaded, maybe its io piling up? Also my download list is quite large.

Last edited by travestree; 04.06.2017 at 03:22.
Reply With Quote
  #10  
Old 04.06.2017, 03:32
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,375
Default

it only ever saves configs if there has been a change

there is checks to only save every x seconds between save.

seems to be only linkcollector (linkgrabber) preference (Settings > advanced settings > filter: 'save')

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #11  
Old 10.06.2017, 23:21
travestree travestree is offline
Giga Loader
 
Join Date: Oct 2013
Posts: 92
Default

Quote:
Originally Posted by raztoki View Post
it only ever saves configs if there has been a change

there is checks to only save every x seconds between save.

seems to be only linkcollector (linkgrabber) preference (Settings > advanced settings > filter: 'save')

raztoki

So does it write excessively if downloading small files like packages filled with jpgs? I've seen the strange behavior where it is writing 3 different download lists at the same time.
Reply With Quote
  #12  
Old 16.06.2017, 14:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,763
Default

I will add more fine tuning settings for this next week.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 07.08.2017, 12:31
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,763
Default

Sorry for long waiting, I will try to find time for it this week
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 16.09.2021, 11:45
TomArrow TomArrow is offline
Super Loader
 
Join Date: Sep 2017
Posts: 25
Default

Any updates on this? And what are the mentioned settings for changing the save delay?

I recently noticed that my SSD that's not even a year old has already degraded to 90% quality. Was pretty shocked and then I saw that jD2 had 2 TB (!!!!) of IO Writes in task manager just in this last session.

The high CPU usage has been an issue forever too. For the love of God, can't you just use an SQLite table for it all? Then you can update a status with barely any disk usage at all. Rewriting a massive zip with everything in it all the time is insanity. The performance of JD2 is one of the worst software issues I have ever experienced in my whole life (using a minimum of 50% CPU all the time once a certain amount of things are in the list) and now I learn it has also half destroyed my SSD.

Please at the very least let me know how to change the save behavior. This is nuts. I'm tearing my hair out at why someone thought this was a good idea.

Edit: With SQLite you also wouldn't have to keep absolutely everything in memory. You could reduce RAM usage to the current list view and active/unfinished items.

Last edited by TomArrow; 16.09.2021 at 11:52.
Reply With Quote
  #15  
Old 16.09.2021, 13:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,763
Default

You can customize the write delay in Settings->Advanced Settings, search for
LinkCollector.minimumsavedelay
LinkCollector.maximumsavedelay
and
DownloadController.minimumsavedelay
DownloadController.maximumsavedelay

Quote:
Originally Posted by TomArrow View Post
Edit: With SQLite you also wouldn't have to keep absolutely everything in memory. You could reduce RAM usage to the current list view and active/unfinished items.
I'm sorry but without knowing the internal of an application you cannot simply do such an assumption.
1.) databases also need memory to hold at least the index to find information fast. without index in memory, access will be very slow!
2.) databases create a (huge) dependency, right now JDownloader can run perfectly fine on nearly any platform/system/environment,see
https://support.jdownloader.org/Know...bedded-devices
3.) the application still needs to hold some (yes, not all) data in memory.
4.) the complete data access need to be reworked in order to properly/synchronized read/write data memory and database/disk
5.) would impact performance on MANY levels. eg filter/search/download and simple things like scroll through the list

And most important. We did use SQL database in the beginning and it had many more disadvantages than
advantages that we moved away from it.

I still agree that storing everything in one large (zip) file has advantages and disadvantages.
And yes, I still do have plans to change to different way/method to store the information.

Advantages:
-easy to read/write AND rescue/restore
-you can easily create/update/modify with normal tools
-good enough for *normal* use cases
-easy to backup, just one file
-copy on write, all or nothing

Disadvantages:
-does perform (very) bad for large lists (*power* use case)
-large cpu/storage requirement for large lists
-large write output for large lists


The recommendation is not use the list as archive/history and only keep those links you want to download.
Database is a perfect example for history/archive/duplicate....stuff.
And in case you don't want to remove finished/archived entries from your lists, increase the
minimumsavedelay and maximumsavedelay settings.
eg set
DownloadController.minimumsavedelay to 300000 ( 5 mins)
DownloadController.maximumsavedelay to 900000 ( 15 mins)
that will cause JDownloader to only write once every 15 mins during downloads and wait
minimum 5 mins after normal change


@TomArrow: Just out of interest, how many links do you have in list? and how large is your .zip file of the list?
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 16.09.2021 at 14:22.
Reply With Quote
  #16  
Old 17.09.2021, 01:11
TomArrow TomArrow is offline
Super Loader
 
Join Date: Sep 2017
Posts: 25
Default

Quote:
Originally Posted by Jiaz View Post
You can customize the write delay in Settings->Advanced Settings, search for
LinkCollector.minimumsavedelay
LinkCollector.maximumsavedelay
and
DownloadController.minimumsavedelay
DownloadController.maximumsavedelay



I'm sorry but without knowing the internal of an application you cannot simply do such an assumption.
1.) databases also need memory to hold at least the index to find information fast. without index in memory, access will be very slow!
2.) databases create a (huge) dependency, right now JDownloader can run perfectly fine on nearly any platform/system/environment,see
**External links are only visible to Support Staff**...
3.) the application still needs to hold some (yes, not all) data in memory.
4.) the complete data access need to be reworked in order to properly/synchronized read/write data memory and database/disk
5.) would impact performance on MANY levels. eg filter/search/download and simple things like scroll through the list

And most important. We did use SQL database in the beginning and it had many more disadvantages than
advantages that we moved away from it.

I still agree that storing everything in one large (zip) file has advantages and disadvantages.
And yes, I still do have plans to change to different way/method to store the information.

Advantages:
-easy to read/write AND rescue/restore
-you can easily create/update/modify with normal tools
-good enough for *normal* use cases
-easy to backup, just one file
-copy on write, all or nothing

Disadvantages:
-does perform (very) bad for large lists (*power* use case)
-large cpu/storage requirement for large lists
-large write output for large lists


The recommendation is not use the list as archive/history and only keep those links you want to download.
Database is a perfect example for history/archive/duplicate....stuff.
And in case you don't want to remove finished/archived entries from your lists, increase the
minimumsavedelay and maximumsavedelay settings.
eg set
DownloadController.minimumsavedelay to 300000 ( 5 mins)
DownloadController.maximumsavedelay to 900000 ( 15 mins)
that will cause JDownloader to only write once every 15 mins during downloads and wait
minimum 5 mins after normal change


@TomArrow: Just out of interest, how many links do you have in list? and how large is your .zip file of the list?
Thanks for your elaborate reply, appreciate that. I'm sorry I'm being a bit rough but it's just very frustrating to me. I have screamed at my computer and this software so many times... sometimes I literally have to wait minutes for the software to become responsive. Often I change the download folder of a single link in the link collector and I have to wait half a minute for the software to respond again. It's beyond frustrating.

The thing is, I like to archive various places and so I crawl them and run the linkcollector. If I delete the finished stuff, it won't know the stuff is already downloaded.

I have 116895 links in 2911 packages in the download list and 247363 links in 20442 packages in the link collector. My CFG folder is 3.5 GB. downloadList[number].zip is 81 MB. linkcollector[number].zip is 229 MB. And that's with me already having pruned the lists twice in the past, leading to me being unable to detect duplicates as mentioned.

After my previous post I checked what settings I could find and the ones you mentioned were among them. I changed them to 180000 and 1800000 (extra zero) and restarted JD2. Since that time, JD2 has already written almost 200 GB according to Task Manager.

This is not "just" a performance issue. This is literally slowly destroying hardware which costs money.

It's also eating electricity costs. JD2 *constantly* eats at minimum one whole CPU core, which is 25% of my entire computer's performance. On average I'd say it's closer to 50% and it spikes to 70-80% or more sometimes. My CPU max is around 40W. Let's say JD2 eats 15 W on average. I run 24/7, so in a year I'll be paying 40 EUR just for the electricity of having an open download manager. Add a new SSD every 2 years and it's probably 100 EUR per year that jD2 costs me. I know this sounds arrogant, but I think this is not justified for a tool whose purpose is the occasional downloading of files. JD2 is by far the biggest consumer of system resources on my PC, and I regularly do video rendering and whatnot, so that says something.

I understand that an SQLite database is not as responsive as keeping everything in memory at all times, but that benefit is not only completely eaten up by how absolutely atrociously this solution scales, it results in such bad performance that the software stops responding, sometimes for a while. And I don't think that my use case of 100k and 200k links respectively is in any shape or form "extreme", compared to what some data hoarders might need.

Portability I understand, but SQLite is cross-platform afaik. It's just a single file that you can basically do partial updates in. Doesn't require a full blown SQL server or anything. And keeping the index in memory certainly would be preferrable to this. Also because of RAM usage, which is considerable too (4+ GB).

Actually, thinking about it, you wouldn't even have to keep the index in memory. You could just query the database whenever the view (list) is scrolled through. You'd just have to keep the unfinished/active records in memory as well as the total count of packages (or links, for non-collapsed packages) so that the scrolling works. I've done this with Javascript on a remote PHP server and it was responsive enough. With a local single-file database it should be perfectly responsive imo. Not straight-outta-memory-responsive maybe, but massively more scalable. And updates could happen instantaneously too. I don't know anything about Java, but in C# I could simply attach an event that watches for changed properties and whenever one is changed, the database is updated. All model-based, no query-writing required even.

Again, sorry for being a d*** about this all, it's just ruined my day so many times over many years, I've become a bit irrational about it lol. I just wish this software could "behave itself" on my computer haha. It's so awesome in every other way, fixing this issue would make it perfect. It's such a frustratingly unnecessary issue to have in an otherwise so wonderful program imo.

Anyway, have a good day and I pray that one day the gods of JD2 will eradicate this issue! :D

Edit: I do have to partially take back what I said about how much it wrote to disk this time because I forgot to set the setting for the linkcollector too... sorry about that. I'll observe it for a bit for a few days and then give feedback on whether it helped. I still think my suggestion is a good one though, for all the other reasons, if only for those. And it would still be better to update records in a table than to rewrite entire files, even if it only happens every few minutes.
Reply With Quote
  #17  
Old 17.09.2021, 18:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,763
Default

Quote:
Originally Posted by TomArrow View Post
I changed them to 180000 and 1800000 (extra zero) and restarted JD2.
Edit: I do have to partially take back what I said about how much it wrote to disk this time because I forgot to set the setting for the linkcollector too... sorry about that.
Thanks for the feedback.
minimum 180 000 =3 mins, so write changes to disk 3 mins after last change
maximum 1800 000 = 30 mins, delay write changes to max 30 mins

Example to explain those values: you add/modify/enable/disable/move a link = a change. 3 mins after this change
the changes will be written to disk.
when you download stuff, then the resume/position information changes while downloading, but as the download continues
the minimum counter is resetted all the time and delayed until maximum (30 mins).
__________________
JD-Dev & Server-Admin
Reply With Quote
  #18  
Old 17.09.2021, 18:47
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,763
Default

Quote:
Originally Posted by TomArrow View Post
I have 116895 links in 2911 packages in the download list and 247363 links in 20442 packages in the link collector. My CFG folder is 3.5 GB. downloadList[number].zip is 81 MB. linkcollector[number].zip is 229 MB. And that's with me already having pruned the lists twice in the past, leading to me being unable to detect duplicates as mentioned.

JD2 has already written almost 200 GB according to Task Manager.
Can you please further feedback on this (because your comment about forgotten changes for Linkcollector).
Mostly downloading stuff and no changes in linkcollector? or manging/adding links to collector?
I'm asking to find out the cause of the high write output. Because default maximum delay is 1 minute, so
during ongoing downloads, the list is saved maximum 60 times per hour, would sum up to 4GB/hour
So must be connected to your linkcollector use? or have you set VERY large buffers in JDownloader
so during download the changes take much longer?

Please help me to understand your usecase
and I will try my best to help with possible/feasible optimizations! I already have new proof of concept
storage in work that reduces file sizes even more.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #19  
Old 17.09.2021, 18:50
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,763
Default

Please don't mind me not going into details about SQLite. We've justed in the past but the experience/performance was terrible and it heavily limited features/possibilities in JDownloader.

I'm sorry but I don't see us going back to this sort of storage
but I'll do my best to further optimize storage to reduce write output.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 17.09.2021, 18:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,763
Default

Quote:
Originally Posted by TomArrow View Post
Again, sorry for being a d*** about this all, it's just ruined my day so many times over many years
Nothing to be sorry for. Most important is that both parties don't loose respect for each other.
I understand your concerns/thoughts on write output and I'm confident that we can improve it
to a point that you can *live* with it
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 13:54.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.