JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 28.12.2016, 23:52
zrato zrato is offline
Giga Loader
 
Join Date: Oct 2011
Posts: 99
Default JD Constantly rewriting cfg\downloadlist__.zip and big CPU load

See attached picture.
The downloads running are ~500kB/s and the static you can barely see. The big humps are JD writing with ~8MB/s the files cfg\downloadlistXXXXXXX.zip - then a little pause and JD will write the next file. Each of those are ~140MB zipped.
The CPU load at the same time is >25% (4 core CPU).
JD is pretty much ripping my SSD apart with over 5 GB writes every 15 Minutes.
Attached Images
File Type: png Unbenannt.png (45.5 KB, 17 views)

Last edited by zrato; 29.12.2016 at 00:00.
Reply With Quote
  #2  
Old 29.12.2016, 04:02
zrato zrato is offline
Giga Loader
 
Join Date: Oct 2011
Posts: 99
Default

Seems I have found the problem thanks to jd.controlling.downloadcontroller.DownloadWatchDog.log.0

> --ID:119TS:1482976127383-29.12.16 02:48:47 - [jd.controlling.downloadcontroller.DownloadWatchDog(setFinalLinkStatus)] -> DownloadCandidate:xxx.zip@fileflyer.com|Host fileflyer.com|Account:Plugin:fileflyer.com|Version:35367|Type:NONE|Account:null|Proxy:Direkt->FILE_UNAVAILABLE

It was a download in my list that was unavailable from fileflyer.com. It was tried like every 30s. If I see it correctly, you save the download list directly into the zip, yes? Every time the download attempt started the package size/loaded MB changed, so I guess this is why an updated file was written?
Also had one similar download from sendmyway that I deleted from the list prior to deleting the fileflyer download, but that one did not change the behavior (probably because the fileflyer download was still in the list?). Now with both those downloads removed, not only did the giant copy orgies stop, the CPU usage dropped significantly and even my RAM usage dropped very significantly (2,6 GB -> 1,9 GB and still falling. EDIT: 1,7 GB now).
Attached Images
File Type: png Unbenannt2.png (45.0 KB, 14 views)

Last edited by zrato; 29.12.2016 at 04:12.
Reply With Quote
  #3  
Old 29.12.2016, 21:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Ticket:


GreeZ pspzockerscene
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #4  
Old 27.03.2017, 14:53
zrato zrato is offline
Giga Loader
 
Join Date: Oct 2011
Posts: 99
Default Please stop writing downloadList__.zip all the time

JD2 backups the download list in the cfg folder to some zip file.
This file for me is ~175MB in size.
Not only does this shred my SSD with giant amounts of writes (compared to what it does normally) but it also uses a lot of CPU as you can see in the screenshot - Windows says 25% CPU for ~25s (on a 4 core CPU).

The problem gets worse (or starts happening), when there are some stray downloads trying to re-run (for example if the host does not respond or there is some problem with the plugin that cannot get the file or such).

Please give the user the option (expert options) to have a hard minimum backup time limit where I can set the minutes between two backups.
Attached Images
File Type: png Unbenannt.png (33.0 KB, 14 views)
Reply With Quote
  #5  
Old 14.04.2017, 23:38
zrato zrato is offline
Giga Loader
 
Join Date: Oct 2011
Posts: 99
Default

Next example, downloads running at 800KB/s, JD writing constantly 7MB/s pretty much without any pause. And giving me a constant 25% CPU on a 4 core i5.
Attached Images
File Type: png Unbenannt.png (47.8 KB, 12 views)
Reply With Quote
  #6  
Old 15.04.2017, 02:02
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

please refrain from making new threads about something you have commented on previously.

I've merged your duplicated threads .

Maybe look at finding solution/work around, for example adjust the saving of configs settings in the mean time (Waiting for SVN ticket to get resolution).

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #7  
Old 03.06.2017, 19:30
travestree travestree is offline
Giga Loader
 
Join Date: Oct 2013
Posts: 96
Default

I just noticed the writes to my ssd, this is still a problem?
Reply With Quote
  #8  
Old 04.06.2017, 01:36
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

nothing has changed in respects to saving to disk.
there are advanced config settings to delay the writing to disk (existing for years), but longer the delay the higher the consequence when it comes to data recovery in a not so nice event.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #9  
Old 04.06.2017, 04:06
travestree travestree is offline
Giga Loader
 
Join Date: Oct 2013
Posts: 96
Default

jdlog://7810804015941/
02.06.17 20.00.13 <--> 03.06.17 16.19.31 jdlog://7810804015941/

The condition seems to occur when there is ram starvation? When I open a bajillion chrome tabs it starts to happen, even after things calm down and I have free memory again.

What is the setting for delaying the saving of the download list?



It seems to resave the download list even if the files being downloaded have not changed status, aka the same file is still being downloaded.

*ok the system has calmed down it isn't doing it as frequently but still every few minutes even though the same file is being downloaded, maybe its io piling up? Also my download list is quite large.

Last edited by travestree; 04.06.2017 at 04:22.
Reply With Quote
  #10  
Old 04.06.2017, 04:32
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

it only ever saves configs if there has been a change

there is checks to only save every x seconds between save.

seems to be only linkcollector (linkgrabber) preference (Settings > advanced settings > filter: 'save')

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #11  
Old 11.06.2017, 00:21
travestree travestree is offline
Giga Loader
 
Join Date: Oct 2013
Posts: 96
Default

Quote:
Originally Posted by raztoki View Post
it only ever saves configs if there has been a change

there is checks to only save every x seconds between save.

seems to be only linkcollector (linkgrabber) preference (Settings > advanced settings > filter: 'save')

raztoki

So does it write excessively if downloading small files like packages filled with jpgs? I've seen the strange behavior where it is writing 3 different download lists at the same time.
Reply With Quote
  #12  
Old 16.06.2017, 15:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

I will add more fine tuning settings for this next week.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 07.08.2017, 13:31
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Sorry for long waiting, I will try to find time for it this week
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 16.09.2021, 12:45
TomArrow TomArrow is offline
Super Loader
 
Join Date: Sep 2017
Posts: 27
Default

Any updates on this? And what are the mentioned settings for changing the save delay?

I recently noticed that my SSD that's not even a year old has already degraded to 90% quality. Was pretty shocked and then I saw that jD2 had 2 TB (!!!!) of IO Writes in task manager just in this last session.

The high CPU usage has been an issue forever too. For the love of God, can't you just use an SQLite table for it all? Then you can update a status with barely any disk usage at all. Rewriting a massive zip with everything in it all the time is insanity. The performance of JD2 is one of the worst software issues I have ever experienced in my whole life (using a minimum of 50% CPU all the time once a certain amount of things are in the list) and now I learn it has also half destroyed my SSD.

Please at the very least let me know how to change the save behavior. This is nuts. I'm tearing my hair out at why someone thought this was a good idea.

Edit: With SQLite you also wouldn't have to keep absolutely everything in memory. You could reduce RAM usage to the current list view and active/unfinished items.

Last edited by TomArrow; 16.09.2021 at 12:52.
Reply With Quote
  #15  
Old 16.09.2021, 14:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

You can customize the write delay in Settings->Advanced Settings, search for
LinkCollector.minimumsavedelay
LinkCollector.maximumsavedelay
and
DownloadController.minimumsavedelay
DownloadController.maximumsavedelay

Quote:
Originally Posted by TomArrow View Post
Edit: With SQLite you also wouldn't have to keep absolutely everything in memory. You could reduce RAM usage to the current list view and active/unfinished items.
I'm sorry but without knowing the internal of an application you cannot simply do such an assumption.
1.) databases also need memory to hold at least the index to find information fast. without index in memory, access will be very slow!
2.) databases create a (huge) dependency, right now JDownloader can run perfectly fine on nearly any platform/system/environment,see
https://support.jdownloader.org/Know...bedded-devices
3.) the application still needs to hold some (yes, not all) data in memory.
4.) the complete data access need to be reworked in order to properly/synchronized read/write data memory and database/disk
5.) would impact performance on MANY levels. eg filter/search/download and simple things like scroll through the list

And most important. We did use SQL database in the beginning and it had many more disadvantages than
advantages that we moved away from it.

I still agree that storing everything in one large (zip) file has advantages and disadvantages.
And yes, I still do have plans to change to different way/method to store the information.

Advantages:
-easy to read/write AND rescue/restore
-you can easily create/update/modify with normal tools
-good enough for *normal* use cases
-easy to backup, just one file
-copy on write, all or nothing

Disadvantages:
-does perform (very) bad for large lists (*power* use case)
-large cpu/storage requirement for large lists
-large write output for large lists


The recommendation is not use the list as archive/history and only keep those links you want to download.
Database is a perfect example for history/archive/duplicate....stuff.
And in case you don't want to remove finished/archived entries from your lists, increase the
minimumsavedelay and maximumsavedelay settings.
eg set
DownloadController.minimumsavedelay to 300000 ( 5 mins)
DownloadController.maximumsavedelay to 900000 ( 15 mins)
that will cause JDownloader to only write once every 15 mins during downloads and wait
minimum 5 mins after normal change


@TomArrow: Just out of interest, how many links do you have in list? and how large is your .zip file of the list?
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 16.09.2021 at 15:22.
Reply With Quote
  #16  
Old 17.09.2021, 02:11
TomArrow TomArrow is offline
Super Loader
 
Join Date: Sep 2017
Posts: 27
Default

Quote:
Originally Posted by Jiaz View Post
You can customize the write delay in Settings->Advanced Settings, search for
LinkCollector.minimumsavedelay
LinkCollector.maximumsavedelay
and
DownloadController.minimumsavedelay
DownloadController.maximumsavedelay



I'm sorry but without knowing the internal of an application you cannot simply do such an assumption.
1.) databases also need memory to hold at least the index to find information fast. without index in memory, access will be very slow!
2.) databases create a (huge) dependency, right now JDownloader can run perfectly fine on nearly any platform/system/environment,see
**External links are only visible to Support Staff**...
3.) the application still needs to hold some (yes, not all) data in memory.
4.) the complete data access need to be reworked in order to properly/synchronized read/write data memory and database/disk
5.) would impact performance on MANY levels. eg filter/search/download and simple things like scroll through the list

And most important. We did use SQL database in the beginning and it had many more disadvantages than
advantages that we moved away from it.

I still agree that storing everything in one large (zip) file has advantages and disadvantages.
And yes, I still do have plans to change to different way/method to store the information.

Advantages:
-easy to read/write AND rescue/restore
-you can easily create/update/modify with normal tools
-good enough for *normal* use cases
-easy to backup, just one file
-copy on write, all or nothing

Disadvantages:
-does perform (very) bad for large lists (*power* use case)
-large cpu/storage requirement for large lists
-large write output for large lists


The recommendation is not use the list as archive/history and only keep those links you want to download.
Database is a perfect example for history/archive/duplicate....stuff.
And in case you don't want to remove finished/archived entries from your lists, increase the
minimumsavedelay and maximumsavedelay settings.
eg set
DownloadController.minimumsavedelay to 300000 ( 5 mins)
DownloadController.maximumsavedelay to 900000 ( 15 mins)
that will cause JDownloader to only write once every 15 mins during downloads and wait
minimum 5 mins after normal change


@TomArrow: Just out of interest, how many links do you have in list? and how large is your .zip file of the list?
Thanks for your elaborate reply, appreciate that. I'm sorry I'm being a bit rough but it's just very frustrating to me. I have screamed at my computer and this software so many times... sometimes I literally have to wait minutes for the software to become responsive. Often I change the download folder of a single link in the link collector and I have to wait half a minute for the software to respond again. It's beyond frustrating.

The thing is, I like to archive various places and so I crawl them and run the linkcollector. If I delete the finished stuff, it won't know the stuff is already downloaded.

I have 116895 links in 2911 packages in the download list and 247363 links in 20442 packages in the link collector. My CFG folder is 3.5 GB. downloadList[number].zip is 81 MB. linkcollector[number].zip is 229 MB. And that's with me already having pruned the lists twice in the past, leading to me being unable to detect duplicates as mentioned.

After my previous post I checked what settings I could find and the ones you mentioned were among them. I changed them to 180000 and 1800000 (extra zero) and restarted JD2. Since that time, JD2 has already written almost 200 GB according to Task Manager.

This is not "just" a performance issue. This is literally slowly destroying hardware which costs money.

It's also eating electricity costs. JD2 *constantly* eats at minimum one whole CPU core, which is 25% of my entire computer's performance. On average I'd say it's closer to 50% and it spikes to 70-80% or more sometimes. My CPU max is around 40W. Let's say JD2 eats 15 W on average. I run 24/7, so in a year I'll be paying 40 EUR just for the electricity of having an open download manager. Add a new SSD every 2 years and it's probably 100 EUR per year that jD2 costs me. I know this sounds arrogant, but I think this is not justified for a tool whose purpose is the occasional downloading of files. JD2 is by far the biggest consumer of system resources on my PC, and I regularly do video rendering and whatnot, so that says something.

I understand that an SQLite database is not as responsive as keeping everything in memory at all times, but that benefit is not only completely eaten up by how absolutely atrociously this solution scales, it results in such bad performance that the software stops responding, sometimes for a while. And I don't think that my use case of 100k and 200k links respectively is in any shape or form "extreme", compared to what some data hoarders might need.

Portability I understand, but SQLite is cross-platform afaik. It's just a single file that you can basically do partial updates in. Doesn't require a full blown SQL server or anything. And keeping the index in memory certainly would be preferrable to this. Also because of RAM usage, which is considerable too (4+ GB).

Actually, thinking about it, you wouldn't even have to keep the index in memory. You could just query the database whenever the view (list) is scrolled through. You'd just have to keep the unfinished/active records in memory as well as the total count of packages (or links, for non-collapsed packages) so that the scrolling works. I've done this with Javascript on a remote PHP server and it was responsive enough. With a local single-file database it should be perfectly responsive imo. Not straight-outta-memory-responsive maybe, but massively more scalable. And updates could happen instantaneously too. I don't know anything about Java, but in C# I could simply attach an event that watches for changed properties and whenever one is changed, the database is updated. All model-based, no query-writing required even.

Again, sorry for being a d*** about this all, it's just ruined my day so many times over many years, I've become a bit irrational about it lol. I just wish this software could "behave itself" on my computer haha. It's so awesome in every other way, fixing this issue would make it perfect. It's such a frustratingly unnecessary issue to have in an otherwise so wonderful program imo.

Anyway, have a good day and I pray that one day the gods of JD2 will eradicate this issue! :D

Edit: I do have to partially take back what I said about how much it wrote to disk this time because I forgot to set the setting for the linkcollector too... sorry about that. I'll observe it for a bit for a few days and then give feedback on whether it helped. I still think my suggestion is a good one though, for all the other reasons, if only for those. And it would still be better to update records in a table than to rewrite entire files, even if it only happens every few minutes.
Reply With Quote
  #17  
Old 17.09.2021, 19:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by TomArrow View Post
I changed them to 180000 and 1800000 (extra zero) and restarted JD2.
Edit: I do have to partially take back what I said about how much it wrote to disk this time because I forgot to set the setting for the linkcollector too... sorry about that.
Thanks for the feedback.
minimum 180 000 =3 mins, so write changes to disk 3 mins after last change
maximum 1800 000 = 30 mins, delay write changes to max 30 mins

Example to explain those values: you add/modify/enable/disable/move a link = a change. 3 mins after this change
the changes will be written to disk.
when you download stuff, then the resume/position information changes while downloading, but as the download continues
the minimum counter is resetted all the time and delayed until maximum (30 mins).
__________________
JD-Dev & Server-Admin
Reply With Quote
  #18  
Old 17.09.2021, 19:47
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by TomArrow View Post
I have 116895 links in 2911 packages in the download list and 247363 links in 20442 packages in the link collector. My CFG folder is 3.5 GB. downloadList[number].zip is 81 MB. linkcollector[number].zip is 229 MB. And that's with me already having pruned the lists twice in the past, leading to me being unable to detect duplicates as mentioned.

JD2 has already written almost 200 GB according to Task Manager.
Can you please further feedback on this (because your comment about forgotten changes for Linkcollector).
Mostly downloading stuff and no changes in linkcollector? or manging/adding links to collector?
I'm asking to find out the cause of the high write output. Because default maximum delay is 1 minute, so
during ongoing downloads, the list is saved maximum 60 times per hour, would sum up to 4GB/hour
So must be connected to your linkcollector use? or have you set VERY large buffers in JDownloader
so during download the changes take much longer?

Please help me to understand your usecase
and I will try my best to help with possible/feasible optimizations! I already have new proof of concept
storage in work that reduces file sizes even more.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #19  
Old 17.09.2021, 19:50
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Please don't mind me not going into details about SQLite. We've justed in the past but the experience/performance was terrible and it heavily limited features/possibilities in JDownloader.

I'm sorry but I don't see us going back to this sort of storage
but I'll do my best to further optimize storage to reduce write output.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 17.09.2021, 19:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by TomArrow View Post
Again, sorry for being a d*** about this all, it's just ruined my day so many times over many years
Nothing to be sorry for. Most important is that both parties don't loose respect for each other.
I understand your concerns/thoughts on write output and I'm confident that we can improve it
to a point that you can *live* with it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #21  
Old 30.09.2021, 08:10
TomArrow TomArrow is offline
Super Loader
 
Join Date: Sep 2017
Posts: 27
Default

Hey,

sorry for the late reply but I wanted to give it a fair shake with the new settings on both linkcollector and downloadlist. So I've had it running non-stop since last time with the higher delays and while it's definitely better, it's still catastrophical imo:

Actual data downloaded during this period was maybe 25 GB. Total bytes written to disk around 215 GB. So just the amount of data written to the hard drive from config files was almost 10 times that of the actual data downloaded. Just crazy!

Performance has become a bit better but still pretty bad considering it's just a download manager. Still occasionally get those hickups where it just takes forever for the GUI to react, but better than before overall. CPU use still gets ridiculously high sometimes.

Thanks for your explanation about the delay values.

To your questions:

My use case is basically, jd2 runs in the background constantly with clipboard monitoring on. I'm on my PC many hours a day and when I see something I like (Youtube video, some file sharing link, tweet, whatever), I just Ctrl+C it so it gets added to the list and I can forget about it. I also regularly add large amounts of downloads to the downloadlist that as a whole will take weeks to complete. To give an example, it might be 50 downloads from filehosters, and we both now how limiting they are sometimes. So basically all that just keeps running in the background, I imagine constantly checking and refreshing statuses and seeing if anything new can be done, if there are any server timeouts active etc. Then some files might hit some kind of server error due to glitches and just never finish, but I have so many downloads that I just don't notice until much later. They can be in the download list for months without ever progressing past an ever-updating but ultimately never meaningfully changing error/warning message. It can be stuff like links/streams that expire, all kinds of weird stuff.

About the buffer size thingie ... I'm not sure what that means tbh. Can you elaborate?

In any case thanks for being so accommodating, I'm genuinely surprised that you're actually giving this thought. Much appreciated and a rare experience. Thank you.

Maybe my SQLite idea isn't ideal as you say, but I hope something can be figured out. Some other ideas I'm just gonna wildly throw in, perhaps uselessly: How about splitting the list by date and/or activity? Having all new stuff and all active/non-finished stuff in its own file, and maybe splitting by date, so that every day of a month has its own file, and then at the end of the month they get merged in one big swoop? For example, here's some stuff I think could be just part of a "big file" that only gets touched once a month: All finished downloads. Also let's say, all link collector stuff that's been added prior to 1 week before now, because the likelihood of that being touched is rather low if it wasn't touched already. The odd/rare changes to these like deletes could be queued in the smaller files maybe? Because let's be real, the vast majority of entries (finished downloads/older entries) never change, but they can still be useful for archival purposes, like finding out if you already downloaded a file in the past). Rewriting those every time is wasteful. It's really just a small percentage of entries that regularly get updated in some form.

Also I've recently come across this fancy thing called zlib, for which you can actually create a dictionary file using a training dataset and in my experiments it has almost rivalled 7zip's LZMA2 in some cases. Though admittedly, at the highest compression level it's likely too slow to compress, but maybe it has some useful properties at the faster modes.

If you have better ideas than these, that's perfectly fine, I won't be too hurt in my pride. I'm just happy at the thought that these issues might be resolved some day.

As a side note, it would be really cool if something could be done about the RAM usage too. It's not quite as problematic as the other issues right now, but I've also encountered the situation in the past where I just had too many entries and even with my 16 GB of RAM, stuff just wouldn't work properly anymore. Even as we speak, I have 3 GB of RAM usage and over 5 GB committed. If I just double the amount of files in my list, it will completely crash the software. It will begin shuffling around memory and hang/freeze itself, with the only way out being to change Java settings to allow it to eat even more RAM. Until I arrive at the point where all my computer's RAM is used solely to maintain jD2 which seems silly... and then to pray and hit Cleanup, deleting the entire history. This sounds like a horror scenario but I've actually been there at least twice. After all the goal for me would be to ideally have millions of files in the download list that are already finished, just so I know not to redownload them.

Thinking of that ... how about making something like another tab like "archived downloads" or something, that is not kept in memory, but still used to check if a file is already downloaded? That would suit my needs in terms of archival while not cluttering the download list. Just a thought, haven't thought it through. Because that's pretty much the main reason I keep the old entries- to know if they're already downloaded. Maybe that's something SQLite could be good for? Could use hashes to tell if a file is already downloaded and those, unlike a full text search, could be searched very efficiently via index? idk if that makes sense with the logic needed and maybe its more complex than that, I'm just brainstorming.

Anyway, let me know what more information I can provide you with to help! Thanks!

Cheers
Reply With Quote
  #22  
Old 30.09.2021, 12:42
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Hey TomAaaow,
I will only answer parts of your last post since Jiaz is the expert regarding JDownloaders core functionality so please don't get mad at me if I'm *ignoring* parts of your reply.

Quote:
Originally Posted by TomArrow View Post
They can be in the download list for months without ever progressing past an ever-updating but ultimately never meaningfully changing error/warning message. It can be stuff like links/streams that expire, all kinds of weird stuff.
You should report such cases to us so we can see what we can do about it.
Nevertheless our system will never be perfect because the download part relis on plugins which are mostly made for specific versions of websites which can change at any time and edge cases we don't know about may also happen at any time.

Quote:
Originally Posted by TomArrow View Post
Because let's be real, the vast majority of entries (finished downloads/older entries) never change, but they can still be useful for archival purposes, like finding out if you already downloaded a file in the past). Rewriting those every time is wasteful. It's really just a small percentage of entries that regularly get updated in some form.
Yes but they can still be changed by the user so you'd at least have to check for "has it changed?".

Quote:
Originally Posted by TomArrow View Post
If I just double the amount of files in my list, it will completely crash the software. It will begin shuffling around memory and hang/freeze itself, with the only way out being to change Java settings to allow it to eat even more RAM. Until I arrive at the point where all my computer's RAM is used solely to maintain jD2 which seems silly... and then to pray and hit Cleanup, deleting the entire history.
JD has never been designed to use the downloadlist as a history.
Unfortunately this is something that quite some users want to do...
Search the forum for "download history" and you should be able to find a lot of similar threads/ideas and even some EventScripter scripts for a kind of "download history" feature.

I did read your post and I know that you'Re already short of memory but I got an idea for a workaround you could use to at least prevent JD from writing so much data to your SSD:
1. Get a RAM-Disk software.
2. Write some kind of script that activates the RAM-disk, moves JD in it and starts it.
3. Now make sure that when closing JD, all of your config/downloadlists or even the complete JD folder is moved back on HDD/SSD.

I'm sure there is something I've overlooked but in general I think this would be possible.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #23  
Old 30.09.2021, 20:33
TomArrow TomArrow is offline
Super Loader
 
Join Date: Sep 2017
Posts: 27
Default

@pspzockerscene

Thanks for your reply.

Oh yeah I'm aware it can never be perfect and that part wasn't actually meant as a criticism, just as a matter-of-fact. Some downloads are just gonna get stuck due to unpredictable issues, so it would be good if such cases didn't trigger constant rewrites of everything.

Indeed, they could still be changed by the user, but you could just keep the unchanged copy in the "archive file" and queue the changes in the "dynamic file" and when the time comes to "flush" the changes, they could be moved over to the archive file. Just an idea ofc.

I don't know if the EventScripter scripts (but maybe I didn't search hard enough) would do what I need. It's not about having a list of everything I downloaded for myself per se, it's more about being able to filter out already downloaded files with the "Already in Downloadlist" filter in the LinkCollector, which is pretty useful.

About the RAMdisk idea... yeah you said it yourself, I'm already short on memory. And jD2 is using up 4 GB of RAM as-is. I don't want to end up using half of my physical RAM for a download manager. I do a lot of video/photo work and rendering and I desperately want to relieve my system of undue stress if that makes sense, to free up more resources for crucial work.

Cheers
Reply With Quote
  #24  
Old 30.09.2021, 21:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@TomArrow: just short answer, sorry not much time right now.
Can you please note the number of the downloadList and linkcollector zip file before/after your tests, so we can trace which one is written how often? Also would be good to know in what time window this happens...in 1 hour, 1 day...your amount of written data is indeed too high, so our first goal is to find the cause of the many writes and where this comes from. Please don't be mad but it prefer to work through issues one by one and not all of them at once.
So next goal is to find out the cause of high write output and how your use of JDownloader is triggering/causing this and how much of write output is caused by downloadlist/linkgrabber list.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #25  
Old 30.09.2021, 23:43
TomArrow TomArrow is offline
Super Loader
 
Join Date: Sep 2017
Posts: 27
Default

@Jiaz

Sure, let's go through it piece by piece.

Here's screenshots from my cfg folder (they're just image links wrapped in the loglink thing, hope that works for you):

https://postimg.cc/bsqq9rgC
https://postimg.cc/Lgt66ZTf


Does that answer the question? The count of zip files hasn't changed I believe from back then, it was always 6-ish main files I think. I looked at a backup from 2020 and it's about the same, 6 files (7 for downloadlist) with date modified varying around 3 to 10 minutes between each, for downloadlist. For linklist in the old one, it's also around 3 to 10 minutes distance between the 5 recent files and about an hour between the 6th and 5th last. Not sure what all the backup zip files do.

So currently I think the download list is the main issue, with link list being about once an hour it seems now. But even so, once an hour: 12 active hours a day, times 240 MB ... 2.8 GB a day, 20 GB a week, 80 GB a month.

Last edited by raztoki; 01.10.2021 at 14:34. Reason: removed loglink
Reply With Quote
  #26  
Old 01.10.2021, 19:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@TomArrow: I'm sorry but I was not clear enough each of the download/linkcollector has a number which increases with every write. I need to know the difference of this number when you start JDownloader and then after time X stop JDownloader, so I know how many downloadlist and linkcollector files were written in time X

backup files are created when loading failed, so it can be imported/restored at later time, see https://support.jdownloader.org/Know...nkgrabber-list
from date of those files, I would say you can delete them
__________________
JD-Dev & Server-Admin
Reply With Quote
  #27  
Old 02.10.2021, 00:26
TomArrow TomArrow is offline
Super Loader
 
Join Date: Sep 2017
Posts: 27
Default

@Jiaz

Ah. Well I'm afraid there's no way for me to find out that old number, however I can make a guess. The last number I know is from March 31st 2021 around noon, that's when I did my last backup of the full Jd2 folder. The number for downloadlist back then was 1199508 for downloadlist and 47570 for the linkcollector. My PC runs pretty much 24/7 so taking the current number of 1520552 and 54590 (although falsified by the last 2 weeks or so), dividing the difference by 184 days, I'd say that's about 321k writes total, 1744 writes per day for downloadlist and 7020 writes total or 38 writes per day for the linkcollector. The downloadlist size at the time was around 50 MB (must have been after a cleanup). So, as a low estimate given 50 MB per write ... 16 TB of downloadlist writing since March 1st. And I've been running jd2 for years. So I'm guessing its likely over 100TB written in the past few years. Linkcollector was already around 170 MB back then, so that would make 1.19 TB as a low estimate.

I'm guessing the linkcollector is a proportionally much smaller issue then, but still way too high...

As for the past 14 days... hard to say I guess. But it's gotten slightly better as I said. Looking at it right now, the distance between the downloadlist saves is actually around half an hour. Probably because I'm actively downloading stuff and it's changing the state constantly, thus increasing the delay to the max.

Total writes since last jd2 restart (that was after changing those settings) are around 240GB. Subtract the around 25GB downloaded (IO read bytes), and it's around 215 GB written during the last 14 days I think.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 01:04.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.