JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #21  
Old 30.09.2021, 07:10
TomArrow TomArrow is offline
Super Loader
 
Join Date: Sep 2017
Posts: 25
Default

Hey,

sorry for the late reply but I wanted to give it a fair shake with the new settings on both linkcollector and downloadlist. So I've had it running non-stop since last time with the higher delays and while it's definitely better, it's still catastrophical imo:

Actual data downloaded during this period was maybe 25 GB. Total bytes written to disk around 215 GB. So just the amount of data written to the hard drive from config files was almost 10 times that of the actual data downloaded. Just crazy!

Performance has become a bit better but still pretty bad considering it's just a download manager. Still occasionally get those hickups where it just takes forever for the GUI to react, but better than before overall. CPU use still gets ridiculously high sometimes.

Thanks for your explanation about the delay values.

To your questions:

My use case is basically, jd2 runs in the background constantly with clipboard monitoring on. I'm on my PC many hours a day and when I see something I like (Youtube video, some file sharing link, tweet, whatever), I just Ctrl+C it so it gets added to the list and I can forget about it. I also regularly add large amounts of downloads to the downloadlist that as a whole will take weeks to complete. To give an example, it might be 50 downloads from filehosters, and we both now how limiting they are sometimes. So basically all that just keeps running in the background, I imagine constantly checking and refreshing statuses and seeing if anything new can be done, if there are any server timeouts active etc. Then some files might hit some kind of server error due to glitches and just never finish, but I have so many downloads that I just don't notice until much later. They can be in the download list for months without ever progressing past an ever-updating but ultimately never meaningfully changing error/warning message. It can be stuff like links/streams that expire, all kinds of weird stuff.

About the buffer size thingie ... I'm not sure what that means tbh. Can you elaborate?

In any case thanks for being so accommodating, I'm genuinely surprised that you're actually giving this thought. Much appreciated and a rare experience. Thank you.

Maybe my SQLite idea isn't ideal as you say, but I hope something can be figured out. Some other ideas I'm just gonna wildly throw in, perhaps uselessly: How about splitting the list by date and/or activity? Having all new stuff and all active/non-finished stuff in its own file, and maybe splitting by date, so that every day of a month has its own file, and then at the end of the month they get merged in one big swoop? For example, here's some stuff I think could be just part of a "big file" that only gets touched once a month: All finished downloads. Also let's say, all link collector stuff that's been added prior to 1 week before now, because the likelihood of that being touched is rather low if it wasn't touched already. The odd/rare changes to these like deletes could be queued in the smaller files maybe? Because let's be real, the vast majority of entries (finished downloads/older entries) never change, but they can still be useful for archival purposes, like finding out if you already downloaded a file in the past). Rewriting those every time is wasteful. It's really just a small percentage of entries that regularly get updated in some form.

Also I've recently come across this fancy thing called zlib, for which you can actually create a dictionary file using a training dataset and in my experiments it has almost rivalled 7zip's LZMA2 in some cases. Though admittedly, at the highest compression level it's likely too slow to compress, but maybe it has some useful properties at the faster modes.

If you have better ideas than these, that's perfectly fine, I won't be too hurt in my pride. I'm just happy at the thought that these issues might be resolved some day.

As a side note, it would be really cool if something could be done about the RAM usage too. It's not quite as problematic as the other issues right now, but I've also encountered the situation in the past where I just had too many entries and even with my 16 GB of RAM, stuff just wouldn't work properly anymore. Even as we speak, I have 3 GB of RAM usage and over 5 GB committed. If I just double the amount of files in my list, it will completely crash the software. It will begin shuffling around memory and hang/freeze itself, with the only way out being to change Java settings to allow it to eat even more RAM. Until I arrive at the point where all my computer's RAM is used solely to maintain jD2 which seems silly... and then to pray and hit Cleanup, deleting the entire history. This sounds like a horror scenario but I've actually been there at least twice. After all the goal for me would be to ideally have millions of files in the download list that are already finished, just so I know not to redownload them.

Thinking of that ... how about making something like another tab like "archived downloads" or something, that is not kept in memory, but still used to check if a file is already downloaded? That would suit my needs in terms of archival while not cluttering the download list. Just a thought, haven't thought it through. Because that's pretty much the main reason I keep the old entries- to know if they're already downloaded. Maybe that's something SQLite could be good for? Could use hashes to tell if a file is already downloaded and those, unlike a full text search, could be searched very efficiently via index? idk if that makes sense with the logic needed and maybe its more complex than that, I'm just brainstorming.

Anyway, let me know what more information I can provide you with to help! Thanks!

Cheers
Reply With Quote
  #22  
Old 30.09.2021, 11:42
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,285
Default

Hey TomAaaow,
I will only answer parts of your last post since Jiaz is the expert regarding JDownloaders core functionality so please don't get mad at me if I'm *ignoring* parts of your reply.

Quote:
Originally Posted by TomArrow View Post
They can be in the download list for months without ever progressing past an ever-updating but ultimately never meaningfully changing error/warning message. It can be stuff like links/streams that expire, all kinds of weird stuff.
You should report such cases to us so we can see what we can do about it.
Nevertheless our system will never be perfect because the download part relis on plugins which are mostly made for specific versions of websites which can change at any time and edge cases we don't know about may also happen at any time.

Quote:
Originally Posted by TomArrow View Post
Because let's be real, the vast majority of entries (finished downloads/older entries) never change, but they can still be useful for archival purposes, like finding out if you already downloaded a file in the past). Rewriting those every time is wasteful. It's really just a small percentage of entries that regularly get updated in some form.
Yes but they can still be changed by the user so you'd at least have to check for "has it changed?".

Quote:
Originally Posted by TomArrow View Post
If I just double the amount of files in my list, it will completely crash the software. It will begin shuffling around memory and hang/freeze itself, with the only way out being to change Java settings to allow it to eat even more RAM. Until I arrive at the point where all my computer's RAM is used solely to maintain jD2 which seems silly... and then to pray and hit Cleanup, deleting the entire history.
JD has never been designed to use the downloadlist as a history.
Unfortunately this is something that quite some users want to do...
Search the forum for "download history" and you should be able to find a lot of similar threads/ideas and even some EventScripter scripts for a kind of "download history" feature.

I did read your post and I know that you'Re already short of memory but I got an idea for a workaround you could use to at least prevent JD from writing so much data to your SSD:
1. Get a RAM-Disk software.
2. Write some kind of script that activates the RAM-disk, moves JD in it and starts it.
3. Now make sure that when closing JD, all of your config/downloadlists or even the complete JD folder is moved back on HDD/SSD.

I'm sure there is something I've overlooked but in general I think this would be possible.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #23  
Old 30.09.2021, 19:33
TomArrow TomArrow is offline
Super Loader
 
Join Date: Sep 2017
Posts: 25
Default

@pspzockerscene

Thanks for your reply.

Oh yeah I'm aware it can never be perfect and that part wasn't actually meant as a criticism, just as a matter-of-fact. Some downloads are just gonna get stuck due to unpredictable issues, so it would be good if such cases didn't trigger constant rewrites of everything.

Indeed, they could still be changed by the user, but you could just keep the unchanged copy in the "archive file" and queue the changes in the "dynamic file" and when the time comes to "flush" the changes, they could be moved over to the archive file. Just an idea ofc.

I don't know if the EventScripter scripts (but maybe I didn't search hard enough) would do what I need. It's not about having a list of everything I downloaded for myself per se, it's more about being able to filter out already downloaded files with the "Already in Downloadlist" filter in the LinkCollector, which is pretty useful.

About the RAMdisk idea... yeah you said it yourself, I'm already short on memory. And jD2 is using up 4 GB of RAM as-is. I don't want to end up using half of my physical RAM for a download manager. I do a lot of video/photo work and rendering and I desperately want to relieve my system of undue stress if that makes sense, to free up more resources for crucial work.

Cheers
Reply With Quote
  #24  
Old 30.09.2021, 20:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,672
Default

@TomArrow: just short answer, sorry not much time right now.
Can you please note the number of the downloadList and linkcollector zip file before/after your tests, so we can trace which one is written how often? Also would be good to know in what time window this happens...in 1 hour, 1 day...your amount of written data is indeed too high, so our first goal is to find the cause of the many writes and where this comes from. Please don't be mad but it prefer to work through issues one by one and not all of them at once.
So next goal is to find out the cause of high write output and how your use of JDownloader is triggering/causing this and how much of write output is caused by downloadlist/linkgrabber list.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #25  
Old 30.09.2021, 22:43
TomArrow TomArrow is offline
Super Loader
 
Join Date: Sep 2017
Posts: 25
Default

@Jiaz

Sure, let's go through it piece by piece.

Here's screenshots from my cfg folder (they're just image links wrapped in the loglink thing, hope that works for you):

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**


Does that answer the question? The count of zip files hasn't changed I believe from back then, it was always 6-ish main files I think. I looked at a backup from 2020 and it's about the same, 6 files (7 for downloadlist) with date modified varying around 3 to 10 minutes between each, for downloadlist. For linklist in the old one, it's also around 3 to 10 minutes distance between the 5 recent files and about an hour between the 6th and 5th last. Not sure what all the backup zip files do.

So currently I think the download list is the main issue, with link list being about once an hour it seems now. But even so, once an hour: 12 active hours a day, times 240 MB ... 2.8 GB a day, 20 GB a week, 80 GB a month.

Last edited by raztoki; 01.10.2021 at 13:34. Reason: removed loglink
Reply With Quote
  #26  
Old 01.10.2021, 18:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,672
Default

@TomArrow: I'm sorry but I was not clear enough each of the download/linkcollector has a number which increases with every write. I need to know the difference of this number when you start JDownloader and then after time X stop JDownloader, so I know how many downloadlist and linkcollector files were written in time X

backup files are created when loading failed, so it can be imported/restored at later time, see https://support.jdownloader.org/Know...nkgrabber-list
from date of those files, I would say you can delete them
__________________
JD-Dev & Server-Admin
Reply With Quote
  #27  
Old 01.10.2021, 23:26
TomArrow TomArrow is offline
Super Loader
 
Join Date: Sep 2017
Posts: 25
Default

@Jiaz

Ah. Well I'm afraid there's no way for me to find out that old number, however I can make a guess. The last number I know is from March 31st 2021 around noon, that's when I did my last backup of the full Jd2 folder. The number for downloadlist back then was 1199508 for downloadlist and 47570 for the linkcollector. My PC runs pretty much 24/7 so taking the current number of 1520552 and 54590 (although falsified by the last 2 weeks or so), dividing the difference by 184 days, I'd say that's about 321k writes total, 1744 writes per day for downloadlist and 7020 writes total or 38 writes per day for the linkcollector. The downloadlist size at the time was around 50 MB (must have been after a cleanup). So, as a low estimate given 50 MB per write ... 16 TB of downloadlist writing since March 1st. And I've been running jd2 for years. So I'm guessing its likely over 100TB written in the past few years. Linkcollector was already around 170 MB back then, so that would make 1.19 TB as a low estimate.

I'm guessing the linkcollector is a proportionally much smaller issue then, but still way too high...

As for the past 14 days... hard to say I guess. But it's gotten slightly better as I said. Looking at it right now, the distance between the downloadlist saves is actually around half an hour. Probably because I'm actively downloading stuff and it's changing the state constantly, thus increasing the delay to the max.

Total writes since last jd2 restart (that was after changing those settings) are around 240GB. Subtract the around 25GB downloaded (IO read bytes), and it's around 215 GB written during the last 14 days I think.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 21:48.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.