[IanMatthysen]

Hi!

I've recently formatted my PC and reinstalled my software (including JDownloader 2). Before that, I did make a backup of the cfg and logs folders. When I replaced the contents of those folders, the Link Grabber list restored successfully, but not so with the list of downloads.

I Google'd and consulted the forum posts for a solution, but I was unable to resolve the issue.

My suspicion is that it might have something to do with the size of the downloadListXXXXXXX.zip file. Currently it's about 414 MB, and it might be possible that JDownloader cannot handle such large .zip files.

Are there alternative ways in which I can restore my download list?

Any responses to my query will be greatly appreciated! Please let me know if you require more details.

Have a great day!

Ian

[pspzockerscene]

Hi,
please post a screenshot of: Help -> About JDownloader

Please try to import your downloadlistXXX.zip file via File -> Load Linkcontainer and tell us what happens.

[pspzockerscene]

Update:
Most likely your JVM does't get enough RAM to load that downloadliste.
Here are your options:
1. Change the xmx parameter in your .vmoptions file: Instructions
or:
2. Re-install JDownloader via jdownloader.org/jdownloader2 which should then auto use correct xmx values.
Afterwards you should be able to import that big downloadlistXXX.zip file.

Are you using the JDownloader downloadlist as a backup/history or why is your file so big?

[IanMatthysen]

Hi!

Thank you so much for the replies! I really do appreciate it.

As requested in your previous reply, I attached a screenshot of my Help -> About JDownloader window. The image file is named JDownloader About.png.

About your second question (Please try to import your downloadlistXXX.zip file via File -> Load Linkcontainer), here is what happens:
1. JDownloader appears to become unresponsive, but I just let it "do it's thing". After about 10 minutes, the becomes responsive again, but the download list is still empty.
2. I also had a look in the log files (particularly the jd.controlling.downloadcontroller.DownloadController.log.0) after this to see if there are any errors logged. This is the contents of the file:

------------------------Thread: 52:jd.controlling.downloadcontroller.DownloadController.log-----------------------
--ID:52TS:1668459268621-2022/11/14 10:54:28 PM - [jd.controlling.downloadcontroller.DownloadController$7(run)] -> Init DownloadList
--ID:52TS:1668459268621-2022/11/14 10:54:28 PM - [jd.controlling.downloadcontroller.DownloadController(findAvailableDownloadLists)] -> Collect Lists
--ID:52TS:1668459268623-2022/11/14 10:54:28 PM - [jd.controlling.downloadcontroller.DownloadController(findAvailableDownloadLists)] -> Lists: [C:\Users\Ian\AppData\Local\JDownloader 2.0\cfg\downloadList1.zip, C:\Users\Ian\AppData\Local\JDownloader 2.0\cfg\downloadList.zip]
--ID:52TS:1668459268623-2022/11/14 10:54:28 PM - [jd.controlling.downloadcontroller.DownloadController(loadFile)] -> Load List: C:\Users\Ian\AppData\Local\JDownloader 2.0\cfg\downloadList1.zip
--ID:52TS:1668459268627-2022/11/14 10:54:28 PM - [jd.controlling.downloadcontroller.DownloadController$7(run)] -> LinkList found: 0/0
------------------------Thread: 159:jd.controlling.downloadcontroller.DownloadController.log-----------------------
--ID:159TS:1668459323866-2022/11/14 10:55:23 PM - [jd.controlling.downloadcontroller.DownloadController(loadFile)] -> Load List: D:\Game Backups\Program Files (x86)_JDownloader_cfg\cfg\downloadList1894058.zip
--ID:159TS:1668460433502-2022/11/14 11:13:53 PM - [] -> Exception thrown at jd.controlling.downloadcontroller.DownloadController.loadFile(DownloadController.java:778):
java.lang.OutOfMemoryError: Java heap space

--ID:159TS:1668460436014-2022/11/14 11:13:56 PM - [jd.controlling.downloadcontroller.DownloadController(loadFile)] -> Entry:18543_1695|EntryIndex:212020|Size:-1|Compressed Size:-1


Yes, I'm running Windows 7, but I tried this on my Windows 10 laptop as well, but I got the same result on my Windows 10 laptop as well. Both PCs have 16GB RAM.

With regards to your second reply, I did not yet have the chance to follow your recommendations, but I will do so as soon as I can and let you know what happens. But to answer your last question ("Are you using the JDownloader downloadlist as a backup/history or why is your file so big?"). You could say I'm using it as a history to prevent me from downloading the same files over and over. I sometimes download Instagram posts, and sometimes there could be 1000+ posts, so it's handy if I only download the newer posts.

Again, thank you so much for your assistance so far.

Kind regards

Ian

[pspzockerscene]

The snippet of your log confirm my thoughts that your JDownloader is running out of memory.
After following my instructions that should work fine again but keep in mind that JDownloaders downloadlist is not ment to be used as a history so you should clean it after a while or you might run into this issue again in some time.

[IanMatthysen]

Thanks again for the reply. I really do appreciate it.

What you said does make sense. If the downloadlist.zip files get too big, it will eventually become a problem. I think I might have to develop my own SQL Server database with a Visual Studio WPF application to store my download list!

I think I should also mention the other things I attempted in order to restore my download list. I tried splitting up the downloadlistXXX.zip file in to 5 smaller files), and then the following:
(1) I copied the .zip files to the cfg folder, hoping that JDownloader will load the contents of each .zip file one at a time, but it did not happen. It loaded the contents of the downloadlistXXX.zip with the smallest XXX numeric value, and renamed the remainder of the downloadlistXXX.zip files by adding a .backup to the extension of the file names.
(2) I tried using File -> Load Linkcontainer approach. I was able to add the contents of downloadList1.zip file, but when I try to add downloadList2.zip, nothing happens.

Could it be that I'm doing something wrong in either of these approaches?
(1) Could it be that the structure of the .zip files is wrong? I always make sure that the extraInfo file is present in each downloadlistXXX.zip file.
(2) Could it be that the file names themselves are incorrect? For example, downloadList1.zip file might contain the files 00000 through 09999_99, and downloadList2.zip might contain 10000 through 19999_99, etc. Should each downloadlistXXX.zip file's file names start at 00000 again?

Again, thanks so much for the assistance and insights into this issue!

[Jiaz]

Originally Posted by IanMatthysen

(2) Could it be that the file names themselves are incorrect? For example, downloadList1.zip file might contain the files 00000 through 09999_99, and downloadList2.zip might contain 10000 through 19999_99, etc. Should each downloadlistXXX.zip file's file names start at 00000 again?

I don't have the full overview right now, have to check but sounds reasonable

[Jiaz]

@IanMatthysen: I will add the custom .vmoptions to created backup files in future. For now on, I would recommend to install 64bit JDownloader Java bundle, see https://jdownloader.org/jdownloader2
(you already do) and also have to customize the vmoption file, see
https://support.jdownloader.org/Know...vmoptions-file
with 16GB of system memory, I would recommend to use

-Xmx4g

to allow JD up to 4gb of memory.
Do not try to workaround the issue with split/other ways in the end your JDownloader needs enough memory in order to load/hold/work with such large list.
We've got some new optimizations in pipeline to further reduce memory size of list/list entries.

Just set a good/high enough max heap value via the .vmoptions file and JDownloader should be fine to load the list.

[Jiaz]

Originally Posted by IanMatthysen

My suspicion is that it might have something to do with the size of the downloadListXXXXXXX.zip file. Currently it's about 414 MB, and it might be possible that JDownloader cannot handle such large .zip files.

Just asking and of course I understand if you dont want to but maybe you can provide such large *real world* file to me so I can analyze and further try to optimize memory consumption. Eg upload to wetransfer or anywhere else and send link to support@jdownloader.org
Real large lists are always better as dummy/fake ones as they don't reflect *real work usage*.

[IanMatthysen]

Hi there pspzockerscene and Jiaz

Apologies for only replying now. If it wasn't work keeping me busy, the flu pounced me... less than 2 months before Christmas!

OK, to reply to your messages:

Originally Posted by Jiaz

@IanMatthysen: I will add the custom .vmoptions to created backup files in future. For now on, I would recommend to install 64bit JDownloader Java bundle, see **External links are only visible to Support Staff**...
(you already do) and also have to customize the vmoption file, see
**External links are only visible to Support Staff**...
with 16GB of system memory, I would recommend to use

to allow JD up to 4gb of memory.
Do not try to workaround the issue with split/other ways in the end your JDownloader needs enough memory in order to load/hold/work with such large list.
We've got some new optimizations in pipeline to further reduce memory size of list/list entries.

Just set a good/high enough max heap value via the .vmoptions file and JDownloader should be fine to load the list.

Thanks again for this valuable information I will definitely let you know what the outcome is. I'm not currently at my PC (where the problem is), but I'll get to it as soon as I can.

Originally Posted by Jiaz

I don't have the full overview right now, have to check but sounds reasonable

It would be very interesting to know!

Originally Posted by Jiaz

Just asking and of course I understand if you dont want to but maybe you can provide such large *real world* file to me so I can analyze and further try to optimize memory consumption. Eg upload to wetransfer or anywhere else and send link to support@jdownloader.org
Real large lists are always better as dummy/fake ones as they don't reflect *real work usage*.

Of course. I'd be more than happy to send you the downloadListXXXXXXX.zip file, because I know that the end result might possibly help improve JDownloader. I'll send it to the e-mail address you supplied at the earliest convenience.

A possible feature suggestion: How about adding a 3rd tab labelled "Archive" where all the completed downloads are kept? That way, the download list can be kept clean.

I'm actually giving it serious thought developing my own C#/WPF app with a SQL Server database. It should be fairly simple to import the files inside downloadListXXXXXXX.zip, because it's stored as JSON.

[Jiaz]

Originally Posted by IanMatthysen

Apologies for only replying now. If it wasn't work keeping me busy, the flu pounced me... less than 2 months before Christmas!

Get well soon!

Originally Posted by IanMatthysen

It would be very interesting to know!

It *should* work as expected. I guess it failed because of *out of memory* issue by limited max heap.

Originally Posted by IanMatthysen

A possible feature suggestion: How about adding a 3rd tab labelled "Archive" where all the completed downloads are kept? That way, the download list can be kept clean.

That's our long termin idea for a history/archive but this should be done with a proper database in backend, else there is no real difference in keeping the internal link class in list X or list Y, still
uses the same amount of memory. An archive/history item should be stored in sort of database that supports the required methods like search/find/contains.
(not reachable by public at the moment)
Idea is to use a database that provides rest api support and then JDownloader can store information in there and use/search/modify when required.

Originally Posted by IanMatthysen

I'm actually giving it serious thought developing my own C#/WPF app with a SQL Server database. It should be fairly simple to import the files inside downloadListXXXXXXX.zip, because it's stored as JSON.

Do not rely on the downloadList format as it might change in future. I would go for rest api and use eventscripter in JDownloader to talk to it

[IanMatthysen]

Hello again gentlemen!

OK, I'll start by providing you with an update on my attempt to restore my massive 414MB downloadList.zip file. Unfortunately I have not been successful. If I allocate anything more than 3GB in the .vmoptions file, I get an error message when JDownloader starts: "The JVM could not be started. The maximum heap size (-Xmx) might be too large or an antivirus or firewall tool could block the execution.".

So did some digging and tried a few things:
1. Installing the latest version of Java and restarting my PC.
2. Disabling my antivirus & firewall for the time being (I use Norton). I haven't spoken to their tech support people as yet, but I think I might.
3. Creating a new environmental variable:
3.1. Right-click on "Computer" (or "This PC") and select "Properties".
3.2. "Advanced System Settings"
3.3. On the pop-up window, select the "Advanced" tab, and click the "Environment Variables" button.
3.4. Under "System variables", click the "New..." button.
3.5. In the "Variable name" text field, type "_JAVA_OPTIONS" and in the "Variable value" text field, type "-Xmx4g"
3.6. Restart PC.
4. Changing the Java settings in the Windows Control Panel
4.1. Go to Control Panel
4.2. Click the Java icon.
4.3. On the pop-up window, click the "Java" tab, and then click the "View..." button.
4.4. Under the "User" tab locate the latest installed version of Java (if there are more than 1 Java installation).
4.5. The "Runtime Parameters" value should be blank. Type "-Xmx4g" in there, and click the "OK" button.
4.6. Restart the PC (just in case).

Sadly none of these attempts worked either. And if I allocate anything between 2GB and 3GB in the .vmoptions file, JDownloader does not start at all. No logs are created in the logs folder and nothing shows up in the Windows Event Viewer.

Apologies for the long e-mail, but I think these hints might help someone else who's stuck in my position.

However, if you have any other recommendations of things I can try, please let me know.

About your idea of using a database to store the download information, I think that's a great idea. I think MySQL would work great, because it's platform-independent (at least, I think it is). I'm just not sure if it supports advanced features like stored procedures and SQL functions like Microsoft SQL Server does, but there are ways around that.

If you want, I can start looking into that during the December holidays. I suppose I'll just need to know the various aspects of the downloads that need to be stored (i.e. the URL, the date added, the status (queue, in progress, etc), the total bytes of the download, etc). If such documentation already exists, I'd like to have a look at it.

[Jiaz]

Originally Posted by IanMatthysen

So did some digging and tried a few things:
3.5. In the "Variable name" text field, type "_JAVA_OPTIONS" and in the "Variable value" text field, type "-Xmx4g"
4.5. The "Runtime Parameters" value should be blank. Type "-Xmx4g" in there, and click the "OK" button.t.

Please undo those changes With 64bit JVM you can just specify max heap via vmoption file

[Jiaz]

@IanMatthysen:

Originally Posted by IanMatthysen

And if I allocate anything between 2GB and 3GB in the .vmoptions file, JDownloader does not start at all.

sounds like you're still using a 32bit JVM?
Have you made sure to replace your JDownloader installation with one that bundles Java 64bit?
see https://board.jdownloader.org/showpo...05&postcount=7

I've added this piece of information to our help article
https://support.jdownloader.org/Know...vmoptions-file

[pspzockerscene]

Originally Posted by IanMatthysen

I suppose the cause behind me downloading and installing the wrong version of JDownloader was because there was only 1 option to choose from: "Download 32/64Bit Webinstaller (Java 1.8)". I'm suspect it has something to do with the fact that I'm still using Windows 7, and that the available download options are shown according to the operating system that is installed.

That installer should auto choose the best Java version.

Originally Posted by IanMatthysen

Just a quick follow-up: Have you received a copy of my 424MB downloadList.zip file which I sent via WeTransfer to support@jdownloader.org on 7 December 2022? Please let me know if you haven't, so I can send it again. I hope it will help your development team improve JDownloader!

Only Jiaz can do this and he is currently very busy.
I doubt that he will find time to look into this any time soon.

Originally Posted by IanMatthysen

On a completely unrelated topic: Is there somewhere on the JDownloader website where I can have a look at the schema of the JSON files (00000_00 - 99999_99) that make up downloadlistXXX.zip and linkcollectorXXX.zip?

No though you can obtain our source code from here:
https://support.jdownloader.org/Know...up-ide-eclipse

Originally Posted by IanMatthysen

Have a lovely day and week!

You too!
Thanks for your super nice feedback