JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 16.12.2020, 05:21
kimkayoung kimkayoung is offline
Ultra Loader
 
Join Date: Jul 2009
Posts: 49
Default Part of downloadlist is gone!

Part of my download list is gone! I tried to recover it from the cfg folder, but other than the backups made today, the earliest backup I can go back to is one that was made in early November.

(I deleted quite a few of the old links in the download list, but I don't think it should have any influence on the integrity of the download list.)

It appears that my linkcollector is fine (although I cannot be 100% sure). Can you tell me why this happened, and how I can restore my download list?

Here is the logs that I think may be relevant:

16.12.20 11.01.56 <--> 16.12.20 10.52.17 jdlog://3683525302851/

Many thanks in advance.

kimkayoung
Reply With Quote
  #2  
Old 16.12.2020, 11:40
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Check your cfg folder for downloadListXXX.zip files. You can restore previous state of linklist by drag/drop or loading the file and JDownloader either will load into downloadList or add to linkgrabber list.

"Part of the downloadlist is gone" sounds like you've deleted links by accident because if the the file is broken or JDownloader cannot load the file, then it's all or nothing.

Whenever this happens (eg deleted by accident), just close JDownloader, backup the downloadListXXX or linkcollectorXXX files from cfg folder and then import them again and restore the missing entries
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 16.12.2020, 11:40
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

see https://support.jdownloader.org/Know...nkgrabber-list
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 16.12.2020, 13:33
kimkayoung kimkayoung is offline
Ultra Loader
 
Join Date: Jul 2009
Posts: 49
Default

Quote:
Originally Posted by Jiaz View Post
see **External links are only visible to Support Staff**...
Dear Jiaz,

Thank you very much for your prompt and detailed support.

I haven't tried all the steps you pointed me to yet, but was puzzled to realize that the earliest downloadlist file that I see before I realized the problem was saved on the Nov. 15 this year. In other words, the saved download list files are dated as follows:

downloadList867597.zip (around 10:51 AM, 12/16/2020)

downloadList851745.zip (around 12:45 PM, 11/15/2020)

I was hoping that I could be able to locate a downloadlist backup file around yesterday or the day before yesterday, but I don't see one in the cfg folder, and the downloadList867597.zip seems to be already damaged.

So, my question is: Is there any way that I can fine one of these between today and 11/15? (or why don't I see a backup file between today and 11/15?)

Many thanks to your support in advance!

kimkayoung
Reply With Quote
  #5  
Old 16.12.2020, 14:21
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Are you sure you're looking at the correct folder? Because what you describe would mean you have not used JDownloader at all since nov15 and dec 16? JDownloader writes a complete new file on every change (after some delay, CopyOnWrite) so there must be many more files
In doubt, try a undelete tool in the cfg folder of JDownloader
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 17.12.2020, 20:13
kimkayoung kimkayoung is offline
Ultra Loader
 
Join Date: Jul 2009
Posts: 49
Default

Dear Jiaz,

First of all, thank you very much for your prompt support as always.

Yes, I'm sure that I'm looking at the correct folder. It appears that my JD automatically saves its download list and link collector for a span of a day or so, but not beyond that, as the screen capture below shows.

Also, I don't see an undelete tool in the cfg folder. Can you please tell me what the tool is called?

Many thanks in advance as always.

kimkayoung



---
Reply With Quote
  #7  
Old 18.12.2020, 10:42
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@kimkayoung: by default it saves list 5 secs minimum, maximum 60 secs after something changes. Check Settings->Advanced Settings->
DownloadController.maximumsavedelay
DownloadController.minimumsavedelay
In case you have not modified those values then something with your Filesystem/System seems not okay. You should find default 5 (GeneralSettings.keepxoldlists) files for downloadList and linkcollector
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 18.12.2020, 10:43
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by kimkayoung View Post
Also, I don't see an undelete tool in the cfg folder. Can you please tell me what the tool is called?
I mean any undelete tool for windows in order to undelete previous existing downloadListXXX files
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 02.03.2022, 19:53
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

I have this problem today with my linkgrabber list. A few links are missing. I'm going to follow the instructions here to fix it hopefully but do I load the one that says backup after it or the one without.

Ex
linkcollectorXX.zip.backup
Or
linkcollectorXX.zip.
Reply With Quote
  #10  
Old 03.03.2022, 16:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@Matts76: whenever something fails with loading the list, JDownloader will create a .backup file of it in cfg folder.
So the .backup is the correct one, see
https://support.jdownloader.org/Know...nkgrabber-list
In case this happens more often, you should check your configuration to allow more memory. This can easily happen for large lists and not well optimized/configured memory settings for java, see
https://support.jdownloader.org/Know...vmoptions-file
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 03.03.2022, 18:01
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

Yes it later occurred to me this is a memory issue. I'll be on my computer again later so I'll increase the memory and that should work. I have some very big folders in there so that must be the problem. Thank you. Can you recommend the best settings?

Last edited by Matts76; 03.03.2022 at 18:13.
Reply With Quote
  #12  
Old 03.03.2022, 19:53
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@Matts76: how much memory does your system have? how large is the list? just wild estimation?
you should use 64bit installer from here https://jdownloader.org/jdownloader2 to support more than 2GB max memory
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 03.03.2022, 21:01
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

I'm not too sure on how much memory it has but now that I think about it the list has quite alot like multiple folders in some links etc. I just gave it as much as it could handle now I think. I'll try to up the memory on this one but if it's not enough I'll do the 64bit option.
Reply With Quote
  #14  
Old 04.03.2022, 12:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@Matts76: can you provide a screenshot of the about dialog?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 04.03.2022, 18:41
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

When I'm home from work later this afternoon I'll post a screenshot.

I tried the 62bit one and when it asks if I want to import the links I say yes but it shows less than what the 32bit original program is showing. I did start removing links from the one I normally work with. There's alot that I don't need in there. It's still not showing those other packages but I still have the original links if all else fails I'll just load them again. Any other things I could try if needed?
Reply With Quote
  #16  
Old 04.03.2022, 19:54
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@Matts76: What do you mean by *it shows less than*?
and "It's still not showing those other packages"?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 04.03.2022, 20:32
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

In the 62bit one it's only showing half of my list of links in linkgrabber. In the 32bit one I normally use it's still not showing those packages that aren't showing up the past couple days are still not appearing but in the list of hosters on the side it looks like they are there judging by the numbers.
Reply With Quote
  #18  
Old 04.03.2022, 20:38
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@Matts76: what do you mean by *half of your list* either the list is complete or empty. JDownloader does not *half import/load* the list. It loads it all or nothing(and creates a backup file)
Are there .backup files in your cfg folder? without any rescue of yourself, JDownloader will not load the .backup files automatically.
Maybe you can provide screenshots to explain what you mean?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #19  
Old 04.03.2022, 22:04
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

That's what I figure too. I have the backup files. I use load link contaner and it says do I want to load those links. I confirm but when it loads many are missing. Maybe I have to set a bigger memory number for the 64bit one. Now on the one I normally use, 32bit, I'm still working on cleaning up links in linkgrabber so I'm thinking it should show those links that are missing once I delete more? I can take screenshots of that and the about me info when I'm home later so you can see that when you get a chance.
Reply With Quote
  #20  
Old 05.03.2022, 02:18
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

I uploaded a screenshot of the about me info. I thought once I deleted some links in linkgrabber that the rest of the list would show up automatically. Do I have to use the load linkgrabber option again to load the file that i know is complete? If possible i can send the backup file to you to see if you get the full list. If you screenshot the last few links I would know.
Attached Thumbnails
Screenshot_20220304-191341_Gallery.jpg  
Reply With Quote
  #21  
Old 06.03.2022, 04:12
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

Here's the screenshot of the About info in the 64bit version. My memory on my computer is showing 3.5g available. I went into the vmoptions file and typed =Xmx2g and nothing else. Is there something more I'm supposed to add? I tried to load the linkgrabber list again and it's still only loading maybe half of the list. I also notice some lag time when setting the directory for downloads in linkgrabber and when starting downloads from there.
Attached Thumbnails
Screenshot_20220305-210649_Gallery.jpg  

Last edited by Matts76; 06.03.2022 at 10:18.
Reply With Quote
  #22  
Old 07.03.2022, 16:15
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@Matts76: Thanks for the screenshot. The first screenshot clearly shows that your JDownloader already runs at max memory. Your 64bit Installation shows that max 1.7GB is possible and JDownloader is near its limit.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #23  
Old 07.03.2022, 16:18
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by Matts76 View Post
I tried to load the linkgrabber list again and it's still only loading maybe half of the list.
Are you importing the .backup file or what do you do?
What do you mean by *half of the list*? I already explained that either JDownloader add the complete list or creates .backup file and tries next one list. It's not even supported to load part of the list.
So the question is what you do mean by *half of the list*? maybe it's older version of the list where the missing packages are just not there?

If you want we can take a look at it together via Teamviewer, just contact me via support@jdownloader.org
__________________
JD-Dev & Server-Admin
Reply With Quote
  #24  
Old 07.03.2022, 18:38
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

I'm importing the .backup file but I don't know why it's only showing links from the beginning of the list to the middle of the list. The backup files are from the period when those links were there. Do my settings look correct from what you see? I have a backup list of the original links so I could just reload them like that.
Reply With Quote
  #25  
Old 07.03.2022, 19:19
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@Matts76: can you upload it to mega and send me link to support@jdownloader.org
OR we do a teamviewer session and check it together.

I would like to check if , for any reason, the loading might fail and check how many links are imported for me. I'm of course open for Teamviewer

Your about dialog looks good, so either the .backup file is already incomplete because not old enough or maybe there is indeed a bug while loading the list.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #26  
Old 07.03.2022, 19:44
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

Due to the time zone I think it's hard to do a Teamviewer session. I'm at work now so I don't have access but I can send you the backup files. Would a Wetransfer link be ok? I would know if it's complete by the last files you would see on the list.
Reply With Quote
  #27  
Old 07.03.2022, 20:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@Matts76: of course wetransfer would be fine too. Any *easy* sharing hoster and send link to support@jdownloader.org
I'm from germany.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #28  
Old 07.03.2022, 20:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by Matts76 View Post
I would know if it's complete by the last files you would see on the list.
I will test the file and check for errors and tell you number of packages/links and last link
__________________
JD-Dev & Server-Admin
Reply With Quote
  #29  
Old 07.03.2022, 22:42
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

Great. I'll send them later this evening to that email and you can check at your leisure.
Reply With Quote
  #30  
Old 08.03.2022, 11:30
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

I got them. Will check and get back to you asap
__________________
JD-Dev & Server-Admin
Reply With Quote
  #31  
Old 08.03.2022, 20:25
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

Thank you. I'll be waiting.
Reply With Quote
  #32  
Old 08.03.2022, 20:53
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@Matts76: okay, this zip file is broken/incomplete
Quote:
java.io.EOFException: Unexpected end of ZLIB input stream
I could restore 3977 packages with 96504 files
You could try following:
1.) filesystem check,maybe it fixes the incomplete file
2.) test so called undelete tools and scan your cfg folder for older(deleted) linkcollectorXXX.zip files that might be complete
__________________
JD-Dev & Server-Admin
Reply With Quote
  #33  
Old 09.03.2022, 09:58
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

Thank you for looking into it. I do have about four more backup files prior to those. I could see if those work. Just curious what was the last file you saw on the list?
Reply With Quote
  #34  
Old 09.03.2022, 12:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@Matts76: Package name contains box and 701-8, last filename 703b
__________________
JD-Dev & Server-Admin
Reply With Quote
  #35  
Old 09.03.2022, 21:57
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

Thank you. It seems like you're showing possibly the whole list. Can you tell me anything more about that last file? Do you know what might cause these files to be broken like that?

Last edited by Matts76; 09.03.2022 at 23:29.
Reply With Quote
  #36  
Old 10.03.2022, 14:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@Matts76: you can check yourself.
just clear the linkgrabber list (you can backup the current one by just create a copy of the latest linkcollectorXXX.zip file, and later can import it again) and drag/drop-import linkcollector60596.zip.backup
overview panel shows "3977 packages with 96504 files". If it's less for the same file, then we have to find out why
__________________
JD-Dev & Server-Admin
Reply With Quote
  #37  
Old 10.03.2022, 14:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by Matts76 View Post
Do you know what might cause these files to be broken like that?
Many different possible causes, eg JDownloader crashed/killed during writing the list. Windows not having fully flushed buffers to disk and computer crashed. filesystem/hardware issues
__________________
JD-Dev & Server-Admin
Reply With Quote
  #38  
Old 10.03.2022, 20:20
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

Thank you I'll do that later. I'll let you know what happens
Reply With Quote
  #39  
Old 11.03.2022, 09:17
Matts76 Matts76 is offline
JD Addict
 
Join Date: Feb 2012
Posts: 168
Default

I got the same thing as you with that file. That's most of the list so I'm satisfied. Thanks for all your help with this.
Reply With Quote
  #40  
Old 11.03.2022, 11:21
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@Matts76: Sorry that I could not help to recover more :( But I already working on some ideas for different type of storage that can *survive* more dataloss
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 07:29.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.