JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 24.04.2016, 17:44
molitar molitar is offline
Mega Loader
 
Join Date: Jun 2010
Posts: 69
Default Skipped disk is ful but I have 62GB free!

I am getting the error skipped disk is full after the latest update. But that one single disk has 62GB free and the entire file is only 3.5GB approximately in size.
Reply With Quote
  #2  
Old 24.04.2016, 18:07
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

There has been no updates for 3 weeks, I doubt that any changes have been made in respects to free space either.

can you please post logs from that session
https://support.jdownloader.org/Know...d-session-logs

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #3  
Old 25.04.2016, 11:09
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Either you've setup JDownloader to use the wrong disk/folder ! (Please note that a changed default folder will only be used for new links. Existing links are not changed!)
Or the drive/folder is missing write permission for your user.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 05.05.2016, 13:06
freelee
Guest
 
Posts: n/a
Default Issue related to file extraction?

I have the same symptoms since several weeks. At least in my case the issue is not related to access rights, it is related to automatic extraction of large files. Based on my observations I have two guesses regarding the source of the issue.

Best guess: A long time ago JDownloader would have just filled a disk until no more space was available. Then the behaviour was changed. JD2 calculates whether after downloading from an URL a configurable amount of disk space will remain available, otherwise a download does not start. I really appreciate this improvement. What still could happen however was that a large extraction happening in parallel would eat up the free space while an URL was downloading. My guess is that more recently the team tried to prevent this also and JD2 now deducts the size of an archive during extraction when doing free space calculations. Because what I observe is that JD2 does no longer start a download, if free space is less than the (expected) size of a just started extraction.

Example: An archive consisting of 30x 1GB RARs starts auto-extracting after the last piece has been downloaded, the next download being only a 500MB does not start despite the disk still having 10GB free.

This wouldn't be bad as such, except for that in my case I download to a fast scratch disk (SSD) and extract to a hard drive, so it does not make sense to reserve the space on the SSD.

If my guess is correct, I would also suggest to think about not starting the extraction, if disk space becomes scarce, instead of stopping downloads. Alternatively in such circumstances downloads could be paused until the archive has been extracted and deleted, instead of being stopped.

My second guess is that heavy disk activity due to simultaneous extraction, eventually CRC checking and downloading leads to timeouts that are misinterpreted as "disk full" errors.
Reply With Quote
  #5  
Old 06.05.2016, 12:35
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@freelee: you are right. JDownloader reserves needed disk space during download and extraction.
Hugh IO should not result in "Disk Full", but maybe there is a bug. I need logs to check/fix that. We have not touched that code for months, so I don't think that IO is causing "Disk Full".

So you prefer Downloading over Extraction? In your example you want the 500 Mbyte file to be downloaded instead the archive to be extracted? If you want I can add some advanced settings/options so you can change behaviour. I could also add logic that auto retries downloads that aborted with "Disk Full" after successfull extraction. That would be best solution I guess

__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 08.05.2016, 03:40
freelee
Guest
 
Posts: n/a
Default

@Jiaz: I am quite convinced, that in my case the issue is simply that JD2 erroneously reserves the needed extraction space on the disk where the archive is, although I have configured the extraction path to another disk and so JD2 should look for space there. I started noticing unexpected download stops every now and then already months ago, but thought that it was some spurious issue. Recently however I had much more of them, so I started to investigate. Guess what, my scratch disk recently was much fuller than usual. I have made more space now (free space > 2x largest download) and the issues are gone.
Could you verify which disks the "reservation" calculation takes into account?

I think for people who extract to the same disk, your auto-retry proposal is the best solution as you mention. An additional setting/option would also be nice, but people might not find the setting, two code paths to test ...
Reply With Quote
  #7  
Old 09.05.2016, 14:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@freelee: disk space reservation does use the correct folder/drive. Works perfectly fine
Please provide logfiles the next time it happens https://support.jdownloader.org/Know...d-session-logs and post logID here
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 12.05.2016, 20:08
Statter Statter is offline
JD Legend
 
Join Date: Aug 2010
Posts: 541
Default

Just some thoughts not mentioned here that one may need to consider now and then also.
This may just be a Jdownloader issue, maybe not.

First where are you dling the files too? Same HD as app is on or another drive?

Second one might need to do a disk check/repair also especially if one is dling a lot or creating and moving files around with other apps also as sometimes the b-trees/disk space and file sizes can get confused at times as well as other other lower level disk reporting type items.

Third if one is using a lot of things requiring a lot of memory which may also include many apps open at once, often the disc space will be then used for swap memory. Also some apps may not release memory properly when not in use or closed, which can also play havoc with disc btrees etc. at times.

Fourth of course is this apps possible problem if it does not clear memory properly when jobs are done, or it does not clear the temp files needed and used when link grabbing, dling, exacting or converting password protected files, etc.

Lots of possibilities to be aware of as a user.
__________________
OS X !0.6.8 Mac Pro Intel (Workhorse)
OS X 10.13.6 MBP Intel 17" (Secondary)
OS X MBP Intel 15" Dual boot 10.6.8 and 10.13.6 (as needed)
Reply With Quote
  #9  
Old 15.05.2016, 14:43
freelee
Guest
 
Posts: n/a
Default

@Statter: Your points are all valid, however if you have read through the thread you have seen, that they do not apply in my case:
1. operating system and swap are on an OS disk
2. download is to a scratch disk
3. extraction is to an archive disk
4. download stops with disk full, if an extraction is going on and free space on download disk is less than expected size of archive being extracted.
5. if there is not enough space on the archive disk, extraction still starts and ultimately fails despite JD2 checking extraction free space (according to Jiaz, see above)

I think taken together these are strong indications that JD2 erroneously reserves free space for extraction on the download target disk instead of the extraction target disk. Also once extraction is finished, JD2 can resume downloads without any problems.

Therefore I asked Jiaz to check whether free space calculation really uses the correct disks in all cases. He confirmed and so I will wait until it happens again and I can provide logs or - if I find time - enforce the issue by filling the scratch disk artificially.
Note: all this refers to the latest JD2 release at the time I wrote the first reply, not the current one.

Last edited by freelee; 16.05.2016 at 22:34. Reason: typo
Reply With Quote
  #10  
Old 23.05.2016, 22:13
freelee
Guest
 
Posts: n/a
Default

@Jiaz:
Ok, here is the result of a controlled test showing the wrong disk space reservation.

System setup:
1. operating system and swap are on an OS disk with 23 GB free
2. download is to a scratch disk with 17 GB free (/media/ssd_in below)
3. extraction is to an archive disk with 322 GB free (/media/ssd_out)

First I have downloaded (with auto-extraction disabled) all parts of an archive with a size of 26GB. This is too much to extract to the download disk, but easily fits onto the extraction target disk.
Then I have restarted JD2 to keep the log small.
Next I have started extraction of the mentioned file.
Next I have added a small file of only 160 MB to the download list. Download started and stopped immediately with "Skipped - Disk is full" message, although 160 MB would fit easily on the available 17GB.

In other words, download stops with disk full, if an extraction is going on and free space on download disk is less than expected size of archive being extracted, although extraction is not to download disk.

I have also observed that if there is not enough space on the archive disk, extraction still starts and ultimately fails despite JD2 reserving space for extraction

I think, those two observations show, that JD2 reserves space for extraction on the wrong disk.

Some additional info:

$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 5.4G 23G 20% /
devtmpfs 427M 0 427M 0% /dev
tmpfs 432M 0 432M 0% /dev/shm
tmpfs 432M 6.5M 425M 2% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 432M 0 432M 0% /sys/fs/cgroup
/dev/sda1 110G 89G 17G 85% /media/ssd_in
/dev/mmcblk0p1 56M 21M 36M 37% /boot
tmpfs 87M 0 87M 0% /run/user/1001
/dev/sdb1 2.7T 2.3T 322G 88% /media/ssd_out

Operating System: Linux (LINUX 32Bit) (arm 32Bit)
Java: 1.8.0_65-b17 (32Bit) (headless), Oracle Corporation Java HotSpot(TM) Client VM
Heap 26.59 MB / 42.12 MB / 219.02 MB
Startup Time 23.05.16 20:31

Log id: jdlog://6954825891641

In case the above log does not contain all necessary info I have also captured stdout and stderr messages created by JD2 for this session in a text file.

BTW another issue is that since a few weeks the web interface in my case does mainly passthrough instead of direct connections. As you can see in the same log, there are a few direct connections every now and then. There is data exchanged in those (verified also with tcpdump), so it is clearly not a host firewall or access issue. It seems that the web interface does not like info (certificate?) it gets from JD2.

Hope this helps. Cheers.
Reply With Quote
  #11  
Old 24.05.2016, 15:54
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Found an issue with space calculation and mounts (unix based os). You were absolute right! Will be fixed with next core update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 24.05.2016, 15:55
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

About direct connection: do you use a fritzbox? please see here, https://support.jdownloader.org/Know...ode---fritzbox
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 25.05.2016, 01:08
freelee
Guest
 
Posts: n/a
Default

@Jiaz: No fritzbox, it's a Motorola
Every now and then there is a single direct tcp connection with real data exchange, some readable text pieces inside of the conversation:
192-168-1-76.mydns.jdownloader.org
GeoTrust Inc.1 0...U....RapidSSL SHA256
**External links are only visible to Support Staff****External links are only visible to Support Staff**
In other words, the browser app finds JD2, can communicate with it, but for some reason most of the time still uses the passthrough connection.

I also tried to see, whether it might be an issue similar to the fritzbox one, but I don't see any DNS requests trying to resolve for hosts in my subnet. On the JD2 host I see DNS requests for
update.appwork.org
api.jdownloader.org
update3.jdownloader.org
On the host running the browser I see DNS requests for
beta.jdownloader.org
api.jdownloader.org
gz.symcd.com
Using Google's DNS servers (bypassing the DNS in my router) mydns.jdownloader.org does not resolve at all. I also never see a request for that name. What is the name that would be used and resolve to a local address?
Reply With Quote
  #14  
Old 25.05.2016, 01:11
freelee
Guest
 
Posts: n/a
Default

If it is 192-168-1-76.mydns.jdownloader.org, this resolves correctly:

$ dig 192-168-1-76.mydns.jdownloader.org

; <<>> DiG 9.8.3-P1 <<>> 192-168-1-76.mydns.jdownloader.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6909
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;192-168-1-76.mydns.jdownloader.org. IN A

;; ANSWER SECTION:
192-168-1-76.mydns.jdownloader.org. 3600 IN A 192.168.1.76
Reply With Quote
  #15  
Old 25.05.2016, 01:26
freelee
Guest
 
Posts: n/a
Default

I did a quick test with JDAnywhere in the same subnet. This app does direct connections, in other words, the passthrough issue is limited to the browser app
Reply With Quote
  #16  
Old 25.05.2016, 01:31
freelee
Guest
 
Posts: n/a
Default

Next Test: From the same iOS device from which JDAnywhere does direct connections, Safari does passthrough connections, again pointing to the browser app.

On the desktop I also tried Firefox, same result: Passthrough
Reply With Quote
  #17  
Old 25.05.2016, 11:22
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@freelee: Apps work differently thats why direct connections work fine.
mydns.jdownloader.org domains are used for https within browser, because we need a wildcard domain to provide/resolv IPs to be able to serve https content.
google dns works fine for those domains for me

Can we do a teamviewer session so we can check this together. Just contact us via support@jdownloader.org
__________________
JD-Dev & Server-Admin
Reply With Quote
  #18  
Old 26.05.2016, 00:30
freelee
Guest
 
Posts: n/a
Default

@Jiaz: I will be away for a few days, but after I come back, we can do a team viewersession. What are your preferred hours?
Reply With Quote
  #19  
Old 26.05.2016, 12:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

You can reach us Mo-Fr from 9 to 17 o'clock (GMT+1 Germany)
just send us an e-mail. In meantime you can test the fixed diskspace stuff. Now the correct mountpoint/device is checked.
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 10:54.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.