JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 05.01.2021, 10:25
mikk88 mikk88 is offline
Junior Loader
 
Join Date: Dec 2020
Posts: 13
Default How to download single mega.nz file (without grabbing the whole folder)

Hello,

im trying to figure out how to download only parts of a mega.nz shared folder.

Example link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

contains:
\SharedFile.jpg
\SharedSubFolder\
\SharedSubFolder\SharedFileUpSideDown.jpg

file links:
SharedFile.jpg -> **External links are only visible to Support Staff****External links are only visible to Support Staff**
SharedFileUpSideDown.jpg -> **External links are only visible to Support Staff****External links are only visible to Support Staff**


No matter which link i add to jdownloader: linkgrabber is always parsing the whole mega.nz folder and ultimately finding both files for download.

If using the subfolder link:
**External links are only visible to Support Staff****External links are only visible to Support Staff** .. it works in the way that only SharedFileUpSideDown.jpg is found and downloaded.

Ultimately I want to add single files (or a folder + list of files to be excluded) for downloading during a crawljob later, but so far i can not even get it to work in the GUI.

Any ideas of what I might be doing wrong?

Thanks in advance!
Mikk
Reply With Quote
  #2  
Old 05.01.2021, 10:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,587
Default

wait for next plugin update, the url format for specifc file of folder was not yet supported
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 05.01.2021, 10:49
mikk88 mikk88 is offline
Junior Loader
 
Join Date: Dec 2020
Posts: 13
Default

ah great, I was wondering if I was doing something wrong as I thought it wouldn't be a rare use case to do.

Thanks in advance already for the update then!
Reply With Quote
  #4  
Old 05.01.2021, 11:04
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,587
Default

Update will be available in about 5 mins, please test and provide feedback
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 05.01.2021, 11:30
mikk88 mikk88 is offline
Junior Loader
 
Join Date: Dec 2020
Posts: 13
Default

seems to work fine now, thanks a lot for the super fast update!
Reply With Quote
  #6  
Old 05.01.2021, 11:35
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,587
Default

thanks for the feedback and you're welcome!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 06.01.2021, 18:41
mikk88 mikk88 is offline
Junior Loader
 
Join Date: Dec 2020
Posts: 13
Default

Hello again,
maybe I found a problem still with the update.

In the LinkGrabber crawling is very slow when adding a list of files instead of folders via a crawljob file.The crawling is extremely slow when the mega.nz folder from where the files are is huge (>TB). Every file to be found when crawling takes 10+ seconds while I can see constant high bandwidth usage.

I suspect the mega.nz account/folder content is not cached and reloaded on every new link (which can be huge data in big mega folders with 10s of thousands of files) .. which would explain the time needed (opening the folder in browser takes a long time too).

Or maybe is there a way to skip the online check and crawling for crawljob files completely? I know the files are there and alive .. I would not mind to add them to the download list blindly and deal with any problems on download only.
Reply With Quote
  #8  
Old 08.01.2021, 19:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,587
Default

There is currently no caching of folder information, so when you add several folder links from single folder, then JD will process them one by one and process the whole folder again.
Are you adding crawljob files with several links or one crawljob for each link/file?

If you place multiple mega file links into one crawljob, I can easily add caching to speed up the process.
If you have a crawljob for each file, then caching becomes more complex
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 11.01.2021, 00:52
mikk88 mikk88 is offline
Junior Loader
 
Join Date: Dec 2020
Posts: 13
Default

yes I'm using a single crawljob with many files inside. seems to get really slow proportionally to the number of files in the share, down to a level where the crawling takes longer than the actual download later.

so if this scenario can be cached and sped up it would be great!

caching over several crawljobs would be overkill I think, also as that is something the user can avoid easily.

thanks again for the effort!
Reply With Quote
  #10  
Old 11.01.2021, 17:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,587
Default

Could you send me an example crawljob with 2 or more links in it, so I can use it for testing?
send to support@jdownloader.org
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 11.01.2021, 17:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,587
Default

check again with next update. I've added caching for each crawling session
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 11.01.2021 at 17:56.
Reply With Quote
  #12  
Old 11.01.2021, 23:22
mikk88 mikk88 is offline
Junior Loader
 
Join Date: Dec 2020
Posts: 13
Default

Hello there,

the share I was using most was a private one that I cant share, but I went to find a worst case example folder on reddit: close to 4Tb share with over 100k of files. For this I created a crawljob with 100 random files and tested it (will send to you as mail now too).

What I can see is that there is only internet activity in the beginning of the crawl. So the caching seems to work just fine. There are also always exactly 100 files found in the end, so the crawling also seems to be correct and complete.
But finishing the crawling seems to be still pretty slow. I think what is new is the high CPU load .. but as I can not test the old version anymore I can not recheck this:
~5min to crawl on a 8x4.5ghz CPU with about 60% load
~10min to crawl on a 8x2.3ghz vCPU with about 30% load

While this is still relatively slow.. I think it is anyway faster than before. (I had crawljobs with less files from smaller shares which took longer still)

Last edited by mikk88; 12.01.2021 at 00:14.
Reply With Quote
  #13  
Old 12.01.2021, 16:47
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,587
Default

Please try again with next update. I've optimized and your crawljob file only took a few seconds now.
I could optimize it even further but I think it's good now + this is very rare/special case with such a extrem large folder
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 12.01.2021, 21:14
mikk88 mikk88 is offline
Junior Loader
 
Join Date: Dec 2020
Posts: 13
Default

I will give it a try tomorrow, thanks already!

And yeah I'm sure this is a rare extreme .. as far as I could tell with testing the slow down gets noticable only at shares with more than 5-10k files in it .. and even that will not be very common.

I will retry some of the bigger files I had before and report back
Reply With Quote
  #15  
Old 12.01.2021, 21:23
mikk88 mikk88 is offline
Junior Loader
 
Join Date: Dec 2020
Posts: 13
Default

I just tried the big extreme file again .. down from 5min to about 10s. I would say no further optimizing needed at all!

thanks a lot for the great improvement!
Reply With Quote
  #16  
Old 13.01.2021, 11:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,587
Default

Thanks for the feedback! I guess it's fast enough then
You're welcome
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 16:02.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.