[mikk88]

Hello,

im trying to figure out how to download only parts of a mega.nz shared folder.

Example link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

contains:
\SharedFile.jpg
\SharedSubFolder\
\SharedSubFolder\SharedFileUpSideDown.jpg

file links:
SharedFile.jpg -> **External links are only visible to Support Staff****External links are only visible to Support Staff**
SharedFileUpSideDown.jpg -> **External links are only visible to Support Staff****External links are only visible to Support Staff**


No matter which link i add to jdownloader: linkgrabber is always parsing the whole mega.nz folder and ultimately finding both files for download.

If using the subfolder link:
**External links are only visible to Support Staff****External links are only visible to Support Staff** .. it works in the way that only SharedFileUpSideDown.jpg is found and downloaded.

Ultimately I want to add single files (or a folder + list of files to be excluded) for downloading during a crawljob later, but so far i can not even get it to work in the GUI.

Any ideas of what I might be doing wrong?

Thanks in advance!
Mikk

[Jiaz]

wait for next plugin update, the url format for specifc file of folder was not yet supported

[mikk88]

ah great, I was wondering if I was doing something wrong as I thought it wouldn't be a rare use case to do.

Thanks in advance already for the update then!

[Jiaz]

Update will be available in about 5 mins, please test and provide feedback

[mikk88]

seems to work fine now, thanks a lot for the super fast update!

[Jiaz]

thanks for the feedback and you're welcome!

[mikk88]

Hello again,
maybe I found a problem still with the update.

In the LinkGrabber crawling is very slow when adding a list of files instead of folders via a crawljob file.The crawling is extremely slow when the mega.nz folder from where the files are is huge (>TB). Every file to be found when crawling takes 10+ seconds while I can see constant high bandwidth usage.

I suspect the mega.nz account/folder content is not cached and reloaded on every new link (which can be huge data in big mega folders with 10s of thousands of files) .. which would explain the time needed (opening the folder in browser takes a long time too).

Or maybe is there a way to skip the online check and crawling for crawljob files completely? I know the files are there and alive .. I would not mind to add them to the download list blindly and deal with any problems on download only.

[Jiaz]

There is currently no caching of folder information, so when you add several folder links from single folder, then JD will process them one by one and process the whole folder again.
Are you adding crawljob files with several links or one crawljob for each link/file?

If you place multiple mega file links into one crawljob, I can easily add caching to speed up the process.
If you have a crawljob for each file, then caching becomes more complex

[mikk88]

yes I'm using a single crawljob with many files inside. seems to get really slow proportionally to the number of files in the share, down to a level where the crawling takes longer than the actual download later.

so if this scenario can be cached and sped up it would be great!

caching over several crawljobs would be overkill I think, also as that is something the user can avoid easily.

thanks again for the effort!

[Jiaz]

Could you send me an example crawljob with 2 or more links in it, so I can use it for testing?
send to support@jdownloader.org

[Jiaz]

check again with next update. I've added caching for each crawling session

[mikk88]

Hello there,

the share I was using most was a private one that I cant share, but I went to find a worst case example folder on reddit: close to 4Tb share with over 100k of files. For this I created a crawljob with 100 random files and tested it (will send to you as mail now too).

What I can see is that there is only internet activity in the beginning of the crawl. So the caching seems to work just fine. There are also always exactly 100 files found in the end, so the crawling also seems to be correct and complete.
But finishing the crawling seems to be still pretty slow. I think what is new is the high CPU load .. but as I can not test the old version anymore I can not recheck this:
~5min to crawl on a 8x4.5ghz CPU with about 60% load
~10min to crawl on a 8x2.3ghz vCPU with about 30% load

While this is still relatively slow.. I think it is anyway faster than before. (I had crawljobs with less files from smaller shares which took longer still)

[Jiaz]

Please try again with next update. I've optimized and your crawljob file only took a few seconds now.
I could optimize it even further but I think it's good now + this is very rare/special case with such a extrem large folder

[mikk88]

I will give it a try tomorrow, thanks already!

And yeah I'm sure this is a rare extreme .. as far as I could tell with testing the slow down gets noticable only at shares with more than 5-10k files in it .. and even that will not be very common.

I will retry some of the bigger files I had before and report back

[mikk88]

I just tried the big extreme file again .. down from 5min to about 10s. I would say no further optimizing needed at all!

thanks a lot for the great improvement!

[Jiaz]

Thanks for the feedback! I guess it's fast enough then
You're welcome

[Nikrondu]

Sorry for bumping this thread.

Some files i am trying to download are 7 folders deep and each link i put in JD takes 3-4 min to resolve and if i do multiple links at the same time through clipboard Observer , some links are not resolved or missed.

I am currently waiting for each link to be resolved and added to download and proceeding to next.

Can something be done ?

The problem doesnt occur for one folder or two folder deep.

[Jiaz]

Originally Posted by Nikrondu

Sorry for bumping this thread.

Nothing to be sorry for Of course this is something we should optimize

Originally Posted by Nikrondu

takes 3-4 min to resolve and if i do multiple links at the same time through clipboard Observer , some links are not resolved or missed.

Sounds like the folder you're trying to add is very large? that can cause long load/parsing times and cause the issues as those lists do require lots of memory and concurrent crawling may result in "out-of-memory" errors and thus fail to process.

Can you please send some example links you're trying to add to support@jdownloader.org , then I can check/try to optimize this

[Nikrondu]

can i send it to you as PM ?

Yes the folders are pretty large.. ranging in 30+ GB .. Multiple folders totaling to some 7+ TB

I have around 32GB of RAM .. pretty latest pc though..

[Jiaz]

@Nikrondu: I've disabled PM. Either send to support@ or jiaz@

[Jiaz]

Originally Posted by Nikrondu

I have around 32GB of RAM .. pretty latest pc though..

Only matters if you customize/tweak java/JDownloader memory settings you can check about dialog of JDownloader and there the current memory usage and max memory is shown, maybe can provide a screenshot of it?

[Nikrondu]

sent..

[Nikrondu]

[Jiaz]

@Nikrondu: Thanks, I've replied by mail. Please backup your config, see https://support.jdownloader.org/Know...-configuration and replace your 32bit JDownloader installation with a 64bit one, see https://jdownloader.org/jdownloader2
Backup your cfg folder and then remove/uninstall 32bit JDownloader and install 64bit one. That way JDownloader/Java can make much more use of your available memory.

No need to blur out build date It should match this one https://support.jdownloader.org/Know...pdate-behavior

Thanks for the feedback. I will try to optimize the use case of yours

[Nikrondu]

Didnt resolve much , but as discussed , will wait for your update. Thanks a lot in advance.

As for blurring out , in a haste , i thought it was showing GMT timing.. My bad.. i too realized my noobish act later only.. :D

[Nikrondu]

[Jiaz]

Thanks for feedback. I'm sure I will be able to optimize this Just give me some time