[Perene]

I tried adding 443 GB spread over more than 23 thousand Google Drive links and they were ALL added successfully in a matter of minutes. This info I got from a TXT file I have here.

However... if we create a single folder/container with all these links, and then save as DLC, this is what happens if try to open that DLC:

1) JDW will read one link every 2 seconds. You read me correctly: it will take FOREVER to read all data from that DLC file.

2) Somehow this task will be interrupted and JDW will ignore I'd say 80% of that original number. It will not say the rest is offline. It will simply not read all of them and give up after some time.

I detected this issue with ALL my DLCs with a great number of links

a) 1507
b) 1891
c) 6028
d) 2393
e) 2374
f) 4181
e) 4592

All of them incomplete. The only way to ADD everything is to do this in TXT format.

Control +C and control + V.

This is what I gathered from DLCs I had created in the past.

DLCs working fine - number of links:

- 303
- 568
- 843
- 380
- 311

And this has nothing to do with the total size.

Because I saw that a DLC with a few terabytes and over 300 links was read quickly and all contents showed there!

All I am saying indicates that DLC creation from a large number of links is flawed. Either this is JDW's fault or it happens with all DLCs regardless of the program used.

If that's the case, can you please inform the max number of links and scenarios in which creating a DLC is a waste of time?

My guess is that anything over 1K creates this problem.

[Jiaz]

@Perene: DLC is NOT for export/import of links! It's just for sharing links and will not retain any meta information and thus JDownloader will have to fully process the links again which will take long time and often not result in the same results (because of missing meta information). Also JDownloader has to parse the full DLC first before it can start working on it which might require lots of memory (due to size of file and number of links in it)

Of course I can check deeper and maybe optimize it, but for this I need some of those DLCs to work with.
You can send me some to support@jdownloader.org
Maybe I can rewrite that part to not process all into memory first but process while reading the file

Also please provide a screenshot of the about dialog of JDownloader, so we can check if memory constraints might be the cause of the issues you encounter, see https://support.jdownloader.org/Know...tion-directory

In case you simply want to *move* your links to a different installation, you can easily copy the cfg/linkcollectorXXX.zip file from your JDownloader to other JDownloader instance and open with via drag/drop
or load container dialog, see https://support.jdownloader.org/Know...nkgrabber-list

[Jiaz]

Originally Posted by Perene

This info I got from a TXT file I have here
However... if we create a single folder/container with all these links, and then save as DLC, this is what happens if try to open that DLC:

are those links the folder links or each link belongs to a single file? folder links are much much faster because with a single link/less requests you will add many links from that folder. but with single file links, each link/file in list will require one/several requests which makes it the whole process much much slower.

[Perene]

This is what I am doing here. Mentioning only one example.

Important: JDW has this setting enabled:
https://support.jdownloader.org/Know...ackage-feature

*******
Example #1: 2374 links. 890 GB.

- JDW read this in less than 2 minutes after I do a control +C and +V
- 157 different packages if JDW has this setting enabled.

Next thing I did:

Select all these 157 packages and "move" them to a single one. With 2374 links/files and 890 GB.

And finally: create a DLC based on this unique folder.

Now try to open this DLC. If it works then it will show the same stats: 2374 links, 890 GB.

So far it is reading 320 links after X minutes. So the reading is painfully slow. Meaning this DLC creation is useless and I should only stick with the TXT and all the links it contains.

My goal was not to get the full URL from every single file, because I already know those (the TXT is there for that purpose). It was to select randomly any of the 2374 links and download if I wish.

That's the reason why I created the DLC in the first place, otherwise I'll always have to rely on the TXT to get the desired content.

I just sent an email with two files for you to check: TXT and DLC.

With these two files you'll be able to check everything I say and if possible optimize how it works.

As for not moving all contents into a single folder, this doesn't solve the slowdown you explained, even if I let JDW to organize them with the "subfolder-for-each-package-feature".

After more than 20, 30 minutes only 50% of the 2374 links were read, as you can see below:

https://i.postimg.cc/WsYrhxZV/IMAGE.jpg

[Jiaz]

Originally Posted by Perene

I just sent an email with two files for you to check: TXT and DLC.
With these two files you'll be able to check everything I say and if possible optimize how it works.

Thanks, I will check those by tomorrow and get back to you.

[Jiaz]

Originally Posted by Jiaz

Also please provide a screenshot of the about dialog of JDownloader, so we can check if memory constraints might be the cause of the issues you encounter, see https://support.jdownloader.org/Know...tion-directory

Would be great if you could provide a screenshot, so I can see the memory constraints

[Perene]

Originally Posted by Jiaz

Would be great if you could provide a screenshot, so I can see the memory constraints

What kind of screenshot would help you?

This is a i5 11400 computer... internet = 600 Mbps (300 for UL), so probably it's slow due to how these DLCs work based on your explanation, not other factors.

If I were to make a suggestion it would be that after JDW tries to read these DLCs, that it should behave exactly the same as it does when you copy-paste from the TXT.

This is the solution, otherwise it really takes forever to read one by one. Mipony's behavior is similar.

If my idea is possible we would waste a few minutes reading the whole thing, instead of each link every X seconds.

When dealing with thousands of links at the same time this would take forever if the DLC is all you have.

I can randomly select content X or Y reviewing these TXTs, still I generated these DLCs for the same task. But if can't load everything quickly, I'll always have to rely on the TXTs.

Also, consider these thousands of links are spread over multiple free accounts. Not a single one.

[Jiaz]

@Perene:
Thanks! As I thought it's caused by this, https://board.jdownloader.org/showpo...30&postcount=3
Your txt file contains lots of folder links. That means one link(very few requests) and many resulting links from this one folder link. But the exported dlc only contains single links. For 2000 single links, JDownloader has to do 2000 requests. But for example there are folder links that contain 69 links that just require 3 requests. And this is what takes time.
By default JDownloader checks every links for status/details and that's the reason why it takes so long.
see my previous answer

Originally Posted by Jiaz

@Perene: DLC is NOT for export/import of links! It's just for sharing links and will not retain any meta information and thus JDownloader will have to fully process the links again which will take long time

You can disable linkcheck in Settings->Advanced Settings->LinkCollector.dolinkcheck, then it only takes about 2-3 seconds for all the files.

Why you want to use DLC for this when you can easily just use your cfg/linkcollectorXXX.zip file to transport/backup your linklist to some other
installation. DLC might work okay for google drive links but may fail with other hosts + slow import/loading because of full processing of the input files again.

[Perene]

I am not sure I am following you...

When you said:

Originally Posted by Jiaz

You can disable linkcheck in Settings->Advanced Settings->LinkCollector.dolinkcheck, then it only takes about 2-3 seconds for all the files.

Does this mean if we do the above we aren't really verifying if the links are online?

I don't need, simply, JDW to list all links. Or to export any of them from a DLC.

I want more: to list the 2000 AND check if they are online. A lot faster than it is currently doing.

For example: let's say 2 days from now I open the DLC I sent you.

If the links are gone JDW needs to me tell me which ones. Even if it's 1999 OK and 1 offline.

If it takes forever for that checking then do you suggest I stop using DLCs?

Originally Posted by Jiaz

DLC is NOT for export/import of links! It's just for sharing links and will not retain any meta information and thus JDownloader will have to fully process the links again which will take long time

Checking every link = takes forever, if you open a DLC with thousands of links.

Not if you do the copy-paste from the TXT.

We already know it's not due to a slow computer or internet speed constraints.

Can't this be modified so it can be a lot faster than actually is?

I haven't noticed this problem until I started saving all into DLCs.

I don't need to export links from a DLC, what I want is to make sure if the links are ALL online.

[Jiaz]

Originally Posted by Perene

I need it to list the 2000 AND check if they are online. A lot faster than it is doing now.

That is not possible because JDownloader has to do 1 request to google for each link which results in 2000 requests
for 2000 files. take average of 50ms/request results in about 20 requests/links per second.

Adding the links via folder link (from your TXT file) is much faster because a folder can hold up many files, eg 100 links and
then you have 100 links with just 3 requests, so you could add for example about 600 links per second with folder links
but about only 20 links per second with single links. given example average request time of 50 ms


The speed difference is that your TXT file contains LOTS of folder links (3 requests = 50-100 links) whereas your DLC contains ONLY single file links (1 request for 1 file). The rest is simple math.

[Jiaz]

Originally Posted by Perene

If it takes forever for that checking then do you suggest I stop using DLCs?

DLC is not for export/import/back. It's for sharing and if you share 2000 links with someone and this person
wants to know the status of each link, then JDownloader has to check/verify each link. Some hosters do provide special
mass check apis but google drive does not (not that I'm aware of)

[Perene]

Originally Posted by Jiaz

The speed difference is that your TXT file contains LOTS of folder links (3 requests = 50-100 links) whereas your DLC contains ONLY single file links (1 request for 1 file). The rest is simple math.

If that's the case would this idea work?

- 2000 links copy-pasted from that TXT:
-- Results in: 157 different folders.

'Create a subfolder by packagename' feature = ENABLED.

Save as DLC all the aforementioned 2000 links, but listing as 157 different folders. Not 2000 links brought together...

I decided to unite all links just for convenience.

[Jiaz]

@Perene: You will always loose the folder speed *advantage* once you've added the links as GoogleDrive does not provide any mass linkcheck api (I'm not aware of any) and after adding the links, JDownloader only can check status one by one.
Another idea, add the links, then use "copy to clipboard" action which copies information in customizable format to clipboard, see rightclick context menu -> menu edit- > copy to clipboard action and there you can customize it. That way you can copy folder links (where folder links do exist) and single links will stay single links. Then create a crawljob, see https://support.jdownloader.org/Know...3/folder-watch
and there you can customize stuff like packagename and paste the links into that file. That way you have a *container* file with the *fast* links in it and
yet be able to customize stuff like packagenames.
then you can load such a crawljob file at later time and have high speed and custom properties support.