#1
|
|||
|
|||
Unable to grab more than just over 10,000 links
I am unable to download more than what can be found on the first run of a search of a YT channel.
I enter the same link again and it finds the same 10,000 links over and over again and not the next 10,000 links. Why is this when the channel has over 80,000 video's My log id: jdlog://0268325302851 Why is it just discovering the same links over and over again and not skipping them to grab the next 10,000 links, is there something I need to set for it to happen? |
#2
|
||||
|
||||
are you sure that google even shows it?
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#3
|
|||
|
|||
It's now 90,013
Edit by Jiaz: removed image
Last edited by Jiaz; 27.10.2020 at 10:29. |
#4
|
||||
|
||||
@pooky2483: plugins are stateless. Adding the link first time is the same as adding the link second/third...means the plugin does not know that it already processed anything and starts from beginning. It already avoids crawling the same video again (when already in list) but chances are very high that you just hit crawling limit with each pagination returning only a few...do the math and very large number of requests. I'm sure you will see issues/blocks when doing this manually
When creating such logs, please do NOT download any files as they just trash the logs.
__________________
JD-Dev & Server-Admin |
#5
|
||||
|
||||
i meant more shows it within a means that user can see a list of 90k videos. at times large pages even with ajax loading stops after say 10k
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
Thread Tools | |
Display Modes | |
|
|