JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 26.10.2020, 22:27
pooky2483 pooky2483 is offline
Vacuum Cleaner
 
Join Date: Oct 2020
Posts: 16
Default Unable to grab more than just over 10,000 links

I am unable to download more than what can be found on the first run of a search of a YT channel.
I enter the same link again and it finds the same 10,000 links over and over again and not the next 10,000 links.
Why is this when the channel has over 80,000 video's

My log id: jdlog://0268325302851

Why is it just discovering the same links over and over again and not skipping them to grab the next 10,000 links, is there something I need to set for it to happen?
Reply With Quote
  #2  
Old 27.10.2020, 00:47
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,226
Default

are you sure that google even shows it?
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #3  
Old 27.10.2020, 02:11
pooky2483 pooky2483 is offline
Vacuum Cleaner
 
Join Date: Oct 2020
Posts: 16
Default It's now 90,013

Edit by Jiaz: removed image

Last edited by Jiaz; 27.10.2020 at 10:29.
Reply With Quote
  #4  
Old 27.10.2020, 10:24
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,433
Default

@pooky2483: plugins are stateless. Adding the link first time is the same as adding the link second/third...means the plugin does not know that it already processed anything and starts from beginning. It already avoids crawling the same video again (when already in list) but chances are very high that you just hit crawling limit with each pagination returning only a few...do the math and very large number of requests. I'm sure you will see issues/blocks when doing this manually

When creating such logs, please do NOT download any files as they just trash the logs.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 28.10.2020, 07:45
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,226
Default

i meant more shows it within a means that user can see a list of 90k videos. at times large pages even with ajax loading stops after say 10k
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 22:37.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.