JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 18.09.2021, 16:43
snuffleupagus snuffleupagus is offline
Ultra Loader
 
Join Date: Feb 2020
Posts: 46
Default Linkgrabber / Crawling slower than previous

Hi,

So I absolutely love jdownloader, amazing work on this. But one thing I noticed is that when I initially used it and recent usage there is an issue with how quickly it parses the clipboard copied items. I don't recall having this issue even with the LinkCollector: Do Link Check enabled in ex 1 bu recently it's extremely slower than previously experienced.

ex. 1
- have about 300 lines in an excel file and only of those 52 have urls with the others are blank.
- in theory it should parse those in less than 30 seconds or so (including link checks) but it's taking considerably longer time.

ex. 2
- have my sms history with embedded URLs
- it's about ~5300 lines
- 896 occurrences of URLs (web site links, social media links, YouTube, etc)

I'd say an hour at most or less. It's going on for 12hrs.

I used https://board.jdownloader.org/showthread.php?t=58776
and disabled as it shared in the thread... Go into the expert settings, type in "LinkCollector: Do Link Check" --> Disable that

Question is there a way to reduce the time significantly?? 12+ for 5k lines seems like something isn't working as intended.

All I want is to parse the urls embedded in the text file quicker.
Reply With Quote
  #2  
Old 18.09.2021, 18:56
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,389
Default

It probably just a co-incidence on the source, and time frame, vs what links are contained within said source medium. For example urls that trigger long decrypt tasks ie. playlists or complete channel/profiles (youtube/twitter) for example could take days depending on said profile. You can create rules to block say twitter and you can use plugin settings for youtube to ask/decline profile decryptions.

Without knowing what links were copied, this is just an educated guess.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #3  
Old 20.09.2021, 18:56
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,936
Default

@snuffleupagus: can you please provide a screenshot of your JDownloader about dialog. Most likely you hit memory limits of Java and need to allow larger heap, see
https://support.jdownloader.org/Know...vmoptions-file
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 20.09.2021, 18:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,936
Default

Quote:
Originally Posted by snuffleupagus View Post
Question is there a way to reduce the time significantly?? 12+ for 5k lines seems like something isn't working as intended.
What sort of links? Maybe you just hit crawling/request limit of a service you are trying to crawl and that's why it takes ages.
Especially some social media sites have introduced heavily limits recently.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 25.09.2021, 13:26
snuffleupagus snuffleupagus is offline
Ultra Loader
 
Join Date: Feb 2020
Posts: 46
Default

Sorry for the delay..
here's the link:


**External links are only visible to Support Staff****External links are only visible to Support Staff**

The links are like everything I share with myself
- reddit
- social media video links

Not sure if there is a limit but definitely seems a bit slower than expected.
Reply With Quote
  #6  
Old 25.09.2021, 14:14
snuffleupagus snuffleupagus is offline
Ultra Loader
 
Join Date: Feb 2020
Posts: 46
Default

Just measured some time
took 1 minute to add 3 - TikTok links to the link grabber interface. I'd hope it could do 1 link a second or faster.

so 40 links in theory would take 13 minutes.
Reply With Quote
  #7  
Old 27.09.2021, 14:59
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,936
Default

@snuffleupagus: tiktok has very strict rate limiting/bot protection. therefore the requests are limited to request per second for tiktok. besides than that there are no limitations.
Please create a debug log. Enable Settings->Advanced Settings->Log.debugmodeenabled and restart JDownloader
Don't download anything but just add some links and then create log, see https://support.jdownloader.org/Know...d-session-logs
and post shown logID here.
Also please provide example links for testing
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 03:50.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.