#1
|
|||
|
|||
Linkgrabber / Crawling slower than previous
Hi,
So I absolutely love jdownloader, amazing work on this. But one thing I noticed is that when I initially used it and recent usage there is an issue with how quickly it parses the clipboard copied items. I don't recall having this issue even with the LinkCollector: Do Link Check enabled in ex 1 bu recently it's extremely slower than previously experienced. ex. 1 - have about 300 lines in an excel file and only of those 52 have urls with the others are blank. - in theory it should parse those in less than 30 seconds or so (including link checks) but it's taking considerably longer time. ex. 2 - have my sms history with embedded URLs - it's about ~5300 lines - 896 occurrences of URLs (web site links, social media links, YouTube, etc) I'd say an hour at most or less. It's going on for 12hrs. I used https://board.jdownloader.org/showthread.php?t=58776 and disabled as it shared in the thread... Go into the expert settings, type in "LinkCollector: Do Link Check" --> Disable that Question is there a way to reduce the time significantly?? 12+ for 5k lines seems like something isn't working as intended. All I want is to parse the urls embedded in the text file quicker. |
#2
|
||||
|
||||
It probably just a co-incidence on the source, and time frame, vs what links are contained within said source medium. For example urls that trigger long decrypt tasks ie. playlists or complete channel/profiles (youtube/twitter) for example could take days depending on said profile. You can create rules to block say twitter and you can use plugin settings for youtube to ask/decline profile decryptions.
Without knowing what links were copied, this is just an educated guess. raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#3
|
||||
|
||||
@snuffleupagus: can you please provide a screenshot of your JDownloader about dialog. Most likely you hit memory limits of Java and need to allow larger heap, see
https://support.jdownloader.org/Know...vmoptions-file
__________________
JD-Dev & Server-Admin |
#4
|
||||
|
||||
Quote:
Especially some social media sites have introduced heavily limits recently.
__________________
JD-Dev & Server-Admin |
#5
|
|||
|
|||
Sorry for the delay..
here's the link: **External links are only visible to Support Staff****External links are only visible to Support Staff** The links are like everything I share with myself - social media video links Not sure if there is a limit but definitely seems a bit slower than expected. |
#6
|
|||
|
|||
Just measured some time
took 1 minute to add 3 - TikTok links to the link grabber interface. I'd hope it could do 1 link a second or faster. so 40 links in theory would take 13 minutes. |
#7
|
||||
|
||||
@snuffleupagus: tiktok has very strict rate limiting/bot protection. therefore the requests are limited to request per second for tiktok. besides than that there are no limitations.
Please create a debug log. Enable Settings->Advanced Settings->Log.debugmodeenabled and restart JDownloader Don't download anything but just add some links and then create log, see https://support.jdownloader.org/Know...d-session-logs and post shown logID here. Also please provide example links for testing
__________________
JD-Dev & Server-Admin |
#8
|
|||
|
|||
@Jiaz thanks. Please consider it solved.
|
#9
|
||||
|
||||
@snuffleupagus: thanks for the feedback!
__________________
JD-Dev & Server-Admin |
|
|