JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 26.04.2022, 16:33
I3ordo I3ordo is offline
DSL Light User
 
Join Date: Mar 2022
Posts: 34
Default Automation for the crawljob?

My last two months have been "religiously" visiting my favourite site and visiting my booked marked (fixed url) search results that shows new entries as first at top: **External links are only visible to Support Staff****External links are only visible to Support Staff**

After opening each post individually, i ctr+a and ctrl+c the whole page so that the link grabber can find the links and add them. Normally, if i don't open the actual paste but paste the link of the results page to JD, it finds nothing even with deep crawl...

It s practical already, but it is semi automated in a way. Would love to know if it can get fully automated = opening each post on the first two pages(there are hundreds of pages out there) and grab their texts that contain links that lead to archives and auto-start download with appropriate provider (which it already does already).

I have checked that there s an extension called "folder watch" which scans the "list.crawljob" and auto adds them to downloads which is a very good starting point.

maybe i should find someone with phyton automation and selenium experience to create a script that auto creates those links for the crawljob?.

So the actual question becomes, can JD auto crawl web site pages daily or should i go for the "phyton-selenium" option?
Reply With Quote
  #2  
Old 26.04.2022, 17:50
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,912
Default

Quote:
Originally Posted by I3ordo View Post
So the actual question becomes, can JD auto crawl web site pages daily or should i go for the "phyton-selenium" option?
I'm sorry but JDownloader does not support such feature *out-of-the-box*.
Best would be to use/write external tools that automate the creation of crawljob files
and/or use the Eventscripter and a script to watch/automate the adding of links, see
https://board.jdownloader.org/showthread.php?t=70525
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 26.04.2022 at 17:53.
Reply With Quote
  #3  
Old 26.04.2022, 18:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,912
Default

Quote:
Originally Posted by I3ordo View Post
After opening each post individually, i ctr+a and ctrl+c the whole page so that the link grabber can find the links and add them.
This you can easily automate via Eventscripter, a script that runs in interval and then loads the html page and parses the links/content you are interested in and then *feed* JDownloader with it.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 03.05.2022, 03:48
I3ordo I3ordo is offline
DSL Light User
 
Join Date: Mar 2022
Posts: 34
Default

is there an already working script that i can start with? i doubt i can find someone that would build a script for a few bucks
Reply With Quote
  #5  
Old 03.05.2022, 12:47
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 76,912
Default

@I3ordo: best would be to write a post in https://board.jdownloader.org/showthread.php?t=70525 and explain what you need/what the script should do
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 09:23.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.