JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 19.10.2010, 00:49
jdlbot
Guest
 
Posts: n/a
Default I wrote a small RSS feed scraper for jDownloader

Grab it here:
Code:
**External links are only visible to Support Staff**
Check out the source code and README here:
Code:
**External links are only visible to Support Staff**
If you have problems, check the wiki here:
Code:
**External links are only visible to Support Staff**
Please feel free to open issues, send patches or give feedback/suggestions. Consider this an alpha release.

I've already spotted a couple of big issues, one of which is that it DOES NOT support ATOM feeds. Only RSS. If you get a parser error and your know it is a valid feed, chances are it's ATOM. I already have a fix for this in mind.

New version 0.1.1 supports Atom! Resolved bugs when adding links.
New version 0.1.2 detects missing parts of multipart rars. Improved TV episode recognition.

I am aware of flexget and it's ilk. I thought that my scripts cobbled together with bits of string and tape would be easier for the uninitiated to use.

Right now, I can only confirm that this works with the latest STABLE jDownloader (and Web Interface).

Happy downloading,
jdlbot

Last edited by jdlbot; 30.11.2010 at 18:32. Reason: new version!
Reply With Quote
  #2  
Old 19.10.2010, 09:01
zerobyte zerobyte is offline
DSL User
 
Join Date: Jan 2010
Posts: 38
Default

much appreciated.

i'll try it out tonight. i'm using feedreader at the moment. works ok, but sometimes copy/paste freaks out.

00h.
Reply With Quote
  #3  
Old 19.10.2010, 10:38
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,437
Default

Nice.

Flexget needs an external program to put the description or HTML page into the clipboard. It could use other interfaces, but that would mean relying on Flexget to find the correct URLs.
Reply With Quote
  #4  
Old 19.10.2010, 18:21
jdlbot
Guest
 
Posts: n/a
Default

@drbits - I should be clear in my initial post. This is an external program that finds the correct urls and sends them to jDownloader. It uses the jDownloader web interface to communicate.

Unlike flexget, it requires minimal configuration. Setup some feeds, make some filters and presto!

@zerobyte - Unlike feedreader, jDlBot is cross platform (will run anywhere perl runs) and will (hopefully) avoid any clipboard issues. Also, it just doesn't scrape the feeds, it will follow the feed links and scrape the resulting pages if desired.
Reply With Quote
  #5  
Old 19.10.2010, 18:24
Greeny
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by jdlbot View Post
It uses the jDownloader web interface to communicate.
Maybe its easier to use the Remote Control for interacting with the JD?
Reply With Quote
  #6  
Old 19.10.2010, 18:47
jdlbot
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by Greeny View Post
Maybe its easier to use the Remote Control for interacting with the JD?
I use the stable version of JD myself and found that the version of the remote control in stable does not work consistently. Also, it gives no feedback on the status of links in the linkgrabber queue.

All of this has been changed in the nightly version of the remote control. I have to say that it's awesome (great work!), and I will be targeting that for JD integration after I clean up the initial version of jdlbot.

I have some experience scraping the web with perl, so scraping the Web Interface page and generating appropriate querystrings wasn't really an issue.
Reply With Quote
  #7  
Old 19.10.2010, 19:07
Greeny
Guest
 
Posts: n/a
Default

Allright! When the Nightly will be the next Stable I'm looking out for the next version of your little app.

Are you familiar with Java? If so, feel free to download the source and implement your app in Java, that we can release it as an official addon :-)
Reply With Quote
  #8  
Old 19.10.2010, 20:17
jdlbot
Guest
 
Posts: n/a
Default

@Greenly - That's actually how I started on this. Unfortunately, the only experience I have with Java is in writing web services and not gui application dev. I wasn't really making much progress on that front, so I just expanded on my existing perl scripts.

I could hack at making it an addon again, but there were several issues that I had yet to solve. Storing and calling feed/filter data, generating the gui forms (no idea what I was doing there), integrating feed reading and screen scraping libs. etc.
Reply With Quote
Old 20.10.2010, 03:51
drbits
Message deleted by drbits.
  #9  
Old 20.10.2010, 04:03
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,437
Default

Feel free to post your program on a file host and link to it here.

If you license the program under GPL3, one of us can take your source, translate it to Java, and convert it into an addon.

On the other hand, it might be better to keep the RSS/ATOM/NNTP feeds as accessory programs (still need the GPL license).

It would be nice if programs that passed just URIs could use the CNL standard. If the program is passing a web page for LinkGrabber, it is best to use the Clipboard.
Reply With Quote
  #10  
Old 20.10.2010, 04:41
jdlbot
Guest
 
Posts: n/a
Default

@drbits - I'll put a license file in my repo sometime tomorrow. If GPLv3 is best for you guys, I'll put all my source under that. The binary distributions however fall under one of the perl licenses (artistic or gpl depending on the modules.)

Does CNL have a way to check the status of the linkgrabber queue? The only documented feature I noticed was adding links.

jDlBot already extracts the links and posts a URI encoded list to the web interface. It should be easy to change if necessary.
Reply With Quote
  #11  
Old 20.10.2010, 22:19
jdlbot
Guest
 
Posts: n/a
Default

Released a new version with a bunch of fixes.
Reply With Quote
  #12  
Old 21.10.2010, 10:10
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,437
Default

CNL does not have a way to check status it is POST only.

The new (Nightly) remote control has a command to check the number of links in the Link Grabber.
To get help: **External links are only visible to Support Staff** You will probably be interested in:
/get/grabber/count
/get/grabber/isbusy

I know that you don't want to use the Nightly Test version, but the only other way to get the information is to scrape the Web interface and that will not be easy.
Reply With Quote
  #13  
Old 21.10.2010, 15:39
jdlbot
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by drbits View Post
... but the only other way to get the information is to scrape the Web interface and that will not be easy.
This is pretty much what I'm doing (**External links are only visible to Support Staff**link) :D

It's not pretty... without getting the links back in the interface, it really just tries to tell which ones were added last and polls the web interface add links page for updates. If every link it thinks it added last is online, then (if desired) it pushes the packages to the download queue.

This could pose a problem if you add links to the clipboard/linkgrabber while jdlbot is pushing links. I've been running this at home for a little while now and it hasn't been an issue.

Last edited by jdlbot; 21.10.2010 at 16:46.
Reply With Quote
  #14  
Old 03.11.2010, 05:13
kiberiada
Guest
 
Posts: n/a
Question A little tutorial

This is exactly a thing we miss from the JDownloader.

Can you give me a hint how to set up the feeds / filters?

I could run and configure the app to access the JDownloader, added some feeds, but but except a short Checking for updates... No new updates sequence nothing happens.

You do suggest something about the filters, but I found the interface is a bit over my capabilities Can you explain please, where and what do I have put to get a feed like this to work?

Code:
**External links are only visible to Support Staff**
Thank you for your effort and patient.
Reply With Quote
  #15  
Old 03.11.2010, 05:36
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,437
Default

This is great!

However, a lot of the JD interaction will be much easier when the new remote control is ready. That just means the next release
Reply With Quote
  #16  
Old 03.11.2010, 23:12
jdlbot
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by kiberiada View Post
This is exactly a thing we miss from the JDownloader.

Can you give me a hint how to set up the feeds / filters?

I could run and configure the app to access the JDownloader, added some feeds, but but except a short Checking for updates... No new updates sequence nothing happens.

You do suggest something about the filters, but I found the interface is a bit over my capabilities Can you explain please, where and what do I have put to get a feed like this to work?

Code:
**External links are only visible to Support Staff**
Thank you for your effort and patient.
The interface is a bit sparse at the moment and will probably change in the next release.

I should make a note also to NOT use Internet Explorer to access the configuration, as IE uh... has issues.

That being said, to add a new feed/filter please follow these steps:

1. Click on the feeds link on the left hand panel
2. Input the feed url, the interval and whether or not you want to follow the feed links
3. Click "Add feed"
4. Wait for either A) an error message or B) the recently added feed to popup above the new feed area.
5. At this point you should see a "Running Watcher" fire in the terminal window. This means your feed is active.

6. Click on the filters link in the left hand pane
7. Add your filter parameters, hover over the inputs to get help bubbles
7a. Be sure to add at least one expected linktype. ie megaupload or hotfile. If there can be multiple link types put them in a pipe delimited list: megaupload|hotfile

I'm currently working on expanding this feature.

8. Click "Add filter" and wait for confirmation.

9. If you want to rerun your feed watcher right that second, go to the feeds page and uncheck and recheck the box next to it. You will see "Running Watcher" appear again in the status window.



Now... after all that I've checked out the link you provided and I see absolutely no useful links in either the feed or resulting pages. I'm not sure why you would even want to add that feed. (unless you're trying to scrape myspace pages... then you would add something like myspace in the linktype field of the filter.)

When I made this I had things like katz or ev0 feeds in mind.
Reply With Quote
  #17  
Old 07.11.2010, 11:43
holtzi
Guest
 
Posts: n/a
Default JDFeedMe

a similar plugin is developed.
check out this: JDFeedMe
Code:
http://board.jdownloader.org/showthread.php?p=114316
**External links are only visible to Support Staff**
maybe we can collaborate

Last edited by holtzi; 07.11.2010 at 12:34.
Reply With Quote
  #18  
Old 07.11.2010, 12:17
remi
Guest
 
Posts: n/a
Default

@holtzi

Thanks for this great feature.

I wonder why this is just an "add-on", because it seems to be properly documented and well integrated with jD.
Reply With Quote
  #19  
Old 08.11.2010, 23:41
jdlbot
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by holtzi View Post
a similar plugin is developed.
check out this: JDFeedMe
Code:
http://board.jdownloader.org/showthread.php?p=114316
**External links are only visible to Support Staff**
maybe we can collaborate
pm sent!
Reply With Quote
  #20  
Old 09.11.2010, 11:27
chaver1
Guest
 
Posts: n/a
Default can you please post a link for jd beta release

cause it cannot work on jdownloader untill they sighn it please help me find a beta release thanks
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 09:03.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.