JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 17.07.2016, 01:39
BWEL BWEL is offline
JD Alpha
 
Join Date: Feb 2016
Location: Toronto, ON
Posts: 24
Question LinkCrawler & Package Name Scenario (Assistance Needed)

Dear JDownloader Contributors,

I would like to present to you my little project with link crawler rules and package name.

I am trying to crawl a certain website for download links:

The configuration is as follows:

1) LINK CRAWLER RULE
Code:
{
  "enabled" : true,
  "maxDecryptDepth" : 1,
  "id" : 1467796149827,
  "name" : "site.name",
  "pattern" : "(.*site\\.name/videohive-.+)",
  "rule" : "DEEPDECRYPT",
  "packageNamePattern" : null
}

2) Link Crawler finds download URL in the following crawled link:
Code:
site.name/videohive-ultimate-infographic-hud-300-9719753/

3) The download link is a zip file and JDownloader Packagizer names it automatically as:
Code:
download.link/view/1BB2C2B6002EE24/EM-9719753.zip -> Package Named to: EM-9719753

What I am trying to do is the following, have the packagizer get the package name from either meta or title headers on the blog url: site.name/videohive-ultimate-infographic-hud-300-9719753/ which reads as:

Code:
<link rel="alternate" type="application/rss+xml" title="Downae &raquo; VideoHive – Ultimate Infographic HUD [300] – 9719753 Comments Feed" href="site.name/videohive-ultimate-infographic-hud-300-9719753/feed/" />

<meta property="og:title" content="VideoHive – Ultimate Infographic HUD [300] – 9719753"/>

or if that would be impossible, by simply getting the name from the URL instead: "videohive-ultimate-infographic-hud-300-9719753"

My question is the following, how should I format "packageNamePattern" in Link Crawler Rules so as to get the name from the header/url name instead of the zip/rar file name?

I am still unclear about this process and I apologize for bothering you guys with this.

Thanks
Reply With Quote
  #2  
Old 17.07.2016, 03:39
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,226
Default

@BWEL
at this time you can not use any reference of the website body to determine packagename or filename. You will need to create a decrypter plugin if you want todo such a thing.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #3  
Old 17.07.2016, 08:36
BWEL BWEL is offline
JD Alpha
 
Join Date: Feb 2016
Location: Toronto, ON
Posts: 24
Default

@Raztoki thanks for the quick reply. I appreciate it. Is there any reference or wiki available on how to create a decrypter plug-in?

I checked the Development Getting Started Wiki, but I couldn't really find any relevant information on the structure of a decrypter plugin.

Without adding to your volume of work any, would you kindly give me some pointers or at least an example of an existing decrypter plugin? I can get things working if I have an example to base my work on.

Thanks a lot,
B-WEL

P.S. I am dual language English/German materials work for me just fine.
Reply With Quote
  #4  
Old 17.07.2016, 08:47
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,003
Default

Packagizer rule:
Downloadurl > contains > download.link/view/.+ > Enable Regex
Sourceurl(s) > contains > site.name/(.+)/ > Enable Regex
Then set > Filename > <jd:source:1>.<jd:orgfiletype>

Sorry, I misread the post at first.

Linkcrawler rule:
Code:
[ {
  "enabled" : true,
  "maxDecryptDepth" : 1,
  "id" : 1466627516456,
  "name" : "site.name",
  "pattern" : ".*site\\.name/videohive-.+/",
  "rule" : "DEEPDECRYPT",
  "packageNamePattern" : ".*site\\.name/(videohive-[^/]+)/"
} ]

Last edited by mgpai; 17.07.2016 at 10:11.
Reply With Quote
  #5  
Old 17.07.2016, 09:10
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,226
Default

@mgpai
could work, but I he said 'either meta or title headers', he would prefer from html source vs url structure

@BWEL
We have hundreds if not thousand (past and present) working examples JDownloader/src/jd/plugins/decrypter. Find a site that you use and understand how it works, then have a look at the decrypter for that site and see what is done in respects to JDownloader. It's quite simple when you understand how the website works.

Guide: https://support.jdownloader.org/Know...lopment-german
the PDF is in English and German.


raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #6  
Old 17.07.2016, 10:08
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,003
Default

Quote:
Originally Posted by BWEL View Post
or if that would be impossible, by simply getting the name from the URL instead: "videohive-ultimate-infographic-hud-300-9719753"
@raztoki

Because it was one of the preferred options, I thought it might also be the easiest.
Reply With Quote
  #7  
Old 17.07.2016, 10:23
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,226
Default

@mgpai
Please don't misunderstand what I was trying to convey, nor was I trying to have a go at you or your solution either. I understand your reasoning =] Hence could work.

I mentioned decrypter was a solution based on what he indicated from html source.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #8  
Old 17.07.2016, 15:04
BWEL BWEL is offline
JD Alpha
 
Join Date: Feb 2016
Location: Toronto, ON
Posts: 24
Default

@mgpai and @raztoki thank you for the support. I greatly appreciate it. I will be on it today and get things working. THANK YOU! I will update you guys with my code later so you can check it out.
Reply With Quote
  #9  
Old 18.07.2016, 12:38
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,412
Default

packageNamePattern is the pattern to parse package name from HTML source of the loaded ressource
so should be possible just fine by providing the correct pattern
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 18.07.2016, 14:27
BWEL BWEL is offline
JD Alpha
 
Join Date: Feb 2016
Location: Toronto, ON
Posts: 24
Default

I am finally cracking Eclipse and installed everything in perfect order. My only problem now is that I did modify a decryptor plugin to suit my needs, but whenever I run JDownloader from Eclipse it doesn't work, pretty much it doesn't collect the links. Can someone please have a look below? Thanks


So pretty much the post's name is wrapped under H3 and the links are usually available via image click or listed below an image with the following formatting:



My undertsanding is that String notThis is the one that declares the possible URL to crawl from, and I have followed the logic in the original Avxhome Plugin, but unfortunatley even though I save it and run JDownloader from org.jdownloader.startup main.java the plugin either doesn't get activated to test it or there's something wrong in the syntax.

Anyone with some experience, please help. THANK A LOT

Last edited by Jiaz; 18.07.2016 at 14:49.
Reply With Quote
  #11  
Old 18.07.2016, 14:48
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,412
Default

I suggest we do a Teamviewer session (send me ID and PW to support@jdownloader.org) and I will help you setup your IDE and get your plugin running, showing you how to use it in standalone JDownloader

did you test packageNamePattern like I explained?
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 13:15.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.