JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 16.06.2019, 13:56
smilies smilies is offline
JD Alpha
 
Join Date: Jan 2019
Posts: 23
Default LinkCrawler Rules - passing information through DEEPDECRYPT

Hi, I'd like to crawl a webpage A (then B, then C, ...) for links but to modify those links. One of the modifications is to insert a custom_string into the links. This custom_string isn't constant, it's different for each webpage A,B,C. The webpage doesn't care what comes after the question mark in the URL, so I can make the URL look like original_url?custom_string if that helps. Besides, the webpage contains Custom_string, i.e. a version of custom_string where the first letter is capitalized.
  • To crawl the webpage for links, I probably have to use DEEPDECRYPT. As far as I understand, DEEPDECRYPT can pass on information to the crawled links only from the contents of the webpage, but not from its address/URL. Is that correct?
If that is correct, then the solution seems to be this:
  • Use DEEPDECRYPT to match the links as well as Custom_string (capital letter!) as well as all_text_between_them. We have to match all_text_between_them because as far as I understand DEEPDECRYPT can only match contiguous strings. Is my understanding correct?
  • Modify the links using REWRITE. This includes my planned modifications mentioned above as well as: remove all_text_between_them and replace Custom_string by custom_string (make first letter lowercase). To make the first letter lowercase, I'll need 26 REWRITE rules: to replace A by a, B by b, and so on. Having 26 REWRITE rules will probably be slow? edit: I'll try to use \l
Do you plan to make the LinkCrawler more powerful? For example DEEPDECRYPT could pass on info from the URL and not only from the page contents. Or REWRITE (or another command) could allow things like adding 32 to the ASCII code of the first letter of Custom_string, so that it is made lowercase. Or what about adding a text box to the JDownloader settings where you can code scripts directly (in any imperative language), with variables etc, without having to set up an IDE and boilerplate code to write and compile entire plugins? Thanks!

Last edited by smilies; 16.06.2019 at 16:19.
Reply With Quote
  #2  
Old 17.06.2019, 17:40
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@smilies:
1.) You should NOT use query to attach information as those are send to the server. you should put additional information to anchor #custom_string

2.) What exactly are you trying to achieve? We can help much better/faster if we know what you want to do. Best would be with real working example links.

3.) DEEPDECRYPT can only parse/find content but NOT modify the results. You can add multiple pattern and it must not be a contiguous string.

4.) REWRITE allows to customize the output url.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 17.06.2019, 17:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by smilies View Post
Do you plan to make the LinkCrawler more powerful?
The linkcrawler is already very very powerful but it doesn't make much sense to add complex methods/stuff/wishlist to linkcrawler rules because the rule engine will become very complex/incomprehensible.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 17.06.2019, 17:44
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by smilies View Post
Or what about adding a text box to the JDownloader settings where you can code scripts directly (in any imperative language), with variables etc, without having to set up an IDE and boilerplate code to write and compile entire plugins? Thanks!
You can also use the eventscripter and write code in javascript, see https://board.jdownloader.org/showthread.php?t=70525

When you can code in java, there is no real reason NOT to write plugins in java as you have access to full/complex/modern language. It's not comparatively to try to add all those full/complex/modern language possibilities to static rules (linkcrawler) or very basic/limited javascript engine.

You want to add 32 to the ascii code of the first letter. next person wants to check the 2nd letter and compare to 5th letter and then add 8th letter....and so on. doesn't make any sense to add such functionality to linkcrawler rules. Either your ideas can be created with eventscripter and supported javascript or must be done with a native java plugin that has nearly *unlimited* possibilities
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 08:19.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.