JDownloader Community - Appwork GmbH

Thread Tools Display Modes
Prev Previous Post   Next Post Next
Old 16.06.2019, 13:56
smilies smilies is offline
Junior Loader
Join Date: Jan 2019
Posts: 13
Default LinkCrawler Rules - passing information through DEEPDECRYPT

Hi, I'd like to crawl a webpage A (then B, then C, ...) for links but to modify those links. One of the modifications is to insert a custom_string into the links. This custom_string isn't constant, it's different for each webpage A,B,C. The webpage doesn't care what comes after the question mark in the URL, so I can make the URL look like original_url?custom_string if that helps. Besides, the webpage contains Custom_string, i.e. a version of custom_string where the first letter is capitalized.
  • To crawl the webpage for links, I probably have to use DEEPDECRYPT. As far as I understand, DEEPDECRYPT can pass on information to the crawled links only from the contents of the webpage, but not from its address/URL. Is that correct?
If that is correct, then the solution seems to be this:
  • Use DEEPDECRYPT to match the links as well as Custom_string (capital letter!) as well as all_text_between_them. We have to match all_text_between_them because as far as I understand DEEPDECRYPT can only match contiguous strings. Is my understanding correct?
  • Modify the links using REWRITE. This includes my planned modifications mentioned above as well as: remove all_text_between_them and replace Custom_string by custom_string (make first letter lowercase). To make the first letter lowercase, I'll need 26 REWRITE rules: to replace A by a, B by b, and so on. Having 26 REWRITE rules will probably be slow? edit: I'll try to use \l
Do you plan to make the LinkCrawler more powerful? For example DEEPDECRYPT could pass on info from the URL and not only from the page contents. Or REWRITE (or another command) could allow things like adding 32 to the ASCII code of the first letter of Custom_string, so that it is made lowercase. Or what about adding a text box to the JDownloader settings where you can code scripts directly (in any imperative language), with variables etc, without having to set up an IDE and boilerplate code to write and compile entire plugins? Thanks!

Last edited by smilies; 16.06.2019 at 16:19.
Reply With Quote

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 03:41.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.