Hi, I am trying to create my very first set of link crawler rules. My goal is to crawl through sequentially numbered sites and grab all available download links. The URLs that I am interested are like this:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
...
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
...
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Page 124 is the final one.

I took the time to visit the regex101 website in order to create and test the following REGEX expression, which I think fits my needs: **External links are only visible to Support Staff****External links are only visible to Support Staff**

http:\/\/www\.webcamrecordings\.com\/modelSearch\/emma_lu1\/page\/([1-9]|[1-9][0-9]|10[0-9]|11[0-9]|12[0-4])$

Accordingly, I have looked at older threads on this forum and copied their templates to fashion my own Link Crawler rule:

[ {
  "enabled" " true",
  "maxDecryptDepth" : 2,
  "id" : 1485801268989,
  "name" : "Emma_Lu1 WebCamRec"
  "pattern" : "http:\/\/www\.webcamrecordings\.com\/modelSearch\/emma_lu1\/page\/([1-9]|[1-9][0-9]|10[0-9]|11[0-9]|12[0-4])$"
  "rule" : "DEEPDECRYPT"
  "packageNamePattern" : null,
  "deepPattern" : null,
  "rewriteReplaceWith" : null
} ]

Nevertheless, I was unable to find a single universal tutorial explaining the significance of these keywords. The importance of the "id" number, "packageNamePattern" and "deepPattern" is particularly confusing. In any case, the above rule does not crawl the URLS that I want. Instead, I get an error message instructing me that the above is not a valid LinkCrawlerRule, so there must be something subtle that I have missed. I would appreciate any help on this matter. Hopefully, I am trying to use Link Crawler in the way it was intended.

[raztoki]

first of all, if you want the total amount of pages there is zero need to supply complicated pattern like that. You would only do so if you want ONLY those pages say between pages 1 and 1000.

Since you want all you just need to use \d+

id should be random generated, you can leave it out and it should insert it.
rewrite you could leave out also
deepdecrypt length I believe should only be 1 (just that given page number right?)

[tony2long]

Id will be created automatically, packageNamePattern to get the packageName from the page, deepPattern to get what you want in the page (I think).

One of the video that I checked is hosted at publish2.me that say: This file is available only for premium members.

Rule example that doesn't give error (catches many pictures):

[ {
  "enabled" : true,
  "maxDecryptDepth" : 0,
  "name" : "webcamrecordings.com",
  "pattern" : "**External links are only visible to Support Staff**,
  "rule" : "DEEPDECRYPT",
  "packageNamePattern" : null,
  "deepPattern" : "class=\"mp4\"><a href=\"([^\"]+)\""
} ]

Originally Posted by tony2long

Id will be created automatically, packageNamePattern to get the packageName from the page, deepPattern to get what you want in the page (I think).

One of the video that I checked is hosted at publish2.me that say: This file is available only for premium members.

Rule example that doesn't give error (catches many pictures):

[ {
"enabled": true,
"maxDecryptDepth": 2,
"name": "Emma_Lu1 WebCamRec",
"pattern": "**External links are only visible to Support Staff**,
"rule": "DEEPDECRYPT",
"packageNamePattern": null,
"deepPattern": null
} ]

I put this into the Link Crawler Rules in Advanced Settings and no error message was thrown. Yay!

Is there something else I need to do to actually initiate the crawling process?

[tony2long]

As usual, enable clipboard monitor and copy the link.

Am I going to have copy all 124 pages into the clipboard manually?

[tony2long]

That is the basic idea, but it might depend on your creativity.

[Jiaz]

@Pbl: you can modify the deepPattern to auto find the next page
Then you only have to add page 1 and JDownloader will be able to find the rest
In case you need help with that, let us know!

[tony2long]

Can we have double deepPattern in one rule? How?

[Jiaz]

Use or
(a|b)

[tony2long]

How to add protocol and domain for "next"?

[Jiaz]

[{
"enabled": true,
"maxDecryptDepth": 10,
"name": "webcamrecordings.com",
"pattern": "**External links are only visible to Support Staff**,
"rule": "DEEPDECRYPT",
"deepPattern": "(class=\"mp4\"><a href=\"([^\"]+)\"|<a href=\"(/modelSearch/.*?/page/\\d+)\")"
}]

This will find the videos and next pages

[Jiaz]

@Tony: use CODE tags instead of QUOTE! CODE keeps the escapings!

Thank you everyone for your help! I really appreciate it.

I actually solved this by setting the maxDecryptDepth to 125 and using linkfilter extensively. That probably took longer used more CPU resources, but I'll see if I can create a more efficient linkcrawler next time.

[Jiaz]

You're welcome