#1
|
|||
|
|||
Link Crawler can get files but not found on download
The website in question is for example is this
**External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** I use this rule and manage to crawl the image files list Code:
[ { "enabled" : true, "cookies" : [ ], "updateCookies" : true, "logging" : false, "maxDecryptDepth" : 9, "id" : 1586231755944, "name" : "my rule", "pattern" : "**External links are only visible to Support Staff**, "rule" : "DEEPDECRYPT", "packageNamePattern" : null, "passwordPattern" : null, "formPattern" : null, "deepPattern" : "(?<=<p>| )<a href=\"([^\"]+theweb\\.tv/(?!(category|tag))\\S+(/|\\.jpg))\"", "rewriteReplaceWith" : null } ] But when I tried to download said files, it got error File not found on the download list. Am I missing something? Thank you for the help. Last edited by marvelfrozen; 07.04.2020 at 10:14. Reason: mask domain |
#2
|
||||
|
||||
Can you open the crawl result with browser?
__________________
FAQ: How to upload a Log |
#3
|
|||
|
|||
Right Click and open in browser opens the source url.
Is it supposed to open the image file instead? Last edited by marvelfrozen; 07.04.2020 at 10:53. |
#4
|
||||
|
||||
I can't test it because theweb.tv is not in the source of that example link.
In the source, the link after "Full size:" is the link that you should get.
__________________
FAQ: How to upload a Log |
#5
|
|||
|
|||
I sent you a private message. Please check.
This is the source (?) that I get if I crawl using the rule https://i.postimg.cc/zJgDKYjB/JDownl...hg-CV0k-Fk.png While this is the one if I didn't use the rule, but copy block the gallery https://i.postimg.cc/c14NDpQf/JDownl...jst-K8x-LT.png Why is it not directing to the image file when I'm using the rule? |
#6
|
||||
|
||||
It seems that non zero "maxDecryptDepth" will make upside down the links.
So try with 2 rules, one for picture only with "maxDecryptDepth" = 0 and "deepPattern" : "Full size: <a href="([^"]+)""
__________________
FAQ: How to upload a Log |
#7
|
|||
|
|||
I have added a second rule, so the setup looks like this
Code:
[ { "enabled" : true, "cookies" : [ ], "updateCookies" : true, "logging" : false, "maxDecryptDepth" : 9, "name" : "rule 1", "pattern" : "**External links are only visible to Support Staff**, "rule" : "DEEPDECRYPT", "packageNamePattern" : null, "passwordPattern" : null, "formPattern" : null, "deepPattern" : "(?<=<p>| )<a href=\"([^\"]+theweb\\.tv/(?!(category|tag))\\S+(/|\\.jpg))\"", "rewriteReplaceWith" : null },{ "enabled" : true, "cookies" : [ ], "updateCookies" : true, "logging" : false, "maxDecryptDepth" : 0, "name" : "rule 2", "pattern" : "**External links are only visible to Support Staff**, "rule" : "DEEPDECRYPT", "packageNamePattern" : null, "passwordPattern" : null, "formPattern" : null, "deepPattern" : "Full size: <a href=\"([^"]+)\"", "rewriteReplaceWith" : null } ] I have also remove the .jpg from the first rule, but that results in the crawler not getting any files at all. Last edited by marvelfrozen; 07.04.2020 at 13:35. |
#8
|
||||
|
||||
Works fine here like this:
Code:
[ { "enabled" : true, "maxDecryptDepth" : 1, "name" : "modelblog.tv replace thumbnail URL to full image URL", "pattern" : "(https?://modelblog\\.tv/wp-content/uploads/\\d{4}/\\d{2}/.*)(-\\d+x\\d+)\\.jpg", "rule" : "REWRITE", "packageNamePattern" : null, "passwordPattern" : null, "formPattern" : null, "deepPattern" : null, "rewriteReplaceWith" : "$1.jpg" }, { "enabled" : true, "updateCookies" : true, "maxDecryptDepth" : 1, "name" : "modelblog.tv grab thumbnails from overview page", "pattern" : "https?://modelblog\\.tv/(?!wp-content)[^/]+/", "rule" : "DEEPDECRYPT", "packageNamePattern" : null, "passwordPattern" : null, "formPattern" : null, "deepPattern" : "(https?://modelblog\\.tv/wp-content/[^\"]+\\.jpg)", "rewriteReplaceWith" : null } ] -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#9
|
|||
|
|||
Thank you, that works just like I wanted.
|
#10
|
||||
|
||||
Thanks for your feedback.
-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
Thread Tools | |
Display Modes | |
|
|