#1
|
|||
|
|||
[LinkCrawler Rules] Request to add fapello
Hi guys
The site is not supported See screenshot I want to add fapello To download photos and videos and everything examples **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** The site is full of everything |
#2
|
||||
|
||||
Hi,
a plugin for this simple website is not needed. You can easily make it work using 1-2 LinkCrawler rules. Please read the linked docs and if you're having trouble understanding it, let us know If you need help with that.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#3
|
|||
|
|||
My friend I don't know I tried I didn't succeed
Can you help me This is my first time using LinkCrawler rules If you want to download this full profile how? **External links are only visible to Support Staff****External links are only visible to Support Staff** If you want to download one image or one video, how? **External links are only visible to Support Staff****External links are only visible to Support Staff** or **External links are only visible to Support Staff****External links are only visible to Support Staff** |
#4
|
|||
|
|||
I have another question
Do I have to make a rule for each profile? |
#5
|
||||
|
||||
Yes.
No. Here is a simple rule which can crawl images/videos from links to single posts: Code:
[ { "enabled": true, "updateCookies": true, "logging": false, "maxDecryptDepth": 1, "name": "example rule for fapello.com single posts", "pattern": "**External links are only visible to Support Staff**, "rule": "DEEPDECRYPT", "packageNamePattern": "<title>(.*?)</title>", "deepPattern": "\"(**External links are only visible to Support Staff**]+)" } ] pastebin.com/raw/Hx2jm3Wr Add this rule to JD and test it. If your JD auto-recognizes images/videos whenever you add single post links, it's working as it should. You need to do the following steps to crawl complete profiles: 1. Open the profile you want to crawl in your browser. 2. Scroll all the way down so you can see all posts. 3. Press CTRL + A followed by CTRL + C to copy all links to all posts and add them to JD.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#6
|
|||
|
|||
Thanks for your help
I put The code is as it is in this place Is it correct as in the picture (1) After placing the code, I tried to experiment When crawling a photo or video He doesn't crawl until after two or three tries until he crawls to something else As in picture (2) **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** And when I scroll down and copy all links Crawls into her low-resolution photos Example **External links are only visible to Support Staff****External links are only visible to Support Staff** Not high quality pictures + other photos As in picture (3) This is the code for the page Code:
**External links are only visible to Support Staff**www.w3.org%2F2000%2Fsvg%22%3E%0A%20%20%20%20%3Ccircle%20fill%3D%22none%22%20stroke%3D%22%23e5e5e5%22%20stroke-width%3D%222%22%20cx%3D%2210%22%20cy%3D%2210%22%20r%3D%227%22%20%2F%3E%0A%3C%2Fsvg%3E%0A **External links are only visible to Support Staff** |
#7
|
||||
|
||||
You didn't make any mistakes!
Indeed that rule missed some .mp4 files and didn't grab all available image qualities. I've updated the rule accordingly: Code:
[ { "cookies" : [], "deepPattern" : "\"(**External links are only visible to Support Staff**]+)|[^\"]+\\.mp4)", "formPattern" : null, "id" : 1659528266827, "maxDecryptDepth" : 1, "name" : "example rule for fapello.com single posts", "packageNamePattern" : "<title>(.*?)</title>", "passwordPattern" : null, "pattern" : "**External links are only visible to Support Staff**, "rewriteReplaceWith" : null, "rule" : "DEEPDECRYPT", "enabled" : true, "logging" : false, "updateCookies" : true } ] pastebin.com/raw/CHRurTN8
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 25.10.2022 at 17:42. |
#8
|
|||
|
|||
Hi, I just tried this script but it's not working. Not for me at least. All that it does is to grab some (not all) mp4 vids. The rest of the vids and all the pics are offline (but they aren't, of course). I hope you can help me. Thank you!
|
#9
|
||||
|
||||
Please provide example URLs which you've used to test that rule and describe exactly, which items were grabbed and which ones were missing.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#10
|
|||
|
|||
I used the script as you posted it and I tried with several links but it is the same for all.
As example **External links are only visible to Support Staff****External links are only visible to Support Staff** or **External links are only visible to Support Staff****External links are only visible to Support Staff** but all i get is: A bit better with some account like **External links are only visible to Support Staff****External links are only visible to Support Staff** I tried some random with vids and, like I said, only a few vids were downloaded |
#11
|
||||
|
||||
@CatHypno
Maybe we should start from the beginning and go through this one by one: The rule I've posted works on URLs that go to single posts and it will extract the individual images/videos. Before doing anything we should make sure that the rule is still working. Websites can change at any point of time, rendering such rules/scripts/plugins useless Example for such a single post: fapello.com/jennifer-lopez/507/ As we can see it looks like the rule is returning only broken links while that image is clearly available and should be downloadable. The direct URL to that picture is: fapello.com/content/j/e/jennifer-lopez/1000/jennifer-lopez_0507.jpg -> Immediately I see that the "deepPattern" of that rule does not allow links with "j" but only with "i" so we'll update the pattern in that rule accordingly: Code:
"deepPattern": "\"(**External links are only visible to Support Staff**j|l|k)/[^\"]+)|[^\"]+\\.mp4)", Next step: You seem to be wanting to crawl complete profiles. While you can add a profile URL and the JDownloader deep-parser will find some single post URLs which will then be handled by said rule, JDownloader will only find all posts of the first page since it does not have a crawler for this website. In this case I'd recommend the following method(s) to collect all single post links and let the LinkCrawler rule do the rest then: https://support.jdownloader.org/Know...orted-websites I'll soon add a followup post to this one in which I will try to improve said LinkCrawler rule even further.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 25.10.2022 at 17:44. |
#12
|
||||
|
||||
Now additionally to what I wrote in my last post I've noticed two things with that rule that could be improved:
Code:
[ { "deepPattern": "=\"(https?://[^\"]+)\"( target=\"_blank\" class=\"uk-align-center\">| /></video>)", "maxDecryptDepth": 1, "name": "example rule for fapello.com single posts", "packageNamePattern": "<title>(.*?)( - Fapello)?</title>", "pattern": "**External links are only visible to Support Staff**, "rule": "DEEPDECRYPT", "enabled": true, "logging": false, "updateCookies": true } ] pastebin.com/raw/DZsZ16qB Edit by Jiaz: small updated rule, see pastebin.com/f0ncSPcX
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by Jiaz; 26.10.2022 at 12:41. |
#13
|
|||
|
|||
Did some tests and everything seems to be working very well. Thank you!
|
#14
|
||||
|
||||
Thanks for your feedback.
Just keep in mind that that website can change at any time, rendering that rule useless.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#15
|
|||
|
|||
Welcome pspzockerscene
Why don't you add it to Plugin Greetings pspzockerscene and Jiaz you are kings |
#16
|
||||
|
||||
Because:
- Its a simple website so a pugin is not needed - The demand for a plugin for this website did not seem to be very high
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#17
|
|||
|
|||
Request to add improved Fapello support
I would like to be able to crawl for the content on a specfic Fapello profile: **External links are only visible to Support Staff****External links are only visible to Support Staff**
Looking back at a previous post on the JD forums, I noticed a previous LinkCrawler rule that was created (https://board.jdownloader.org/showpo...7&postcount=12), and while this managed to crawl some links, it only managed 32 out of the 186 total media under that Fapello profile. Is there any way to create a new/improve the rule so that it is able to fetch all the deep links, and not just the first 32? |
#18
|
||||
|
||||
Merged fapello.com threads.
@Adrian Using the rule and these instructions (+browser addon "Link Gopher"), I was able to add all 186 items very easily.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#19
|
|||
|
|||
Thank you, I didn't know that such an extension exists. I wonder if one day this functionaility can be adapted for the jDownloader browser extension to supplement the existing LinkCrawler functionality, which failed to fetch the links for all 186 files.
|
#20
|
||||
|
||||
It fails because the *more* links are dynamically loaded via javascript and not directly available via html source and yes, there are plans for such feature but unknown when it will be worked on.
__________________
JD-Dev & Server-Admin |
#21
|
||||
|
||||
*This is not to be seen as a bug as dynamic fetching has never been supported via LinkCrawler rules and/or myjd browser-extension.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#22
|
|||
|
|||
I've been playing around with this for a few hours now, trying to get it to be as efficient as possible. However, it seems to be grabbing unrelated images and placing them in the correct folder with the correct image.
E.g. example folder - example.img - wrongimage.img when I want it to be example folder - example.img Edit: I'd also like to point out that sometime it doesn't even grab the related image..Anyone got some insight? Cheers Last edited by Geekyfire; 30.12.2022 at 17:00. |
#23
|
||||
|
||||
@Geekyfire: can you please share example link and your rules, so we can check and help you with them. Either post here or send to support@jdownloader.org
__________________
JD-Dev & Server-Admin |
#24
|
|||
|
|||
@Jiaz,
Seems to have corrected itself, thanks for replying, though! |
#25
|
||||
|
||||
@Geekyfire: Thanks for the feedback! In case you need further help, just provided requested information
__________________
JD-Dev & Server-Admin |
#26
|
|||
|
|||
Quote:
Link "Edit by Jiaz" expired. Thanks! |
#27
|
||||
|
||||
For me, the rule provided inside my last post is working just fine and I got no idea which parts of it were changed by Jiaz back then.
Please describe your problem and provide example URLs.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
Thread Tools | |
Display Modes | |
|
|