JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 01.08.2022, 00:56
400 400 is offline
Modem User
 
Join Date: Jul 2022
Posts: 4
Lightbulb [LinkCrawler Rules] Request to add fapello

Hi guys
The site is not supported See screenshot
I want to add fapello
To download photos and videos
and everything
examples
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
The site is full of everything
Attached Images
File Type: png 43643.PNG (107.0 KB, 1 views)
Reply With Quote
  #2  
Old 01.08.2022, 14:06
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,908
Default

Hi,
a plugin for this simple website is not needed.
You can easily make it work using 1-2 LinkCrawler rules.

Please read the linked docs and if you're having trouble understanding it, let us know If you need help with that.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #3  
Old 02.08.2022, 06:56
400 400 is offline
Modem User
 
Join Date: Jul 2022
Posts: 4
Default

My friend I don't know I tried I didn't succeed
Can you help me This is my first time using LinkCrawler rules
If you want to download this full profile how?
**External links are only visible to Support Staff****External links are only visible to Support Staff**

If you want to download one image or one video, how?

**External links are only visible to Support Staff****External links are only visible to Support Staff** or **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #4  
Old 02.08.2022, 06:57
400 400 is offline
Modem User
 
Join Date: Jul 2022
Posts: 4
Default

I have another question
Do I have to make a rule for each profile?
Reply With Quote
  #5  
Old 02.08.2022, 11:57
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,908
Default

Quote:
Originally Posted by 400 View Post
Can you help me This is my first time using LinkCrawler rules
Yes.

Quote:
Originally Posted by 400 View Post
Do I have to make a rule for each profile?
No.

Here is a simple rule which can crawl images/videos from links to single posts:
Code:
[
  {
    "enabled": true,
    "updateCookies": true,
    "logging": false,
    "maxDecryptDepth": 1,
    "name": "example rule for fapello.com single posts",
    "pattern": "**External links are only visible to Support Staff**,
    "rule": "DEEPDECRYPT",
    "packageNamePattern": "<title>(.*?)</title>",
    "deepPattern": "\"(**External links are only visible to Support Staff**]+)"
  }
]
Rule as plaintext for easier copy & paste:
pastebin.com/raw/Hx2jm3Wr

Add this rule to JD and test it.
If your JD auto-recognizes images/videos whenever you add single post links, it's working as it should.

You need to do the following steps to crawl complete profiles:
1. Open the profile you want to crawl in your browser.
2. Scroll all the way down so you can see all posts.
3. Press CTRL + A followed by CTRL + C to copy all links to all posts and add them to JD.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #6  
Old 03.08.2022, 05:04
400 400 is offline
Modem User
 
Join Date: Jul 2022
Posts: 4
Default

Thanks for your help

I put
The code is as it is in this place Is it correct as in the picture (1)

After placing the code, I tried to experiment

When crawling a photo or video
He doesn't crawl until after two or three tries until he crawls to something else As in picture (2)

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**


And when I scroll down
and copy all links
Crawls into her low-resolution photos
Example
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Not high quality pictures
+ other photos
As in picture (3)
This is the code for the page
Code:
**External links are only visible to Support Staff**www.w3.org%2F2000%2Fsvg%22%3E%0A%20%20%20%20%3Ccircle%20fill%3D%22none%22%20stroke%3D%22%23e5e5e5%22%20stroke-width%3D%222%22%20cx%3D%2210%22%20cy%3D%2210%22%20r%3D%227%22%20%2F%3E%0A%3C%2Fsvg%3E%0A
**External links are only visible to Support Staff**
Attached Images
File Type: png 1.PNG (40.0 KB, 1 views)
File Type: png 2.PNG (20.9 KB, 1 views)
File Type: png 3.PNG (114.4 KB, 1 views)
Reply With Quote
  #7  
Old 03.08.2022, 15:07
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,908
Default

You didn't make any mistakes!

Indeed that rule missed some .mp4 files and didn't grab all available image qualities.
I've updated the rule accordingly:
Code:
[
 {
  "cookies"            : [],
  "deepPattern"        : "\"(**External links are only visible to Support Staff**]+)|[^\"]+\\.mp4)",
  "formPattern"        : null,
  "id"                 : 1659528266827,
  "maxDecryptDepth"    : 1,
  "name"               : "example rule for fapello.com single posts",
  "packageNamePattern" : "<title>(.*?)</title>",
  "passwordPattern"    : null,
  "pattern"            : "**External links are only visible to Support Staff**,
  "rewriteReplaceWith" : null,
  "rule"               : "DEEPDECRYPT",
  "enabled"            : true,
  "logging"            : false,
  "updateCookies"      : true
 }
]
Plaintext for easier copy & paste:
pastebin.com/raw/CHRurTN8
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 25.10.2022 at 17:42.
Reply With Quote
  #8  
Old 24.10.2022, 00:43
CatHypno CatHypno is offline
Modem User
 
Join Date: Oct 2022
Posts: 3
Default

Hi, I just tried this script but it's not working. Not for me at least. All that it does is to grab some (not all) mp4 vids. The rest of the vids and all the pics are offline (but they aren't, of course). I hope you can help me. Thank you!
Reply With Quote
  #9  
Old 24.10.2022, 18:06
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,908
Default

Please provide example URLs which you've used to test that rule and describe exactly, which items were grabbed and which ones were missing.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #10  
Old 25.10.2022, 16:47
CatHypno CatHypno is offline
Modem User
 
Join Date: Oct 2022
Posts: 3
Default

I used the script as you posted it and I tried with several links but it is the same for all.
As example **External links are only visible to Support Staff****External links are only visible to Support Staff**
or **External links are only visible to Support Staff****External links are only visible to Support Staff**

but all i get is:


A bit better with some account like
**External links are only visible to Support Staff****External links are only visible to Support Staff**



I tried some random with vids and, like I said, only a few vids were downloaded
Reply With Quote
  #11  
Old 25.10.2022, 17:41
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,908
Default

@CatHypno
Maybe we should start from the beginning and go through this one by one:
The rule I've posted works on URLs that go to single posts and it will extract the individual images/videos.
Before doing anything we should make sure that the rule is still working. Websites can change at any point of time, rendering such rules/scripts/plugins useless
Example for such a single post:
fapello.com/jennifer-lopez/507/

As we can see it looks like the rule is returning only broken links while that image is clearly available and should be downloadable.
The direct URL to that picture is:
fapello.com/content/j/e/jennifer-lopez/1000/jennifer-lopez_0507.jpg
-> Immediately I see that the "deepPattern" of that rule does not allow links with "j" but only with "i" so we'll update the pattern in that rule accordingly:
Code:
 "deepPattern": "\"(**External links are only visible to Support Staff**j|l|k)/[^\"]+)|[^\"]+\\.mp4)",
Now that rule should work and you can continue testing and update it until it's working for all single post URLs.


Next step:
You seem to be wanting to crawl complete profiles.
While you can add a profile URL and the JDownloader deep-parser will find some single post URLs which will then be handled by said rule, JDownloader will only find all posts of the first page since it does not have a crawler for this website.
In this case I'd recommend the following method(s) to collect all single post links and let the LinkCrawler rule do the rest then:
https://support.jdownloader.org/Know...orted-websites

I'll soon add a followup post to this one in which I will try to improve said LinkCrawler rule even further.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 25.10.2022 at 17:44.
Reply With Quote
  #12  
Old 25.10.2022, 17:52
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,908
Default

Now additionally to what I wrote in my last post I've noticed two things with that rule that could be improved:
  1. Package name always ends with " - Fapello"
  2. The deepPattern is quite complicated and prone to failure. Also it would return broken image URLs resulting in more results than you'd want to have.
Both possible issues have been addressed in the rule below:
Code:
[
  {
    "deepPattern": "=\"(https?://[^\"]+)\"( target=\"_blank\" class=\"uk-align-center\">| /></video>)",
    "maxDecryptDepth": 1,
    "name": "example rule for fapello.com single posts",
    "packageNamePattern": "<title>(.*?)(  - Fapello)?</title>",
    "pattern": "**External links are only visible to Support Staff**,
    "rule": "DEEPDECRYPT",
    "enabled": true,
    "logging": false,
    "updateCookies": true
  }
]
Rule as plaintext for easier copy & paste:
pastebin.com/raw/DZsZ16qB

Edit by Jiaz: small updated rule, see pastebin.com/f0ncSPcX
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by Jiaz; 26.10.2022 at 12:41.
Reply With Quote
  #13  
Old 01.11.2022, 01:11
CatHypno CatHypno is offline
Modem User
 
Join Date: Oct 2022
Posts: 3
Default

Did some tests and everything seems to be working very well. Thank you!
Reply With Quote
  #14  
Old 02.11.2022, 12:33
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,908
Default

Thanks for your feedback.
Just keep in mind that that website can change at any time, rendering that rule useless.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #15  
Old 05.11.2022, 03:50
the first the first is offline
I will play nice!
 
Join Date: Nov 2022
Posts: 2
Default

Welcome pspzockerscene
Why don't you add it to Plugin
Greetings pspzockerscene and Jiaz you are kings
Reply With Quote
  #16  
Old 07.11.2022, 17:41
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,908
Default

Quote:
Originally Posted by the first View Post
Why don't you add it to Plugin
Because:
- Its a simple website so a pugin is not needed
- The demand for a plugin for this website did not seem to be very high
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #17  
Old 03.12.2022, 01:19
Adrian Adrian is offline
Junior Loader
 
Join Date: Aug 2021
Posts: 12
Lightbulb Request to add improved Fapello support

I would like to be able to crawl for the content on a specfic Fapello profile: **External links are only visible to Support Staff****External links are only visible to Support Staff**

Looking back at a previous post on the JD forums, I noticed a previous LinkCrawler rule that was created (https://board.jdownloader.org/showpo...7&postcount=12), and while this managed to crawl some links, it only managed 32 out of the 186 total media under that Fapello profile.

Is there any way to create a new/improve the rule so that it is able to fetch all the deep links, and not just the first 32?
Reply With Quote
  #18  
Old 05.12.2022, 23:55
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,908
Default

Merged fapello.com threads.

@Adrian
Using the rule and these instructions (+browser addon "Link Gopher"), I was able to add all 186 items very easily.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #19  
Old 06.12.2022, 22:33
Adrian Adrian is offline
Junior Loader
 
Join Date: Aug 2021
Posts: 12
Thumbs up

Quote:
Originally Posted by pspzockerscene View Post
Merged fapello.com threads.

@Adrian
Using the rule and **External links are only visible to Support Staff**... (+browser addon "Link Gopher"), I was able to add all 186 items very easily.
Thank you, I didn't know that such an extension exists. I wonder if one day this functionaility can be adapted for the jDownloader browser extension to supplement the existing LinkCrawler functionality, which failed to fetch the links for all 186 files.
Reply With Quote
  #20  
Old 07.12.2022, 11:38
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Quote:
Originally Posted by Adrian View Post
I wonder if one day this functionaility can be adapted for the jDownloader browser extension to supplement the existing LinkCrawler functionality, which failed to fetch the links for all 186 files.
It fails because the *more* links are dynamically loaded via javascript and not directly available via html source and yes, there are plans for such feature but unknown when it will be worked on.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #21  
Old 08.12.2022, 00:55
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,908
Default

Quote:
Originally Posted by Adrian View Post
...which failed to fetch the links for all 186 files.
*This is not to be seen as a bug as dynamic fetching has never been supported via LinkCrawler rules and/or myjd browser-extension.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #22  
Old 30.12.2022, 16:27
Geekyfire Geekyfire is offline
Modem User
 
Join Date: Dec 2022
Posts: 3
Default

I've been playing around with this for a few hours now, trying to get it to be as efficient as possible. However, it seems to be grabbing unrelated images and placing them in the correct folder with the correct image.

E.g.

example folder
- example.img
- wrongimage.img

when I want it to be

example folder
- example.img

Edit: I'd also like to point out that sometime it doesn't even grab the related image..Anyone got some insight?

Cheers

Last edited by Geekyfire; 30.12.2022 at 17:00.
Reply With Quote
  #23  
Old 30.12.2022, 17:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@Geekyfire: can you please share example link and your rules, so we can check and help you with them. Either post here or send to support@jdownloader.org
__________________
JD-Dev & Server-Admin
Reply With Quote
  #24  
Old 31.12.2022, 05:34
Geekyfire Geekyfire is offline
Modem User
 
Join Date: Dec 2022
Posts: 3
Default

@Jiaz,

Seems to have corrected itself, thanks for replying, though!
Reply With Quote
  #25  
Old 31.12.2022, 19:12
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@Geekyfire: Thanks for the feedback! In case you need further help, just provided requested information
__________________
JD-Dev & Server-Admin
Reply With Quote
  #26  
Old 12.02.2024, 16:47
restland restland is offline
I will play nice!
 
Join Date: Jun 2021
Posts: 6
Default

Quote:
Originally Posted by pspzockerscene View Post
Now additionally to what I wrote in my last post I've noticed two things with that rule that could be improved:
  1. Package name always ends with " - Fapello"
  2. The deepPattern is quite complicated and prone to failure. Also it would return broken image URLs resulting in more results than you'd want to have.
Both possible issues have been addressed in the rule below:
Code:
[
  {
    "deepPattern": "=\"(https?://[^\"]+)\"( target=\"_blank\" class=\"uk-align-center\">| /></video>)",
    "maxDecryptDepth": 1,
    "name": "example rule for fapello.com single posts",
    "packageNamePattern": "<title>(.*?)(  - Fapello)?</title>",
    "pattern": "**External links are only visible to Support Staff**,
    "rule": "DEEPDECRYPT",
    "enabled": true,
    "logging": false,
    "updateCookies": true
  }
]
Rule as plaintext for easier copy & paste:
pastebin.com/raw/DZsZ16qB

Edit by Jiaz: small updated rule, see pastebin.com/f0ncSPcX
Can someone share the working rule?
Link "Edit by Jiaz" expired.
Thanks!
Reply With Quote
  #27  
Old 15.02.2024, 15:25
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,908
Default

For me, the rule provided inside my last post is working just fine and I got no idea which parts of it were changed by Jiaz back then.

Please describe your problem and provide example URLs.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 14:46.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.