JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 23.07.2021, 17:30
denylx denylx is offline
Wind Gust
 
Join Date: Aug 2019
Posts: 43
Default [LinkCrawler Rule] Add support for preggophilia.com

Right now images from the threads in the forum can only be viewed and downloaded with an account.

Jd2 has no problem with external hoster links in the threads but embedded images are not crawled by linkgrabber without account login support.

Images in this **External links are only visible to Support Staff**thread are not crawled by linkgrabber.

Image example links-
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #2  
Old 23.07.2021, 19:18
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,156
Default

You have to create a linkcrawler rule for that board, so JDownloader can use existing cookies/session and access the images/attachments.
Please see
https://support.jdownloader.org/Know...kcrawler-rules
https://support.jdownloader.org/Know...le-deepdecrypt

In case you need further help, please ask but please try for yourself first
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 25.07.2021, 12:37
denylx denylx is offline
Wind Gust
 
Join Date: Aug 2019
Posts: 43
Default

Quote:
Originally Posted by Jiaz View Post
You have to create a linkcrawler rule for that board, so JDownloader can use existing cookies/session and access the images/attachments.
Please see
**External links are only visible to Support Staff**...
**External links are only visible to Support Staff**...

In case you need further help, please ask but please try for yourself first
Yeah.. I did have a look at these rules but its quite complicated and guides linked are not really beginner friendly. Help will be much appreciated.
Reply With Quote
  #4  
Old 26.07.2021, 14:18
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,881
Default

Hi again,
please send your login credentials for the mentioned website to us via support@jdownloader.org.

We will then try to help you by providing a LinkCrawler rule containing your login cookies of that website.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #5  
Old 26.07.2021, 18:57
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,156
Default

@denylx: thanks, I've assigned the mail to pspzockerscene
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 26.07.2021, 19:32
denylx denylx is offline
Wind Gust
 
Join Date: Aug 2019
Posts: 43
Default

Quote:
Originally Posted by Jiaz View Post
@denylx: thanks, I've assigned the mail to pspzockerscene
okay
Reply With Quote
  #7  
Old 27.07.2021, 12:49
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,881
Default

Hi again,
here is a simple LinkCrawler Rule that will work for single direct-URLs of that forum.
You need to replace "CENSORED" (without "") with the value of your own "mybbuser" cookie.

Code:
[
  {
    "enabled": true,
    "cookies": [
      [
        "mybbuser",
        "CENSORED"
      ]
    ],
    "updateCookies": true,
    "logging": false,
    "maxDecryptDepth": 1,
    "name": "example rule preggophilia.com",
    "pattern": "**External links are only visible to Support Staff**,
    "rule": "DIRECTHTTP"
  }
]
Rule as plaintext for easier copy & paste:
pastebin.com/dBGyUDa1

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #8  
Old 27.07.2021, 13:14
denylx denylx is offline
Wind Gust
 
Join Date: Aug 2019
Posts: 43
Default

Quote:
Originally Posted by pspzockerscene View Post
Hi again,
here is a simple LinkCrawler Rule that will work for single direct-URLs of that forum.
You need to replace "CENSORED" (without "") with the value of your own "mybbuser" cookie.

Code:
[
  {
    "enabled": true,
    "cookies": [
      [
        "mybbuser",
        "CENSORED"
      ]
    ],
    "updateCookies": true,
    "logging": false,
    "maxDecryptDepth": 1,
    "name": "example rule preggophilia.com",
    "pattern": "**External links are only visible to Support Staff**,
    "rule": "DIRECTHTTP"
  }
]
Rule as plaintext for easier copy & paste:
pastebin.com/dBGyUDa1

-psp-
Maybe a stupid question, but how do I get the value of mybbuser cookie?
Reply With Quote
  #9  
Old 27.07.2021, 13:49
denylx denylx is offline
Wind Gust
 
Join Date: Aug 2019
Posts: 43
Default

Okay..I'm getting this error when pasting it in linkcrawler rule

Code:
'[
  {
    "enabled": true,
    "cookies": [
      [
        "mybbuser",
        CENSORED
      ]
    ],
    "updateCookies": true,
    "logging": false,
    "maxDecryptDepth": 1,
    "name": "example rule preggophilia.com",
    "pattern": "**External links are only visible to Support Staff**,
    "rule": "DIRECTHTTP"
  }
]' is not a valid 'java.util.List<jd.controlling.linkcrawler.LinkCrawlerRuleStorable>'
Reply With Quote
  #10  
Old 27.07.2021, 13:50
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,881
Default

Quote:
Originally Posted by denylx View Post
Maybe a stupid question, but how do I get the value of mybbuser cookie?
For Chrome:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Even easier way (also Chrome):
Click on the "Lock-Icon" left to your address bar --> Cookies

For other browsers: Use a search engine of your choice to find instructions.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 27.07.2021 at 13:53.
Reply With Quote
  #11  
Old 27.07.2021, 13:52
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,881
Default

Quote:
Originally Posted by denylx View Post
Okay..I'm getting this error when pasting it in linkcrawler rule
1. Do NEVER!! Post your cookies publicly!
It allows other users to use your account!
I've edited your post accordingly.

2. You've mis-interpreted my post and destroyed the format of that rule.
Add the value in between of the "" --> Leave "" in there just like the rest of this structure.

It is a json obejct.
You can use online tools like the following one to edit/verify json structures:
jsoneditoronline.org

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #12  
Old 27.07.2021, 14:29
denylx denylx is offline
Wind Gust
 
Join Date: Aug 2019
Posts: 43
Default

Thank you so much for all the help. Everything seems to be working well.
Reply With Quote
  #13  
Old 27.07.2021, 14:33
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,881
Default

Thanks for your feedback.
See our LinkCrawler instructions in case you need more such rules for other websites in the future.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #14  
Old 27.07.2021, 14:45
denylx denylx is offline
Wind Gust
 
Join Date: Aug 2019
Posts: 43
Default

Quote:
Originally Posted by pspzockerscene View Post
Thanks for your feedback.
See our **External links are only visible to Support Staff**... in case you need more such rules for other websites in the future.

-psp-
The guides make so much more sense now that I have a frame of reference.
Reply With Quote
  #15  
Old 27.07.2021, 15:16
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,881
Default

Quote:
Originally Posted by denylx View Post
The guides make so much more sense now that I have a frame of reference.
You also get that frame of reference by simply looking for more examples like this in our forum - there are a lot

Thanks for your feedback!
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #16  
Old 27.07.2021, 15:41
denylx denylx is offline
Wind Gust
 
Join Date: Aug 2019
Posts: 43
Default

Quote:
Originally Posted by pspzockerscene View Post
You also get that frame of reference by simply looking for more examples like this in our forum - there are a lot

Thanks for your feedback!
Okay now that I have download 100+ images from the board, I see that a lot of images downloaded by JD are corrupt. The same images when downloaded through the web browser are completely fine. What might be causing this issue?

Image link - https://i.imgur.com/MjiBywH.png
Reply With Quote
  #17  
Old 27.07.2021, 15:42
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,881
Default

I was not able to reproduce this in my tests.
Try the following:
1. Set max. chunks- and downloads to 1.
2. Try to manually only download a hand full of images via JD and see if that also happens.

They might have some kind of bot protection in place or you got a hardware issue.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #18  
Old 27.07.2021, 16:15
denylx denylx is offline
Wind Gust
 
Join Date: Aug 2019
Posts: 43
Default

Quote:
Originally Posted by pspzockerscene View Post
I was not able to reproduce this in my tests.
Try the following:
1. Set **External links are only visible to Support Staff**... to 1.
2. Try to manually only download a hand full of images via JD and see if that also happens.

They might have some kind of bot protection in place or you got a hardware issue.

-psp-
Setting the values to 1 worked.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 13:14.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.