JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 10.07.2020, 20:33
armindvd armindvd is offline
DSL Light User
 
Join Date: Apr 2020
Posts: 30
Default Access to member area by cookies

Hi

I want to download all video and photo files from a member area of the website (it is a single page area with load more button). I added following rule to the Link crawler rule but it does not work.

Code:
[{
  "id" : 1586007825050,
  "name" : "MyVideos",
  "enabled" : true,
  "logging" : true,
  "maxDecryptDepth" : 1,
  "domainPattern" : ".*fancentro\\.com",
   "rule" : "DEEPDECRYPT",
  "packageNamePattern" : null,
  "formPattern" : null,
  "deepPattern" : null,
"cookies": [ ["__ax","1"], ["ConsentSettings", "2"] ,["exo_uid", "1"],
"updateCookies": true
}]
I have some more cookies which i remove them here. And These are sample link for images and videos:

Spoiler:
**External links are only visible to Support Staff****External links are only visible to Support Staff**



Spoiler:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #2  
Old 13.07.2020, 13:13
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,143
Default

Hi,

1. The cookies shown in your example rule don't really seem personalized.
2. Your website might use "Basic Authentification" which would be easier than cookies.
3. Please send me the URL to the login area of that website via PN so I can check it out - I won't be able tp help you just using the ifo you've provided so far.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #3  
Old 13.07.2020, 18:38
armindvd armindvd is offline
DSL Light User
 
Join Date: Apr 2020
Posts: 30
Default

Quote:
Originally Posted by pspzockerscene View Post
Hi,

1. The cookies shown in your example rule don't really seem personalized.
2. Your website might use "Basic Authentification" which would be easier than cookies.
3. Please send me the URL to the login area of that website via PN so I can check it out - I won't be able tp help you just using the ifo you've provided so far.

-psp-

I sent you a message with all details.
Reply With Quote
  #4  
Old 14.07.2020, 14:16
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,143
Default

@armindvd
I've answered your PN already but here again in public:
Your PN only contains your cookies - to check your request I need your full username & password for that website.
You can of course change it afterwards!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #5  
Old 14.07.2020, 15:46
armindvd armindvd is offline
DSL Light User
 
Join Date: Apr 2020
Posts: 30
Default

Quote:
Originally Posted by pspzockerscene View Post
@armindvd
I've answered your PN already but here again in public:
Your PN only contains your cookies - to check your request I need your full username & password for that website.
You can of course change it afterwards!

-psp-
I thought it should be enough to login to site. credentials submitted.
Reply With Quote
  #6  
Old 14.07.2020, 16:25
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,143
Default

Hi again,

here is a simple rule that will accept "feeds" URLs and crawl video content.
Of course this can be extended to grab other stuff too:
pastebin.com/DZtAkgX8
Also, I've noticed that it is possible to download the "paid" content even without logging in which means even my test-rule will work fine.

Also, the only cookie really needed to be logged in is the "snapcentro_auth" cookie.
You can grab this from your browser and replace the "CENSORED" in my example rule with the content of said cookie.

Here is the rule again so we won't use it.
Our boardsoftware will censor URLs for users which is why it is so complicated and also why you should use the rule I've linked via pastebin above.

Spoiler:

Code:
[ {
  "enabled" : true,
  "cookies" : [ ["snapcentro_auth", "CENSORED"] ],
  "updateCookies" : true,
  "logging" : false,
  "maxDecryptDepth" : 1,
  "name" : "fancentro.com example rule for single videos with cookies",
  "pattern" : "https?://fancentro\\.com/[^/]+/feed/\\d+",
  "rule" : "DEEPDECRYPT",
  "packageNamePattern" : "<title>(.*?)</title>",
  "passwordPattern" : null,
  "formPattern" : null,
  "deepPattern" : "\"(https?://[^\"]+\\.mp4[^\"]+)",
  "rewriteReplaceWith" : null
} ]


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #7  
Old 14.07.2020, 18:01
armindvd armindvd is offline
DSL Light User
 
Join Date: Apr 2020
Posts: 30
Default

Quote:
Originally Posted by pspzockerscene View Post
Hi again,

here is a simple rule that will accept "feeds" URLs and crawl video content.
Of course this can be extended to grab other stuff too:
pastebin.com/DZtAkgX8
Also, I've noticed that it is possible to download the "paid" content even without logging in which means even my test-rule will work fine.

Also, the only cookie really needed to be logged in is the "snapcentro_auth" cookie.
You can grab this from your browser and replace the "CENSORED" in my example rule with the content of said cookie.

Here is the rule again so we won't use it.
Our boardsoftware will censor URLs for users which is why it is so complicated and also why you should use the rule I've linked via pastebin above.

Spoiler:

Code:
[ {
  "enabled" : true,
  "cookies" : [ ["snapcentro_auth", "CENSORED"] ],
  "updateCookies" : true,
  "logging" : false,
  "maxDecryptDepth" : 1,
  "name" : "fancentro.com example rule for single videos with cookies",
  "pattern" : "https?://fancentro\\.com/[^/]+/feed/\\d+",
  "rule" : "DEEPDECRYPT",
  "packageNamePattern" : "<title>(.*?)</title>",
  "passwordPattern" : null,
  "formPattern" : null,
  "deepPattern" : "\"(https?://[^\"]+\\.mp4[^\"]+)",
  "rewriteReplaceWith" : null
} ]


-psp-
Thanks for your help.

After you logged in, the system will generate video and image files links and then you can download them wherever you want but the links will expire after almost 30 minutes. so you need to login again.

If I enter a direct link from the feed such as

**External links are only visible to Support Staff**
it is working without problem. but is it possible to crawl all the link inside the root feed ? for example

**External links are only visible to Support Staff****External links are only visible to Support Staff**

is it possible to extend deepPattern to find multiple file type ? if yes. could you please give me an example?

Last edited by armindvd; 14.07.2020 at 18:54.
Reply With Quote
  #8  
Old 15.07.2020, 13:44
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,143
Default

Hi again,

1. For your "feed" URL, the contents are dynamically loaded.
This means it is impossible to do this via LinkCrawler rule.
A "real" JD crawler plugin would be better for that.

2. Yes sure you can extend the deepPattern to match more - its all just Regular Expressions (that's the word to google).
There also exists a neat online tool for testing RegExes:
regex101.com

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 13:21.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.