JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 08.05.2020, 15:34
MrBojangles_ MrBojangles_ is offline
Junior Loader
 
Join Date: May 2020
Posts: 13
Default [LinkCrawler rule] Soundsnap.com Login & Captcha

Hi guys,

I'm trying to download a list of files from **External links are only visible to Support Staff**www.soundsnap.com. I don't need to use the crawler in JD because I've already compiled the list with another crawler.

The only thing is I need to pass login details to the site, and they also force a captcha after a certain amount of downloads (even though I respect their bandwidth by limiting download speed and only doing one simultaneous download)

The ideal solution is to bypass/automate the captcha altogether which would be amazing but if there's a way to make the captcha pop up in JD so I can click it and continue through the download list that would also be great.

Thanks so much!

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Example download page (they're all like this): **External links are only visible to Support Staff****External links are only visible to Support Staff**
Example download link: **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #2  
Old 08.05.2020, 15:37
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,711
Default

Hi,

the ideal case for this would probably be a plugin.
I can create a ticket for you but I promise it won't happen anywhere in the near future.
The 2nd best thing you could do is to add a link crawler rule for this host and add the cookies of that website so JD would be able to download these direct-URLs.

If you want me to attempt to create a link crawler rule for you for this site, please send me your logindata via PN and include example URLs.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3  
Old 08.05.2020, 15:46
MrBojangles_ MrBojangles_ is offline
Junior Loader
 
Join Date: May 2020
Posts: 13
Default

Hi,

Thanks for the amazingly quick reply, I'll pm the logindata over to you now.
Reply With Quote
  #4  
Old 08.05.2020, 16:31
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,711
Default

Hi again,

actually this website is quite simple.
All you have to do is the following:
1. Login via browser - get the current "PHPSSID" cookie of that website.
Getting the cookies of a browser is different for each browser.
In Chrome you can see the cookies of the website of the tab you're in via:
Click on the "lock" icon in the address bar --> Cookies --> Expand everything until you see "PHPSSID".

2. Add the cookie to the following String by replacing "CENSORED" (leave the "" before- and after!):
Code:
[ {
  "enabled" : true,
  "cookies" : [ [ "PHPSESSID", "CENSORED" ] ],
  "updateCookies" : false,
  "logging" : true,
  "maxDecryptDepth" : 0,
  "name" : "soundsnap.com example rule single audio download via loggin cookie",
  "pattern" : "https?://(www\\.)?soundsnap\\.com/audio/download/.+",
  "rule" : "DIRECTHTTP",
  "packageNamePattern" : null,
  "passwordPattern" : null,
  "formPattern" : null,
  "deepPattern" : null,
  "rewriteReplaceWith" : null
} ]
3. Put the above code into:
Settings --> Advanced Settings --> link crawler rules (2nd result)

4. Now to test, add one of the URLs you want to download --> It should be recognized and get displayed as online.

Once these URLs get displayed as offline again this means you will have to solve the captcha in your browser --> You will NOT have to update that cookie - by letting the links check again, it will work.

Now here is a little extra input:
You know that these URLs are online.
JDownloader checking them will cause you to run into that limit unnecessarily quickly.
What you can do to disable linkchecking in JD is this:
Settings --> Advanced Settings --> do link check --> Disable --> Restart JD
--> Keep in mind that for normal JD usage, this is not recommended (it will annoy you!) and you should set it back to the default setting once you do not need it anymore!

Now you should e.g. be able to add as much URLs as you like --> Download --> Solve captcha once needed --> Continue downloading

I've sent you the same "Link Crawler Rule" as above including my current soundsnap login cookie though it is a better idea to use your own!
You can now change the password of your soundsnap account (keep in mind, this may also invalidate the cookie in the rule in the PN I sent you!)


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 08.05.2020 at 16:52.
Reply With Quote
  #5  
Old 08.05.2020, 19:11
MrBojangles_ MrBojangles_ is offline
Junior Loader
 
Join Date: May 2020
Posts: 13
Default

Wow thanks for sorting that out, even just this step makes the process far less tedious!

As there are many many thousands to process I suspect I may get a bit bored with doing this captcha over and over which makes me think that a plugin is a better long term solution? You mentioned in PM that you and the developers don't take money to work on specific plugins...are there any regular users on this forum who could take this sort of work?

If not then I could try making it myself...I have some knowledge of js and can maybe figure things out by looking through existing examples. From what you saw of the site do you think the solution would be relatively easy or is this going to be just too difficult for a first-timer?

Thank you again for how quickly you came up with a solution, I'll of course donate to the project in gratitude!

EDIT: So I've run it a little and it's working (thank you) but I noticed that when the downloads start failing (i.e. I have to do the captcha) JD will carry on through the list...is there a setting I can make to stop all downloads after the first failure? This is to help avoid lots of failed connection attempts which might look suspicious.

EDIT 2: Sorted the above problem with a little script. I've since found out a lot more about how the site works. Can I maybe jump on IRC and chat to you guys about it because I think there could be a really simple plugin/script solution but would need a bit of assistance to get started?

Last edited by MrBojangles_; 09.05.2020 at 17:51.
Reply With Quote
  #6  
Old 11.05.2020, 14:14
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,711
Default

Quote:
Originally Posted by MrBojangles_ View Post
As there are many many thousands to process I suspect I may get a bit bored with doing this captcha over and over which makes me think that a plugin is a better long term solution? You mentioned in PM that you and the developers don't take money to work on specific plugins...are there any regular users on this forum who could take this sort of work?
No.

And you would have to solve it anyways ... or then use a captcha solver service like 9kw.eu.

Quote:
Originally Posted by MrBojangles_ View Post
If not then I could try making it myself...I have some knowledge of js and can maybe figure things out by looking through existing examples. From what you saw of the site do you think the solution would be relatively easy or is this going to be just too difficult for a first-timer?
Hard.

Quote:
Originally Posted by MrBojangles_ View Post
Thank you again for how quickly you came up with a solution, I'll of course donate to the project in gratitude!
No problem.

Quote:
Originally Posted by MrBojangles_ View Post
EDIT: So I've run it a little and it's working (thank you) but I noticed that when the downloads start failing (i.e. I have to do the captcha) JD will carry on through the list...is there a setting I can make to stop all downloads after the first failure? This is to help avoid lots of failed connection attempts which might look suspicious.

EDIT 2: Sorted the above problem with a little script. I've since found out a lot more about how the site works. Can I maybe jump on IRC and chat to you guys about it because I think there could be a really simple plugin/script solution but would need a bit of assistance to get started?
Sometimes we are online on our IRC.

Regarding JD going on whenever you need to enter a captcha:
There is no way around that other than:
1. Add a real plugin for this website.
or
2. Make an EventScripter Script which stops once one single file of this host gets an offline status.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 09:12.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.