JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 17.08.2020, 19:14
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default PornHub.com - Not all videos are downloaded from playlist

I tried downloading this playlist:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

I checked the JD link list and saw several videos were missing, some of them:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

and

**External links are only visible to Support Staff****External links are only visible to Support Staff**

I tried inserting them in JD manually and then they worked, but not from the playlist link.

How to fix this?
Reply With Quote
  #2  
Old 18.08.2020, 11:03
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Can anyone please help with this?

Thanks in advance.
Reply With Quote
  #3  
Old 18.08.2020, 14:20
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Hm I see a total of 14 pages with total number of items: 494.
Both of your missing items are found just fine here and both are located on page 2.
You can manually scroll through playlists by adding e.g. "?page=2" to the playlist-URL.

I don't see any issue here ...
Please keep in mind that the titles of the videos in JD may vary from the ones you see in browser - pornhub displays different titles for different languages.
Via JD you will usually get English video-titles of pornhub content.

Please post your log-ID here | bitte poste deine Log-ID hier.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #4  
Old 18.08.2020, 15:11
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Quote:
Originally Posted by pspzockerscene View Post
Hm I see a total of 14 pages with total number of items: 494.
Both of your missing items are found just fine here and both are located on page 2.
You can manually scroll through playlists by adding e.g. "?page=2" to the playlist-URL.

I don't see any issue here ...
Please keep in mind that the titles of the videos in JD may vary from the ones you see in browser - pornhub displays different titles for different languages.
Via JD you will usually get English video-titles of pornhub content.

Please post **External links are only visible to Support Staff**... | bitte poste **External links are only visible to Support Staff**....

-psp-
18.08.20 14.14.16 <--> 18.08.20 14.15.29 jdlog://5530225302851/

Last edited by accelheim; 18.08.2020 at 15:18.
Reply With Quote
  #5  
Old 18.08.2020, 15:30
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

I'm unable to reproduce your issue.

Please wait for the next CORE-update - I've added some more loggers to our pornhub crawler.
Please post a new log once you've received the update.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 18.08.2020 at 15:59.
Reply With Quote
  #6  
Old 18.08.2020, 15:50
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Quote:
Originally Posted by pspzockerscene View Post
I'm unable to reproduce your issue.

Please wait for the next CORE-update - I've added some more loggers to our pornhub crawler.
Please post a new log once you've received the update.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!


-psp-
Alright, thanks. How long until the update?

Does that mean you got every single video link from that playlist?
Reply With Quote
  #7  
Old 18.08.2020, 15:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Quote:
Originally Posted by accelheim View Post
Alright, thanks. How long until the update?
Please read the above linked article. Thanks.

Quote:
Originally Posted by accelheim View Post
Does that mean you got every single video link from that playlist?
I've checked my array of results and yes, it was definitely found.
Also, as said, it is on the 2nd page so it got added "right at the start".

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #8  
Old 18.08.2020, 17:45
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Quote:
Originally Posted by pspzockerscene View Post
Please read the above linked article. Thanks.


I've checked my array of results and yes, it was definitely found.
Also, as said, it is on the 2nd page so it got added "right at the start".

-psp-
I've now checked exactly how much videos are missing from that playlist. The playlist has 494 videos in it and JD grabs 477 packages and 486 links (why is the number here diferent as well?).
Reply With Quote
  #9  
Old 18.08.2020, 19:10
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Quote:
Originally Posted by accelheim View Post
[...]The playlist has 494 videos in it and JD grabs 477 packages and 486 links (why is the number here diferent as well?).
Some videos might have less available qualities and/o multiple videos ger merged into one package then.

By marking all and doing Strg + C you can copy all source-URLs.

You can then compare/check what's missing.

It is working just fine here ...

Again:
Please don't put any preassure on us - wait for the next update, try again and post a new log if the issue keeps happening.
If you can't wait, you can get our source code and build the current dev. version on your own:
https://jdownloader.org/knowledge/wi...nt/get-started

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #10  
Old 18.08.2020, 19:30
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Quote:
Originally Posted by pspzockerscene View Post
Some videos might have less available qualities and/o multiple videos ger merged into one package then.

By marking all and doing Strg + C you can copy all source-URLs.

You can then compare/check what's missing.

It is working just fine here ...

Again:
Please don't put any preassure on us - wait for the next update, try again and post a new log if the issue keeps happening.
If you can't wait, you can get our source code and build the current dev. version on your own:
**External links are only visible to Support Staff**...

-psp-
Thanks for the update. How exactly can I compare them after copying all the links?
Reply With Quote
  #11  
Old 18.08.2020, 19:36
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Well you somehow have to manually get a full list of the playlist and then compare against what JD has found.

Either that or you wait for the update and report back then.

Sure you can also grab our source code/fix it yourself or contribute code - we're open source.
(Yes I know, I've repeated myself here.)

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #12  
Old 18.08.2020, 20:35
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Quote:
Originally Posted by pspzockerscene View Post
Well you somehow have to manually get a full list of the playlist and then compare against what JD has found.

Either that or you wait for the update and report back then.

Sure you can also grab our source code/fix it yourself or contribute code - **External links are only visible to Support Staff**....
(Yes I know, I've repeated myself here.)

-psp-
Ok, thanks. One more question: Is it better to paste the link into the link collector, or "add links"? So there are the options "paste link", "add links" and just copying the link and it goes automatically.
Reply With Quote
  #13  
Old 19.08.2020, 11:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

It doesn't matter how you do it.

I prefer adding URLs via clipboard (copy & paste) as it is the easiest way for me.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #14  
Old 20.09.2020, 22:55
groolyfooly groolyfooly is offline
Baby Loader
 
Join Date: Sep 2020
Posts: 5
Default

I am also experiencing this. Recent update did not change it. Strangely, it produces a different number of files each time I try. The playlist has 861 files, but the most I have seen jdownloader crawl is 502. On two other attempts it was 9 and 63.
Reply With Quote
  #15  
Old 21.09.2020, 15:46
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Hi,

please post an example playlist URL so we can check this.
The last update of our pornhub crawler was: 2020-08-28

-psp-
EDIT

I've just tested with a random playlist containing 115 items and I'm unable to confirm your issue!


Please post your log-ID here | bitte poste deine Log-ID hier.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 21.09.2020 at 18:37.
Reply With Quote
  #16  
Old 22.09.2020, 01:11
groolyfooly groolyfooly is offline
Baby Loader
 
Join Date: Sep 2020
Posts: 5
Default

**External links are only visible to Support Staff****External links are only visible to Support Staff** here is that playlist
Reply With Quote
  #17  
Old 22.09.2020, 15:02
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Finds 864 items here on a total of 26 "pages" (full page = 36 items).

I guess youporn might be rate-limiting you based on country/IP/whatever which means without a log I'm unable to tell you what has happened there.

Please post your log-ID here | bitte poste deine Log-ID hier.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #18  
Old 28.09.2020, 02:13
groolyfooly groolyfooly is offline
Baby Loader
 
Join Date: Sep 2020
Posts: 5
Default

27.09.20 21.18.39 <--> 27.09.20 23.13.30 jdlog://1200325302851/

Here is my log-id
Reply With Quote
  #19  
Old 28.09.2020, 16:07
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Works just fine here.

Please turn on debug logging (see above linked log instructions) and post another log after the next CORE-update:

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 28.09.2020 at 17:20.
Reply With Quote
  #20  
Old 30.09.2020, 12:43
groolyfooly groolyfooly is offline
Baby Loader
 
Join Date: Sep 2020
Posts: 5
Default

30.09.20 08.20.32 <--> 30.09.20 09.43.13 jdlog://0070325302851/

What is strange is that I am pretty sure I saw it find more than 122 files in the little pop up status thing. But only 118/122 show in the linkgrabber tab.
Reply With Quote
  #21  
Old 30.09.2020, 17:13
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Your issue happens because:
Code:
HTTP/1.0 429 Too Many Requests
It won't be easy for us to just "fix" this as we cannot know their exact limits.
So far I've found a small inssue in our login handling which caused some avoidable http requests.
This may- or may not already fix your issue or at least let you crawl more items until you will hit that limit again.
This is not an uncommon issue ... you're adding a lot of items which their system detects as an "unusual behavior" thus limiting you based on your account- and/or IP.


Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
EDIT

After about 10 minutes, I got a total of 879 items.

9 of these are displayed as offline.

All of these 9 items are private videos which require you to be friends with the uploader so that you can watch them.
EDIT2

Also be sure to enable the following setting before adding your playlist-URL:
Settings -> Plugins -> pornhub.com -> "Enable fast linkcheck"
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 30.09.2020 at 17:24.
Reply With Quote
  #22  
Old 01.10.2020, 08:48
groolyfooly groolyfooly is offline
Baby Loader
 
Join Date: Sep 2020
Posts: 5
Default

Ah I see. I will try that thanks. I wonder if there is a way for me to limit the pace at which it crawls.

Yes, fast linkcheck was enabled during these logs.

Ah okay strangely I was watching the bubble the whole time and it did find all the videos, but in the linkgrabber they don't all show up. Maybe I have a linkgrabber setting wrong?

EDIT: ah it works now thanks. I think the problem was that I had previous partially completed attempts in the linkgrabber. I deleted those and tried again and it worked.

Last edited by groolyfooly; 01.10.2020 at 12:08.
Reply With Quote
  #23  
Old 01.10.2020, 14:47
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Thanks for your feedback.

We may further optimize that plugin in the future but it should be okay for now.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 14:53.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.