JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 24.08.2022, 02:41
accelheim accelheim is offline
Bandwidth Beast
 
Join Date: Jan 2020
Posts: 137
Default Pixiv.net - Not all images analyzed

This user has over 500 galleries, but only 128 packets are analysed for me.

Link:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #2  
Old 24.08.2022, 11:19
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

@accelheim: Have you entered your pixiv account in JDownloader? not all images are available when not being logged in
You may also use cookie login method, see https://support.jdownloader.org/Know...n-instructions
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 24.08.2022, 19:02
accelheim accelheim is offline
Bandwidth Beast
 
Join Date: Jan 2020
Posts: 137
Default

Quote:
Originally Posted by Jiaz View Post
@accelheim: Have you entered your pixiv account in JDownloader? not all images are available when not being logged in
You may also use cookie login method, see **External links are only visible to Support Staff**...
Yes, I am logged in with the cookie method. I had to scan this user twice to get all images:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

The other user in my OP I scanned 5 times and still didn't get all images.
Reply With Quote
  #4  
Old 24.08.2022, 20:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

@accelheim: Thanks for the feedback. will look at it by tomorrow
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 25.08.2022, 12:17
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,693
Default

Indeed they got some kind of rate-limit.
I've added a limit of one request every 500ms.
Try again after the next CORE-update.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #6  
Old 31.08.2022, 16:24
accelheim accelheim is offline
Bandwidth Beast
 
Join Date: Jan 2020
Posts: 137
Default

I've tried again, and it seems to be analyzing more packages now, but lots are still missing.

Like this user with 897 galleries/packages:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

I've now analyzed it 3 times. First time I got about 840 packages. Second time 5 new ones. Third time 4 new ones.
Reply With Quote
  #7  
Old 31.08.2022, 16:31
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

@accelheim: can you please provide a debug log, see https://support.jdownloader.org/Know...d-session-logs
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 31.08.2022, 20:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

@accelheim: issue is caused by rate limit/crawl protection. I've set higher request limits and limit concurrent use of plugin, maybe this helps but of course takes longer to complete
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 31.08.2022, 20:35
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

@accelheim: wait for next update, I've tweaked the request limits and looks like your example link can be crawled in one go but of course slower
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 05.09.2022, 17:24
accelheim accelheim is offline
Bandwidth Beast
 
Join Date: Jan 2020
Posts: 137
Default

Quote:
Originally Posted by Jiaz View Post
@accelheim: wait for next update, I've tweaked the request limits and looks like your example link can be crawled in one go but of course slower
It looks like it's working now! Thank you.
Reply With Quote
  #11  
Old 05.09.2022, 17:46
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

Quote:
Originally Posted by accelheim View Post
It looks like it's working now! Thank you.
Thanks for the feedback and you're welcome!
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 23:45.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.