JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 29.04.2018, 08:36
ozora
Guest
 
Posts: n/a
Default Pixiv plugin has problems with single images.

Hi,

The pixiv plugin seems to have issues determining the difference between a single image submission and a multi image submission when crawling galleries.

When crawling a gallery such as **External links are only visible to Support Staff****External links are only visible to Support Staff** , it correctly manages to find all of the submissions in the gallery by ID, but then treats them all as multi image submissions, which use a different URL than single image submissions. In the case of this gallery, that means most of the data links are broken and display "Plugin Defect!member_illust.php".

For example, one of the download links here was
**External links are only visible to Support Staff****External links are only visible to Support Staff**
which is broken, the correct link for this type of submission should be
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Even when crawling a single image link (eg **External links are only visible to Support Staff****External links are only visible to Support Staff**) directly, it returns "Plugin Defect!member_illust.php", so perhaps site changes have just left single image links broken.
Reply With Quote
  #2  
Old 30.04.2018, 19:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

I've optimized crawler to better handle single/multi-image links.
Please wait for next plugin update. Please enable debug mode in Settings-Advanced Settings-Log.debugmodeenabled and restart JDownloader.
In case you encounter Plugin defect again, please create a log, see https://support.jdownloader.org/Know...d-session-logs and post the shown logID here
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 28.05.2018, 02:14
Galuthien Galuthien is offline
Vacuum Cleaner
 
Join Date: May 2018
Posts: 18
Default Download Flash / animations in Pixiv with Jdownloader

Hi, I wanted to know if it is possible to download Flash / animations in Pixiv using the Jdownloader add-on ...

I can download all the images with the help of the add-on, but the animations are downloaded in a zip file, then, when I extract the file, several images (in sequence) are extracted from the animation I want to download.

Is it possible that the animations are downloaded in Jdownloader directly as swf files and not as zip files?
Reply With Quote
  #4  
Old 28.05.2018, 11:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

JDownloader can only download what is available and as far as I know the animations are just single images that are played back in browser, so no, no swf file available to download. But of course I can check, please provide example links
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 29.05.2018, 00:44
simon_7617118 simon_7617118 is offline
JD Alpha
 
Join Date: Oct 2015
Posts: 24
Default

Hi. I'm not sure if the plugin has been updated, but I'm experiencing the same issue as the first poster. Links from image sets seem to be crawled correctly, albeit with a lot of extraneous broken links. Single image posts are not getting picked up.
28.05.18 15.37.06 <--> 28.05.18 15.41.47 jdlog://1698854433151/
Thank you.
Reply With Quote
  #6  
Old 29.05.2018, 01:38
Galuthien Galuthien is offline
Vacuum Cleaner
 
Join Date: May 2018
Posts: 18
Default

Quote:
Originally Posted by Jiaz View Post
JDownloader can only download what is available and as far as I know the animations are just single images that are played back in browser, so no, no swf file available to download. But of course I can check, please provide example links
In this link there are some animations:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #7  
Old 29.05.2018, 11:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@simon_7617118: Please provide example links. I could not reproduce any issue. I did test gallery links and single image links without any error. Your log does not contain a single error?!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 29.05.2018, 11:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@Galuthien: The animation is provided as zip and JDownloader also shows the zip when you add the link/gallery.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 29.05.2018, 20:51
simon_7617118 simon_7617118 is offline
JD Alpha
 
Join Date: Oct 2015
Posts: 24
Default

This would be a good one to test because it's huge and has a mix of single and multi image entries:
**External links are only visible to Support Staff**...

There are 317 entries, but JD is grabbing only 87 multi-image entries. All the single posts are ignored when I use the gallery url, and produce a "Plugin Defect!member illust" entry when I input them individually.

This happens with every gallery. My JD version is up to date.
Reply With Quote
  #10  
Old 30.05.2018, 05:10
Galuthien Galuthien is offline
Vacuum Cleaner
 
Join Date: May 2018
Posts: 18
Default

@Jiaz: Yes, it happens to me...

Maybe in the future it would be good to consider correcting that flaw so that you can download animations from Pixiv with the addon in Jdownloader.
Reply With Quote
  #11  
Old 30.05.2018, 15:42
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@simon_7617118: can you please send the link to support@jdownloader.org
Also please provide a debug log for the plugin defect!
Enable Settings-Advanced Settings-https://support.jdownloader.org/Knowledgebase/Article/View/47/0/how-to-create-and-upload-session-logs and restart JDownloader
Now reproduce the issue and create log, see https://support.jdownloader.org/Know...d-session-logs
and post the shown logID or send to support@jdownloader.org
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 30.05.2018, 15:42
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@Galuthien: JDownloader already supports downloading the animations as zips. What flaw do you mean then?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 30.05.2018, 15:43
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

I cannot reproduce any issue, so please provide logs (see post above) and example links. In case you're using an account, please also send me logins to support@jdownloader.org, maybe the issues are linked to account usage as I'm not using any account at all
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 30.05.2018, 21:23
simon_7617118 simon_7617118 is offline
JD Alpha
 
Join Date: Oct 2015
Posts: 24
Default

The problem has become more complicated with some investigation.

Pixiv seems to have been rolling out a new layout to only some accounts. My old account login uses the new layout. I created a new account, and it uses the old layout. It seems this is random and one does not have control over it, nor can it be reverted.

Urls are the same, image urls are the same, but the layout of the page is different. I've copied them to Pastebin so you can take a look.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Old layout style. Account that get this old layout can still grab links correctly in JDownloader:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

New layout style. Accounts that get this cannot grab links properly in JDownloader:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

I hope this helps.
Reply With Quote
  #15  
Old 31.05.2018, 03:39
Galuthien Galuthien is offline
Vacuum Cleaner
 
Join Date: May 2018
Posts: 18
Default

Quote:
Originally Posted by Jiaz View Post
@Galuthien: JDownloader already supports downloading the animations as zips. What flaw do you mean then?
The problem is that after extracting the zip file does not appear the flash animation in swf format, but several images appear .jpg (in sequence) of the animation that I want to download ...

The normal thing would be that after downloading the animation in Jdownloader it should appear directly in .swf format, and not in a sequence of images in jpg format ...

I know a page where I can download the Pixiv animations, but I find it tedious to do it by that method since the page can not download all the animations at once, because I have to download the animations one by one ...
Reply With Quote
  #16  
Old 01.06.2018, 20:29
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@Galuthien: The animation does not exist as swf but the zip from server simply contains the single images and played back via javascript. There is no swf at all. I will try to find an easy solution that can put images into swf file.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 01.06.2018, 20:34
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@simon: Thanks for the information and html source

You do get new layout only with account, don't you?
Or also without any account at all?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #18  
Old 02.06.2018, 02:11
simon_7617118 simon_7617118 is offline
JD Alpha
 
Join Date: Oct 2015
Posts: 24
Default

Without account, the plugin works fine with one caveat: it's not grabbing anything past the first gallery page, because only registered users get full access.

My personal account, which is pretty old, has been rolled to the new layout. The account I just registered uses the old layout, but I assume somewhere down the line it will switch over. I bet this is why people with newer accounts aren't getting problems in JDownloader yet.

Multi-image posts aren't getting this problem yet at all, because as far as I can tell the layout for those posts are unchanged.
Reply With Quote
  #19  
Old 04.06.2018, 10:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Could you please provide some more pastebins? one/two? if possible old/new ?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 05.06.2018, 10:31
simon_7617118 simon_7617118 is offline
JD Alpha
 
Join Date: Oct 2015
Posts: 24
Default

Hello,

Single Pic
**External links are only visible to Support Staff****External links are only visible to Support Staff**
New format: **External links are only visible to Support Staff****External links are only visible to Support Staff**
Old Format: **External links are only visible to Support Staff****External links are only visible to Support Staff**

Single Pic
**External links are only visible to Support Staff****External links are only visible to Support Staff**
New format: **External links are only visible to Support Staff****External links are only visible to Support Staff**
Old Format: **External links are only visible to Support Staff****External links are only visible to Support Staff**

Multi-pic (12) (the images open in a second window/page 2)
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
New format page 1: **External links are only visible to Support Staff****External links are only visible to Support Staff**
New format page 2: **External links are only visible to Support Staff****External links are only visible to Support Staff**
Old format page 1: **External links are only visible to Support Staff****External links are only visible to Support Staff**
Old format page 2: **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #21  
Old 05.06.2018, 12:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Would it be possible to also provide screenshots how the pages look like?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #22  
Old 05.06.2018, 12:59
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Please test with next update!

Untested(no example link, no account):
-age restricted images, please test and in case it fails, please provide example links/new html source/+ account if required

Working fine for me (no account, old layout):
-member sites (member.php?id=xyz), please test and in case it fails, please provide example links/new html source
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 05.06.2018 at 13:03.
Reply With Quote
  #23  
Old 06.06.2018, 03:18
simon_7617118 simon_7617118 is offline
JD Alpha
 
Join Date: Oct 2015
Posts: 24
Default

Hello. I just tested with old and new accounts, both were able to grab all files without any discrepancies. Thank you very much, I assure you many users breathed a sigh of relief over this.
Reply With Quote
  #24  
Old 06.06.2018, 10:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Thanks for the feedback! Would be nice if you can check the *x rated* images as well? Add with and without and check if download works. Thanks!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #25  
Old 06.06.2018, 20:26
simon_7617118 simon_7617118 is offline
JD Alpha
 
Join Date: Oct 2015
Posts: 24
Default

The adult images are grabbed just fine. Linkgrabber will also ignore adult images when the profile at Pixiv is set to do so (changes are applied after toggling the account in JD). So everything is working as they should.

I wish I could tell you about the animations, but they're harder to come across.
Reply With Quote
  #26  
Old 06.06.2018, 23:42
Galuthien Galuthien is offline
Vacuum Cleaner
 
Join Date: May 2018
Posts: 18
Default

Still I can not download animations in Jdownloader.

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #27  
Old 07.06.2018, 11:18
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@Galuthien: What exactly doesn't work? Your example link works fine and downloads the zip as provided by pixiv.

@simon: thanks for the feedback!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #28  
Old 07.06.2018, 20:31
Galuthien Galuthien is offline
Vacuum Cleaner
 
Join Date: May 2018
Posts: 18
Default

I already mentioned it in this topic:

Quote:
Originally Posted by Galuthien View Post
The problem is that after extracting the zip file does not appear the flash animation in swf format, but several images appear .jpg (in sequence) of the animation that I want to download ...

The normal thing would be that after downloading the animation in Jdownloader it should appear directly in .swf format, and not in a sequence of images in jpg format ...

I know a page where I can download the Pixiv animations, but I find it tedious to do it by that method since the page can not download all the animations at once, because I have to download the animations one by one ...
If it is not possible to download animations in Jdownloader, then I will look for another method or maybe another program ...

For the moment I will have to settle on downloading animations one by one in this page: **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #29  
Old 08.06.2018, 11:31
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Can you provide some more animation example links? I can try to add support for animation converting via ffmpeg, eg to webm/gif/mp4.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #30  
Old 08.06.2018, 18:57
Galuthien Galuthien is offline
Vacuum Cleaner
 
Join Date: May 2018
Posts: 18
Default

Here I leave some example Links:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #31  
Old 08.06.2018, 19:51
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

thanks, will do some testing next week
__________________
JD-Dev & Server-Admin
Reply With Quote
  #32  
Old 09.06.2018, 00:05
Galuthien Galuthien is offline
Vacuum Cleaner
 
Join Date: May 2018
Posts: 18
Default

Ok. You can also try downloading Pixiv animations using a premium account. Maybe using a premium account if you can download the animations in jdownloader directly as a swf file.

I know that the theme is about Pixiv, but unlike Pixiv on other sites like Instagram and Deviantart if I can download animations using Jdownloader. Perhaps in Pixiv there is a restriction so that free users can not download the animations, or perhaps the restriction is for all users in general...
Reply With Quote
  #33  
Old 15.06.2018, 17:22
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default


@Galuthien: the animation doesn't exist as SWF. Pixiv provides it as a zip and the images are shown via javascript. no swf available. the site you mentioned does convert the images to swf.
JDownloader does not convert, it simply downloads the file as provided by the site/service
__________________
JD-Dev & Server-Admin
Reply With Quote
  #34  
Old 15.06.2018, 23:14
Galuthien Galuthien is offline
Vacuum Cleaner
 
Join Date: May 2018
Posts: 18
Default

Ok, it's a shame that unlike other sites like Instagram and Deviantart can not bedownload of Pixiv animations in Jdownloader directly.

I follow many artists who make animations in Pixiv and for me it would have been much more comfortable to download the animations in the same way that the images can be downloaded.

Thanks for trying. I will continue downloading animations one by one from the page that I mentioned previously...
Reply With Quote
  #35  
Old 18.06.2018, 13:48
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

I've linked the ticket because we will try to add support for it. Other sites work because they directly provide the video/animation in a playable format. Pixiv just provides a zip file that JDownloader can download. The ticket is about auto converting those images to a video file
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 18:34.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.