JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #41  
Old 05.05.2021, 22:13
mex mex is offline
Baby Loader
 
Join Date: Dec 2020
Posts: 6
Default

Quote:
Originally Posted by beronz View Post
Lol, coomers are EVERYWHERE.
Can't speak for everyone when I say this, but personally I don't use the instagram plugin to coom.
Reply With Quote
  #42  
Old 06.05.2021, 13:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 57,523
Default

@beronz
Thanks for your nice feedback!

I've checked our Instagram saved posts crawler and that seems to work just fine.
Also we've narrowed down the issue "many folders/packages" issue:
Seems like our crawler is randomly grabbing re-posts too.
These are from other users and are then (correctly) placed into separate packages.

We're still working on it.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #43  
Old 06.05.2021, 21:05
uzarluzar uzarluzar is offline
I will play nice!
 
Join Date: Apr 2021
Posts: 4
Default

Quote:
Originally Posted by pspzockerscene View Post
@beronz
Thanks for your nice feedback!

I've checked our Instagram saved posts crawler and that seems to work just fine.
Also we've narrowed down the issue "many folders/packages" issue:
Seems like our crawler is randomly grabbing re-posts too.
These are from other users and are then (correctly) placed into separate packages.

We're still working on it.

-psp-
Thank you. For me now there are still the separate packages (for images/videos posted by other users/accounts) but there is still the problem of unrelated images/videos in the main package where only the images/videos of that one user should be, and instead there appear to be images from other users who might have tagged the main user whose images I am trying to download.
Reply With Quote
  #44  
Old 07.05.2021, 13:23
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 57,523
Default

Quote:
Originally Posted by uzarluzar View Post
Thank you. For me now there are still the separate packages (for images/videos posted by other users/accounts) but there is still the problem of unrelated images/videos in the main package where only the images/videos of that one user should be
Yes as said it is still being worked on - we haven't published any bugfixes for this and I'm unsure whether this will happen this week.

If you urgently need to download an IG profile, you might need to use other tools as a temporary workaround until we're able to fix our plugin.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #45  
Old 09.05.2021, 16:16
goldensun87 goldensun87 is offline
DSL Light User
 
Join Date: Feb 2012
Posts: 33
Default

While you guys are working on this issue, if it is possible, please upgrade the Instagram crawler to also grab any and all existing Reels videos and IGTV videos when copying an account url. Before the bug occurred, I am not sure if the crawler was already grabbing these, so if that was the case, feel free to disregard this.
Reply With Quote
  #46  
Old 09.05.2021, 19:09
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 70,703
Default

@goldensun87: Please provide example links for Reels Video and IGTV Videos
__________________
JD-Dev & Server-Admin
Reply With Quote
  #47  
Old 09.05.2021, 20:58
goldensun87 goldensun87 is offline
DSL Light User
 
Join Date: Feb 2012
Posts: 33
Default

Quote:
Originally Posted by Jiaz View Post
@goldensun87: Please provide example links for Reels Video and IGTV Videos
Ok so, after I posted that message, I noticed that a Reels tab and IGTV tab have dedicated urls. However, copying these urls does not make the crawler isolate the Reels or IGTV videos. The crawler tries to grab everything on the account. And since the existing bug is not fixed yet, the buggy results occur.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Also, I want to bring up another bug which has existed with the Instagram crawler, regarding individual reels and individual IGTV videos. So, in IGTV and Reel urls, respectively, the "p" is replaced with "tv" and "reel", respectively. However, the Insta crawler is not able to work with these urls properly. Strangely, however, if the "tv" or "reel" is replaced with "p", and the link is copied as such, the crawler grabs the IGTV or reel video perfectly. I will provide example urls.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**

So, to recap:

1. Crawling of individual IGTV and Reel video urls should be fixed, although there is a workaround.

2. If possible, for account reels urls and account IGTV urls, the crawler should be fixed so that only reels or IGTVs are grabbed.

3. If it hasn't already been done, and if possible, upgrade the Insta crawler so that when an account url is copied, the crawler grabs all existing Reels and IGTVs on the account, in addition to all the regular posts.

Finally, Saved Collection urls currently have the same problem as the reels and IGTV urls. For starters, it should be noted that currently it is still only possible to create new collections on the Mobile version of Instagram. Once created and accessed on the Web version of Instagram, each collection has its own url, in the format "instagram.com/*username*/saved/*collection name*/" . No matter which collection url is copied, the Insta crawler grabs all saved posts on the account. Ironically, this is how I accidentally discovered a way to grab all saved posts, because in the recent past, grabbing of saved posts was bugged. Anyway, since saved posts are private for each user, I cannot provide example urls, one of you will have to test and confirm using your own accounts.
Reply With Quote
  #48  
Old 10.05.2021, 17:32
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 57,523
Default

1. Those single URLs are working just fine here no matter whethe I'm adding "/reel/..." or "/p/...".
Please post a DEBUG-log if you can reproduce this issue!

2. Our crawler never had the ability to crawl any of these "sections":
You can either crawl all items of a users' account and sort them afterwards in JD (doesn't that also contain all of these "REELS"?) or grab single items/posts.
EDIT
I've tested that with your example and my result was as expected:
After having crawled ALL items of a user, these also included ALL of these "reels"!
How is a "reel" defined? Is it just a video? So the "reels" section always contains all videos of a particular user?


3. See my answer/question in 2.

4. Grabbing your own saved items/collections:
Please either add a description on how to create own collections (is that also possible without using the app, only via website?)
If not possible:
Please send us your Instagram logindata via support@jdownloader.org and also provide example URLs to existing "saved items collections".
At this moment our crawler can only grab all saved items so it not being able to pickup specified collections is again not a bug but rather a feature suggestion of yours.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 10.05.2021 at 18:22.
Reply With Quote
  #49  
Old 11.05.2021, 10:33
Shmi Shmi is offline
Junior Loader
 
Join Date: Jan 2020
Posts: 10
Default

Did you guys bring a fix or was it instagram that changed it behaviour back? Because I was just able to grab an entire profile with more than 2500 posts (6370 files) (which wasn't possible the last days)

Also tested on a few other profiles that wouldn't work.
Now it works (public and private profiles).
Reply With Quote
  #50  
Old 11.05.2021, 11:35
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 70,703
Default

@Shmi: we found the underlying cause and fixed it. Thanks for the feedback
Waiting for some more feedback
__________________
JD-Dev & Server-Admin
Reply With Quote
  #51  
Old 11.05.2021, 12:34
PocasNuckie PocasNuckie is offline
Modem User
 
Join Date: May 2021
Posts: 4
Default

Tested with 3 profiles of different sizes (80, 250, 800 files) and so far so good.

Thanks A LOT for the effort put into this, the ig crawler is really important to me.
Reply With Quote
  #52  
Old 11.05.2021, 13:09
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 70,703
Default

@PocasNuckie: thanks for the feedback
__________________
JD-Dev & Server-Admin
Reply With Quote
  #53  
Old 11.05.2021, 14:38
j4sch4 j4sch4 is offline
Modem User
 
Join Date: May 2021
Posts: 2
Default

Yesss, for me it's working again. Yesterday I was able to download new images added to an Instagram account with 3729 images. I already downloaded 670 images a couple of months ago but stopped the download that time and wanted to download the rest (3059) last week. And last week it didn't work anymore, but now it's working as before and I was able to download the 3059 images. Many many thanks for this fast fix. JDownloader 2 is simply the best downloader there is, and that because of the hard working people behind it. You are doing a absolutely fantastic job, and I want to thank you all for this. And I am going to contribute right now, and hope you keep up the good work :-)
Reply With Quote
  #54  
Old 11.05.2021, 14:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 57,523
Default

@j4sch4
Thanks for your feedback!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #55  
Old 11.05.2021, 17:08
beronz beronz is offline
Super Loader
 
Join Date: Jun 2019
Posts: 29
Default

Quote:
Originally Posted by Jiaz View Post
@Shmi: we found the underlying cause and fixed it. Thanks for the feedback
Waiting for some more feedback
I've chosen Maddie Ziegler one, the biggest profile i've used for my test, and i can confirm that now works fine. Only the Maddie folder, with over 2500 posts (both videos and photos)
Excellent work as always
Reply With Quote
  #56  
Old 11.05.2021, 17:48
goldensun87 goldensun87 is offline
DSL Light User
 
Join Date: Feb 2012
Posts: 33
Default

Quote:
Originally Posted by pspzockerscene View Post
1. Those single URLs are working just fine here no matter whethe I'm adding "/reel/..." or "/p/...".
Please post a DEBUG-log if you can reproduce this issue!

2. Our crawler never had the ability to crawl any of these "sections":
You can either crawl all items of a users' account and sort them afterwards in JD (doesn't that also contain all of these "REELS"?) or grab single items/posts.
EDIT
I've tested that with your example and my result was as expected:
After having crawled ALL items of a user, these also included ALL of these "reels"!
How is a "reel" defined? Is it just a video? So the "reels" section always contains all videos of a particular user?


3. See my answer/question in 2.

4. Grabbing your own saved items/collections:
Please either add a description on how to create own collections (is that also possible without using the app, only via website?)
If not possible:
Please send us your Instagram logindata via support@jdownloader.org and also provide example URLs to existing "saved items collections".
At this moment our crawler can only grab all saved items so it not being able to pickup specified collections is again not a bug but rather a feature suggestion of yours.

-psp-
1. The "reel" and "tv" urls still do not work for me. That is ok, I suppose replacing them with "p" is not too much work xD .

2. Yes, it does seem to be, that when the account url is copied, any existing reels and IGTV are grabbed along with the regular posts. So, this also makes the individual url problem lesser of an issue.

To answer your questions, both IGTV and Reels are videos only. IGTV was implemented a while back. If I remember correctly, the primary purpose of IGTV was for users to be able to make videos of longer duration. However, at this point, I am not sure what are the current duration limit differences between "regular" video posts, and IGTV video posts. One thing that is definitely unique about IGTV, is that any video created and posted as an IGTV video, gets posted under a dedicated IGTV tab, which can be accessed on the user's main profile. Also, I believe any Lives that a user films and wants to save, automatically gets saved as an IGTV video.

Reels are basically Instagram's version and response to TikTok. Like IGTV videos, Reels also have a dedicated tab on users' profiles, and all videos uploaded as Reels will be posted under this tab. Also, both IGTV and Reels have dedicated algorithmic feeds, however at the moment, these feeds can only be accessed on the mobile version of Instagram (the app), not on the Desktop/Web version.

4. I honestly do not want to share login details. It's ok, the crawler grabbing all saved posts certainly is superior. And yes, wanting the crawler to grab particular collections is a feature request I guess. If it possible to implement, please do when you guys have the time. Do you want/need me to post the request in a dedicated feature request thread?

To answer your question, currently, new Saved Collections can only be created while using the mobile version of Instagram (the smartphone/tablet app). If you want to do it while on your desktop, there are both Firefox and Chrome browser extensions which can emulate the Instagram Mobile app, and you can create new Saved Collections in your account while working on your desktop/laptop.

Finally, to add to the existing feedback, after the latest JD update, the Instagram crawler is properly grabbing all posts/reels/IGTV from an account. Thank you for fixing this problem in a relatively short amount of time. Now that I have utilized the Packagizer to automatically add upload dates to all grabbed files, I will be scraping more accounts moving forward. So, I will keep an eye for any new bugs that occur in the future. If you don't mind saying, what was the underlying cause you guys found?
Reply With Quote
  #57  
Old 11.05.2021, 18:07
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 70,703
Default

@goldensun87: your example "reel" and "tv" urls work fine out of the box.Tested in 3 different JDownloader installations
Please provide a debug log for this. Enable Settings->Advanced Settings->Log.debugmodeenabled and restart JDownloader. Now try again and then create a log, see https://support.jdownloader.org/Know...d-session-logs
and post shown logID here
__________________
JD-Dev & Server-Admin
Reply With Quote
  #58  
Old 11.05.2021, 18:09
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 70,703
Default

Quote:
Originally Posted by goldensun87 View Post
4. I honestly do not want to share login details. It's ok, the crawler grabbing all saved posts certainly is superior. And yes, wanting the crawler to grab particular collections is a feature request I guess. If it possible to implement, please do when you guys have the time. Do you want/need me to post the request in a dedicated feature request thread?
Can you provide example links how such URLs look alike?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #59  
Old 11.05.2021, 18:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 70,703
Default

Quote:
Originally Posted by goldensun87 View Post
So, I will keep an eye for any new bugs that occur in the future. If you don't mind saying, what was the underlying cause you guys found?
error in queryHash parser, resulting in different search/api results
__________________
JD-Dev & Server-Admin
Reply With Quote
  #60  
Old 11.05.2021, 18:54
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 57,523
Default

@goldensun87
1. Still can't reproduce -> We need a log

2. Thanks for the extra information.
To be able to support this "/profileName/reels" we'd have to create another crawler function.
Depending on the demand this may make sense and/or also speeds up the process for you as JD would not have to crawl all account items first BUT at this moment it shouldn't be much more work for you either - only for our parser:
a. Add all items of a user.
b. Filter via type in Linkgrabber, uncheck everything except video
c. Done.
At least for us it would be easir only having to maintain one crawler functionality

4. I absolutely understand that.
Please do as Jiaz said and provide some of these URLs.
Alternatively you could create an extra account and add such "saved posts".
I've tested it with my test-account: Seems like it is not possible to create such "categories" via the IG website, maybe App-only...
Are they then displayed via website too once created in the IG app?
Regarding the underlying issue:
Already answered by Jiaz.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 14:09.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.