JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #21  
Old 15.11.2020, 18:56
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

Do not thank me, thank him
__________________
JD-Dev & Server-Admin
Reply With Quote
  #22  
Old 16.11.2020, 18:38
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,927
Default

Very interesting that there are collections of saved items available similar to Facebook (well - it's Facebook anyways^^).

At this moment these ones are not supported and also I don't have a test-account available with such collections.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #23  
Old 19.11.2020, 03:37
onyxlee onyxlee is offline
Vacuum Cleaner
 
Join Date: Sep 2020
Posts: 19
Default

Thanks for the update, you can feel free to poke around in my account that I shared with you, or you can do it in your own account. It's OK if the collections are not supported, as long as the general saved posts can be downloaded, that will be GREAT! Cheers :p
Reply With Quote
  #24  
Old 19.11.2020, 18:53
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,927
Default

Got it working.
Feel free to test it once the next update gets released.

I'll leave your account in my test JD for now.
You can kill my session in your Instagram account whenever you've tested the changes and are fine with them.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #25  
Old 19.11.2020, 20:52
onyxlee onyxlee is offline
Vacuum Cleaner
 
Join Date: Sep 2020
Posts: 19
Default

It's lovely to hear this great news, I can't thank you enough! But for some reason, the link grabber can't grab anything. (it can still grab all the content under a specific account, just not the saved posts.) My Instagram account is in the account manager and everything looks fine. I made sure to update JD and even restarted it after the update. Anything I might have missed?

Last edited by onyxlee; 19.11.2020 at 20:57.
Reply With Quote
  #26  
Old 19.11.2020, 21:52
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,927
Default

Which URL did you add?
If you want to crawl all items of a user, you need to add:
instagram.com/username
If you want to grab your own saved stuff you have to add:
instagram.com/YourOwnUsername/saved/
To grab your own saved items, you of course need to have your own instagram account added to JD first.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #27  
Old 20.11.2020, 06:51
onyxlee onyxlee is offline
Vacuum Cleaner
 
Join Date: Sep 2020
Posts: 19
Default

Quote:
Originally Posted by pspzockerscene View Post
If you want to grab your own saved stuff you have to add:
instagram.com/YourOwnUsername/saved/
To grab your own saved items, you of course need to have your own instagram account added to JD first.
-psp-
Hi PSP. this one doesn't work for me:
instagram.com/MyOwnUsername/saved/
I already have my account added to JD.

I also tested instagram.com/MyOwnUsername/saved/all-posts/, which is the "collection of all the saved post", also didn't work.

It seems like the link grabber worked for a second, but without any error messages, it just gave up without getting any links.
Reply With Quote
  #28  
Old 20.11.2020, 13:46
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

@onyxlee: Please provide a debug log, see https://support.jdownloader.org/Know...d-session-logs
Enable Settings->Advanced Settings->Log.debugmodeenabled
restart JDownloader, reproduce the issue, create log and post shown logID here
__________________
JD-Dev & Server-Admin
Reply With Quote
  #29  
Old 20.11.2020, 14:50
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,927
Default

Fixed - sorry, my mistake!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #30  
Old 22.11.2020, 10:25
onyxlee onyxlee is offline
Vacuum Cleaner
 
Join Date: Sep 2020
Posts: 19
Default

Omg!!! It worksssss!!!!!!!!!!!!!!!! Tearssssssss!!! Waited for this for my whole life!!!

Thank you psp!!! Thank you jiaz!!! Merry Christmas, hahaha!

Last edited by onyxlee; 22.11.2020 at 10:28.
Reply With Quote
  #31  
Old 23.11.2020, 10:45
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

Thanks for the feedback
All thanks goes to psp
__________________
JD-Dev & Server-Admin
Reply With Quote
  #32  
Old 24.11.2020, 00:25
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,927
Default

Quote:
Originally Posted by onyxlee View Post
Omg!!! It worksssss!!!!!!!!!!!!!!!! Tearssssssss!!! Waited for this for my whole life!!!
Wohoo I've never seen such a crazy reaction in our forum

Quote:
Originally Posted by onyxlee View Post
Thank you psp!!! Thank you jiaz!!! Merry Christmas, hahaha!
No problem - thanks for trusting me and thanks for your feedback

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #33  
Old 12.12.2020, 00:03
onyxlee onyxlee is offline
Vacuum Cleaner
 
Join Date: Sep 2020
Posts: 19
Default

Hi, psp, just had a very crucial discovery that I have to share with you right away.

So my most likely theory is, Instagram very likely disabled my account after I downloaded hundreds of Instagram posts with my own account.

The details:
After I shared with you my spare account (account B), Instagram only sent an email asking me whether the login in your country was permitted. Since then Instagram didn't bother me on this account at all.

After the function worked, I signed in my primary account (account A) on JDownloader, crawled my saved posts, then downloaded all those saved posts with my account A. Then, probably a few days or a week after that, my account was disabled. I can't log in, nor see my posts, so I can't be sure whether someone hacked into my account and did something terrible.

(Instagram said I violated the community guideline, but they didn't mention using a crawler to go through your own account and download posts. They did mention using bots or 3rd party services to massively like your posts to boost the traffic, which is a violation to their rules, which I never did as well.)

Instagram asked me to verify my identity, I did so. Nothing happened. (Let's skip how awful Instagram's customer support is.) I had to report my issue to them as a business owner, not just a typical user. They came back to me with the weirdest request, asking me to handwrite some specific number, hold it against my face with both hands, and then take a picture (I guess with the third hand) and send it back to them. I haven't heard from them since then.

I have been going through all the possibilities. If I ignore the possibility of a hacker compromising my account, the only possible reason might be me downloading my posts with my own account.

The solution is pretty simple. Just use the function you developed recently as is to crawl all the links. Before you download them, disable the Instagram account in the account manager. That's it. Then you can download all you want as anonymous.

Of course, this is only my theory. Without my account, I can't further confirm it. But it's so far the one that makes the most sense.

I would like to share it with you and hopefully everyone who might use this function, which is a lifesaver for me and many people. Sincerely hope nobody has to deal with the worst customer support of the world, Instagram.
Reply With Quote
  #34  
Old 14.12.2020, 14:14
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

I've created a ticket for this,
optional try to download without account first and only fallback to account if required
that should help a lot in this situation
__________________
JD-Dev & Server-Admin
Reply With Quote
  #35  
Old 14.12.2020, 17:23
onyxlee onyxlee is offline
Vacuum Cleaner
 
Join Date: Sep 2020
Posts: 19
Default

Quote:
Originally Posted by Jiaz View Post
I've created a ticket for this,
optional try to download without account first and only fallback to account if required
that should help a lot in this situation
Yes, can't believe how tight Instagram is running their service right now. I will update you whenever I have some new discoveries.
Reply With Quote
  #36  
Old 14.12.2020, 18:01
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,927
Default

I've tried to optimize this handling without the need of another setting.
The following information is simplified:

The host plugin will only login whenever needed e.g. if a direct-URL times out and needs to be re-freshed.
The crawler will still always login at this moment but if wanted, that could also be optimized more.

Most of all times and especially for users who do not use JD to mass-download Instagram items, JD will really only need the account for the initial crawling - it will store the picture direct-URLs for downloading.
And again:
In theory, Instagram should e.g. put a token in those direct-URLs so that when downloading them, they can assign these http requests to the initial account -> Maybe unlikely but possible!

Also, login is required for the "original quality download" so to minimize login requests furthermore, deactivate this in Settings -> Plugins -> instagram.com (default = off already)

Everything I've written here has certain edge cases e.g.:
- If crawling your own saved posts, best quality will always be grabbed without requiring extra http requests
- Refreshing direct-URLs of user story elements always requires an active account

Also, no matter how far we optimize this, you can always trigger their rate-limits or an account ban by simply adding tons of items - there is no permanent solution for this!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 14.12.2020 at 18:04.
Reply With Quote
  #37  
Old 14.12.2020, 18:59
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,927
Default

Hm okay I've optimized the crawler too.
Now it will only login if needed and if the situation is unclear e.g. profile URL, it will try without account first and retry with account if the profile is private and an account is available.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #38  
Old 14.12.2020, 19:16
onyxlee onyxlee is offline
Vacuum Cleaner
 
Join Date: Sep 2020
Posts: 19
Default

Thanks, PSP, you are right, in order not to upset Instagram and trigger some rate limits, I will now only download 50 items or so every time.

Too bad that Instagram doesn't list these limits anywhere, I used to use some chrome extension to grab my saved posts, every time I download thousands of posts without triggering any bans. Seems like those days are gone. (These extensions can skip a huge part of the posts without letting you know, so I haven't used them for a long time.)
Reply With Quote
  #39  
Old 14.12.2020, 19:31
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,927
Default

Quote:
Originally Posted by onyxlee View Post
Too bad that Instagram doesn't list these limits anywhere, ...
They have no reason to do this

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #40  
Old 15.12.2020, 15:29
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

Quote:
Originally Posted by onyxlee View Post
Too bad that Instagram doesn't list these limits anywhere, I used to use some chrome extension to grab my saved posts, every time I download thousands of posts without triggering any bans. Seems like those days are gone. (These extensions can skip a huge part of the posts without letting you know, so I haven't used them for a long time.)
They want you to use their platform, not to crawl it
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 09:42.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.