JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 10.07.2021, 08:16
DeusExBestia DeusExBestia is offline
Super Loader
 
Join Date: Feb 2018
Posts: 29
Default Twitter Plugin occassionally entire media pages as offline?

So I'm lazy, and JDownloader has been instrumental in helping further my lazy hoarder ways, and with the assistance of a small EventScripter script that mgpai cobbled together for me I was able to convert Twitter posts into a url pointing directly to the account's media page, and the laziness became even more extreme as I no longer had to use notepad++ to mass convert links and then manually paste them into JDownloader, which was fantastic.

But recently, in either an update to JDownloader or to Twitter, or possibly even just the lack of updates for nearly a year to the iOS "MyJD Remote" App (which I know isn't made by y'all, just listing that as a potential reason as well), randomly these converted twitterCOM/<USERNAME>/media links show up as offline, despite being online. Not the individual files inside the /media pages, but the actual whole link itself to /media. This happens for both locked accounts and unlocked accounts.

I am unable to determine what causes it, as it appears to be sporadic and random. Sometimes the whole apparatus works, and sometimes I'm just inundated with offline links, and have to manually re-add them. I turned on debug logging yesterday morning, tested out adding a bunch of links, some of them went through, and about 5 got added to the offline folder, but can not figure out where in the logs to look for either where the handoff between the script went bad, or if the iOS app just pooped out and said list it as offline.

Any help would be appreciated.
Reply With Quote
  #2  
Old 11.07.2021, 06:43
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,449
Default

Please upload the log and post the log Id here.
__________________
FAQ: How to upload a Log
Reply With Quote
  #3  
Old 11.07.2021, 11:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,211
Default

@DeusExBestia: have you added your twitter account to JDownloader?
does this happen random or after specific amount of time? similiar to https://board.jdownloader.org/showthread.php?t=87857

check again with next update, I've updated plugin to refresh the token. most likely the lifetime of the guest token has been lowered by twitter, causing all the issues
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 11.07.2021, 19:15
DeusExBestia DeusExBestia is offline
Super Loader
 
Join Date: Feb 2018
Posts: 29
Default

Quote:
Originally Posted by tony2long View Post
Please upload the log and post the log Id here.
Which log though? There are several. Not sure which one to be looking in to see where the problem comes from.
Reply With Quote
  #5  
Old 11.07.2021, 19:20
DeusExBestia DeusExBestia is offline
Super Loader
 
Join Date: Feb 2018
Posts: 29
Default

Quote:
Originally Posted by Jiaz View Post
@DeusExBestia: have you added your twitter account to JDownloader?
does this happen random or after specific amount of time? similiar to **External links are only visible to Support Staff**...

check again with next update, I've updated plugin to refresh the token. most likely the lifetime of the guest token has been lowered by twitter, causing all the issues
I do have my account set up. I’m not entirely sure when it happens, I tested it out to see if it was after a set amount of time or if it was just random by sending links through the iOS share button then browsing to the link collector page but it doesn’t show what’s in queue, just what it is currently crawling or has already crawled. It looks like after a few links it starts puttering out and labeling links as offline.

I will update the FreeBSD install and see if it happens.

As of 1022 pacific time not seeing an update available for the FreeBSD install. So testing will have to wait.

Last edited by DeusExBestia; 11.07.2021 at 19:22. Reason: Add info.
Reply With Quote
  #6  
Old 12.07.2021, 12:35
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,211
Default

@DeusExBestia: Does this also happen when you add the link directly and not via iOS MyJD app?
Does the issue persist when you restart JDownloader?

Please create a debug log, enable Settings->Advanced Settings->Log.debugmodeenabled and restart JDownloader. Now wait for issue to happen again and create a log, see https://support.jdownloader.org/Know...d-session-logs
and post shown logID here
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 15.07.2021, 09:24
DeusExBestia DeusExBestia is offline
Super Loader
 
Join Date: Feb 2018
Posts: 29
Default

Quote:
Originally Posted by Jiaz View Post
@DeusExBestia: Does this also happen when you add the link directly and not via iOS MyJD app?
Does the issue persist when you restart JDownloader?

Please create a debug log, enable Settings->Advanced Settings->Log.debugmodeenabled and restart JDownloader. Now wait for issue to happen again and create a log, see **External links are only visible to Support Staff**...
and post shown logID here
So, digging through the logs, it looks like in the decrypter it's giving a rate limit exceeded error, (code 88), and that's what is putting the whole /<account>/media link into the offline package thing all by its lonesome.
Reply With Quote
  #8  
Old 15.07.2021, 11:44
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,211
Default

Can you provide a log for this error? then we can check/try to auto handle it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 20.07.2021, 19:38
DeusExBestia DeusExBestia is offline
Super Loader
 
Join Date: Feb 2018
Posts: 29
Default

Quote:
Originally Posted by Jiaz View Post
Can you provide a log for this error? then we can check/try to auto handle it
Sorry for the delay, it was my birthday this Sunday so I was trying to get everything in order and then recover for a few days from all the fun.

I do believe this should be the log

Code:
jdlog://9615825302851
I did my best to ensure that once I was able to get it to dump everything into the /offline package I didn't add anything else to this headless install.
Reply With Quote
  #10  
Old 21.07.2021, 15:35
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,910
Default

Quote:
Originally Posted by DeusExBestia View Post
Sorry for the delay, it was my birthday this Sunday so I was trying to get everything in order and then recover for a few days from all the fun.
Belated best wishes

Quote:
Originally Posted by DeusExBestia View Post
I did my best to ensure that once I was able to get it to dump everything into the /offline package I didn't add anything else to this headless install.
Your log shows you running into one of twitters rate-limits:
Code:
{"errors":[{"message":"Rate limit exceeded","code":88}]}
This is quite rare but it can happen if you keep adding URLs.
Possible workarounds:
- Try again later
- Get a new IP and try again
- Add less URLs --> Wait in between

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #11  
Old 21.07.2021, 16:35
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,910
Default

After our next twitter plugin update, you'll see a more exact errormessage in the dummy item that gets added to the linkgrabber if an error occurs.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #12  
Old 23.07.2021, 08:06
DeusExBestia DeusExBestia is offline
Super Loader
 
Join Date: Feb 2018
Posts: 29
Default

Quote:
Originally Posted by pspzockerscene View Post
After our next twitter plugin update, you'll see a more exact errormessage in the dummy item that gets added to the linkgrabber if an error occurs.
-psp-
Is there a way to set it up so it's delayed, or counts until the rate limit is reached and then waits until it is cleaned? Like, I know the Furaffinity plugin doesn't have this issue, at least not that I've seen, yet.

Also, in curiousness, does the rate limit count against pictures I've already downloaded? Like it counts just for crawling? So if I have an account that had 800 media files last time I added them, and this time they have 80 new media files on top of that same old 800, will i hit 880 on the rate limit or just the 80?
Reply With Quote
  #13  
Old 23.07.2021, 09:29
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,211
Default

DeusExBestia: with latest update JDownloader has set request limits.

Only twitter knows about the rate limit. Most often the number of requests in time x is checked,so let's see if the new request limit helps.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 23.07.2021, 12:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,910
Default

Quote:
Originally Posted by DeusExBestia View Post
Also, in curiousness, does the rate limit count against pictures I've already downloaded? Like it counts just for crawling? So if I have an account that had 800 media files last time I added them, and this time they have 80 new media files on top of that same old 800, will i hit 880 on the rate limit or just the 80?
You can do some research on this topic but in the end it is just like Jiaz wrote: Only twitter knows about their rate-limits and there is no reason to expose them 100%.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #15  
Old 24.07.2021, 07:28
DeusExBestia DeusExBestia is offline
Super Loader
 
Join Date: Feb 2018
Posts: 29
Default

Quote:
Originally Posted by pspzockerscene View Post
You can do some research on this topic but in the end it is just like Jiaz wrote: Only twitter knows about their rate-limits and there is no reason to expose them 100%.

-psp-
Quote:
Originally Posted by Jiaz View Post
DeusExBestia: with latest update JDownloader has set request limits.

Only twitter knows about the rate limit. Most often the number of requests in time x is checked,so let's see if the new request limit helps.
Thank you both of you for your answers and your hard work.
Reply With Quote
  #16  
Old 26.07.2021, 12:23
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,211
Default

@DeusExBestia: Please test and provide feedback if the new added requested limits do help
__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 04.08.2021, 08:39
DeusExBestia DeusExBestia is offline
Super Loader
 
Join Date: Feb 2018
Posts: 29
Default

Quote:
Originally Posted by Jiaz View Post
@DeusExBestia: Please test and provide feedback if the new added requested limits do help
So, I've given it about a week or so of testing. I now get packages in the download section labeled "Content offline!Rate limit exceeded" with the only item in the package being an item of the same name, which I can get the link to the account/media page I was trying to crawl.

Still can't determine exactly HOW MANY accounts to crawl is too many, but eventually I'll figure it out I guess.
Reply With Quote
  #18  
Old 04.08.2021, 10:47
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,211
Default

@DeusExBestia: those "Rate limit exceeded" links should remain in Linkgrabber section and you can retry(continue) them later via rightclick context menu
I will increase the request limits little more, maybe it helps
__________________
JD-Dev & Server-Admin
Reply With Quote
  #19  
Old 10.08.2021, 16:38
DeusExBestia DeusExBestia is offline
Super Loader
 
Join Date: Feb 2018
Posts: 29
Default

Quote:
Originally Posted by Jiaz View Post
@DeusExBestia: those "Rate limit exceeded" links should remain in Linkgrabber section and you can retry(continue) them later via rightclick context menu
I will increase the request limits little more, maybe it helps
So, I have a packagizer set up to automatically organize sources into their own folders, because why not? The packagizer setup also has auto start enabled, so unfortunately instead of leaving the content rate limited links I’ve sent in the link grabber queue to restart later, it puts them in the downloader queue with an error saying the file is offline. I’m not near my desktop so I can’t adjust the packagizer settings in the JDownloader client proper and export them for the headless unit.

Is there an error correction setting so it would start ones that are good to go and hold ones that are not? Like it automatically doesn’t add links that were already downloaded before?
Reply With Quote
  #20  
Old 10.08.2021, 17:14
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,910
Default

I did read and understand your post but a screenshot of your rule or your rule exported would be good.

I guess you could simply enable "is online" as condition to filter out failed crawler-dummy URLs.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 13:56.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.