#1
|
|||
|
|||
Twitter Plugin occassionally entire media pages as offline?
So I'm lazy, and JDownloader has been instrumental in helping further my lazy hoarder ways, and with the assistance of a small EventScripter script that mgpai cobbled together for me I was able to convert Twitter posts into a url pointing directly to the account's media page, and the laziness became even more extreme as I no longer had to use notepad++ to mass convert links and then manually paste them into JDownloader, which was fantastic.
But recently, in either an update to JDownloader or to Twitter, or possibly even just the lack of updates for nearly a year to the iOS "MyJD Remote" App (which I know isn't made by y'all, just listing that as a potential reason as well), randomly these converted twitterCOM/<USERNAME>/media links show up as offline, despite being online. Not the individual files inside the /media pages, but the actual whole link itself to /media. This happens for both locked accounts and unlocked accounts. I am unable to determine what causes it, as it appears to be sporadic and random. Sometimes the whole apparatus works, and sometimes I'm just inundated with offline links, and have to manually re-add them. I turned on debug logging yesterday morning, tested out adding a bunch of links, some of them went through, and about 5 got added to the offline folder, but can not figure out where in the logs to look for either where the handoff between the script went bad, or if the iOS app just pooped out and said list it as offline. Any help would be appreciated. |
#2
|
||||
|
||||
Please upload the log and post the log Id here.
__________________
FAQ: How to upload a Log |
#3
|
||||
|
||||
@DeusExBestia: have you added your twitter account to JDownloader?
does this happen random or after specific amount of time? similiar to https://board.jdownloader.org/showthread.php?t=87857 check again with next update, I've updated plugin to refresh the token. most likely the lifetime of the guest token has been lowered by twitter, causing all the issues
__________________
JD-Dev & Server-Admin |
#4
|
|||
|
|||
Which log though? There are several. Not sure which one to be looking in to see where the problem comes from.
|
#5
|
|||
|
|||
Quote:
I will update the FreeBSD install and see if it happens. As of 1022 pacific time not seeing an update available for the FreeBSD install. So testing will have to wait. Last edited by DeusExBestia; 11.07.2021 at 19:22. Reason: Add info. |
#6
|
||||
|
||||
@DeusExBestia: Does this also happen when you add the link directly and not via iOS MyJD app?
Does the issue persist when you restart JDownloader? Please create a debug log, enable Settings->Advanced Settings->Log.debugmodeenabled and restart JDownloader. Now wait for issue to happen again and create a log, see https://support.jdownloader.org/Know...d-session-logs and post shown logID here
__________________
JD-Dev & Server-Admin |
#7
|
|||
|
|||
Quote:
|
#8
|
||||
|
||||
Can you provide a log for this error? then we can check/try to auto handle it
__________________
JD-Dev & Server-Admin |
#9
|
|||
|
|||
Quote:
I do believe this should be the log Code:
jdlog://9615825302851 |
#10
|
||||
|
||||
Quote:
Quote:
Code:
{"errors":[{"message":"Rate limit exceeded","code":88}]} Possible workarounds: - Try again later - Get a new IP and try again - Add less URLs --> Wait in between -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#11
|
||||
|
||||
After our next twitter plugin update, you'll see a more exact errormessage in the dummy item that gets added to the linkgrabber if an error occurs.
Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#12
|
|||
|
|||
Quote:
Also, in curiousness, does the rate limit count against pictures I've already downloaded? Like it counts just for crawling? So if I have an account that had 800 media files last time I added them, and this time they have 80 new media files on top of that same old 800, will i hit 880 on the rate limit or just the 80? |
#13
|
||||
|
||||
DeusExBestia: with latest update JDownloader has set request limits.
Only twitter knows about the rate limit. Most often the number of requests in time x is checked,so let's see if the new request limit helps.
__________________
JD-Dev & Server-Admin |
#14
|
||||
|
||||
Quote:
-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#15
|
|||
|
|||
Quote:
|
#16
|
||||
|
||||
@DeusExBestia: Please test and provide feedback if the new added requested limits do help
__________________
JD-Dev & Server-Admin |
#17
|
|||
|
|||
Quote:
Still can't determine exactly HOW MANY accounts to crawl is too many, but eventually I'll figure it out I guess. |
#18
|
||||
|
||||
@DeusExBestia: those "Rate limit exceeded" links should remain in Linkgrabber section and you can retry(continue) them later via rightclick context menu
I will increase the request limits little more, maybe it helps
__________________
JD-Dev & Server-Admin |
#19
|
|||
|
|||
Quote:
Is there an error correction setting so it would start ones that are good to go and hold ones that are not? Like it automatically doesn’t add links that were already downloaded before? |
#20
|
||||
|
||||
I did read and understand your post but a screenshot of your rule or your rule exported would be good.
I guess you could simply enable "is online" as condition to filter out failed crawler-dummy URLs. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#21
|
|||
|
|||
Quote:
Code:
{"filetypeFilter":{"enabled":false, "audioFilesEnabled":false, "matchType":"IS", "useRegex":false, "hashEnabled":false, "videoFilesEnabled":false, "docFilesEnabled":false, "subFilesEnabled":false, "exeFilesEnabled":false, "archivesEnabled":false, "imagesEnabled":false, "customs":null}, "downloadDestination":"/Data/JDownloader/Twitter/<jd:packagename>", "originFilter":{"enabled":false, "matchType":"IS", "origins":[]}, "hosterURLFilter":{"enabled":false, "matchType":"CONTAINS", "regex":"", "useRegex":false}, "onlineStatusFilter":{"enabled":false, "matchType":"IS", "onlineStatus":"OFFLINE"}, "chunks":-1, "created":1597737194153, "sourceURLFilter":{"enabled":true, "matchType":"CONTAINS", "regex":"twitter.com", "useRegex":false}, "conditionFilter":{"enabled":false, "matchType":"IS_TRUE", "conditions":[]}, "testUrl":"", "enabled":true, "valid":true, "packagenameFilter":{"enabled":false, "matchType":"CONTAINS", "regex":"", "useRegex":false}, "filenameFilter":{"enabled":false, "matchType":"CONTAINS", "regex":"", "useRegex":false}, "name":" Downloads - Server", "matchAlwaysFilter":{"enabled":false}, "pluginStatusFilter":{"enabled":false, "matchType":"IS", "pluginStatus":"PREMIUM"}, "autoAddEnabled":true, "autoStartEnabled":true, "filesizeFilter":{"enabled":false, "from":0, "matchType":"BETWEEN", "to":0}} Last edited by DeusExBestia; 11.08.2021 at 15:08. Reason: Siri is awful. |
#22
|
||||
|
||||
Try to additionally set condition "File -> is -> Online".
I've done this here for you:
Spoiler:
Code:
{ "filetypeFilter" : { "enabled" : false, "audioFilesEnabled" : false, "matchType" : "IS", "useRegex" : false, "hashEnabled" : false, "videoFilesEnabled" : false, "docFilesEnabled" : false, "subFilesEnabled" : false, "exeFilesEnabled" : false, "archivesEnabled" : false, "imagesEnabled" : false, "customs" : null }, "downloadDestination" : "/Data/JDownloader/Twitter/<jd:packagename>", "originFilter" : { "enabled" : false, "matchType" : "IS", "origins" : [ ] }, "hosterURLFilter" : { "enabled" : false, "matchType" : "CONTAINS", "regex" : "", "useRegex" : false }, "onlineStatusFilter" : { "enabled" : true, "matchType" : "IS", "onlineStatus" : "ONLINE" }, "chunks" : -1, "created" : 1597737194153, "sourceURLFilter" : { "enabled" : true, "matchType" : "CONTAINS", "regex" : "twitter.com", "useRegex" : false }, "conditionFilter" : { "enabled" : false, "matchType" : "IS_TRUE", "conditions" : [ ] }, "testUrl" : "", "enabled" : true, "valid" : true, "packagenameFilter" : { "enabled" : false, "matchType" : "CONTAINS", "regex" : "", "useRegex" : false }, "filenameFilter" : { "enabled" : false, "matchType" : "CONTAINS", "regex" : "", "useRegex" : false }, "name" : " Downloads - Server", "matchAlwaysFilter" : { "enabled" : false }, "pluginStatusFilter" : { "enabled" : false, "matchType" : "IS", "pluginStatus" : "PREMIUM" }, "autoAddEnabled" : true, "autoStartEnabled" : true, "filesizeFilter" : { "enabled" : false, "from" : 0, "matchType" : "BETWEEN", "to" : 0 } } -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
Thread Tools | |
Display Modes | |
|
|