[Solved] X/Twitter all crawling not working

[JTK]

X/Twitter can't crawl any link today. It worked fine a couple of days ago.
My status show that Account status is OK.

When press crawling it starts and stop after a couple of seconds. That page works fine on the browser. I tried with 10+ links with no result.

[DukeM]

Just want to second this report. Haven't used JD2 in a few weeks but when I did today, it just shows the crawling icon on the lower right then disappears but nothing gets added to the LinkGrabber tab. Thanks!

[gabor]

It was working fine yesterday(2024.01.19, 8-10PM GMT+1)

For me it doesn't even start the auto crawl, when I do it manually it shows the crawl windows for a few seconds but nothing happens. Other sites(youtube, pixeldrain) work fine.

Example links:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Log:
20.01.24 15.06.03 <--> 20.01.24 15.07.42 jdlog://3530411370661/

[NatSpectre]

I concur.
Is this a new setting in X/Twitter or some glitch on the latest JDownloader2 update?
I can download individual links in X/Twitter i.e. Right Click Link & Copy and JDownloader2 adds it.
Please address this ASAP
Thanks
~ NaTe

[mex]

Same issue here.

[vesselsense]

I'm pasting a twitter link into the LinkGrabber, the crawling box shows up for a second or two, and nothing happens. This problem started last Friday on January 19, 2024 and has not been resolved.

[NatSpectre]

I concur!

[pspzockerscene]

Merged Twitter threads.

The plugin is fully operational here.
My guess is that Twitter is doing some A/B testing or they are about to re-new their website and/or internal API.

According of the logs of some of you, their API just responded with a blank page while it is working fine here.

I'm unable to help with this at this moment.
Either way I recommend moving away from Twitter/X...

[gabor]

So, something changed, now instead of nothing I get Plugin error!media

**External links are only visible to Support Staff****External links are only visible to Support Staff**
23.01.24 19.01.11 <--> 23.01.24 19.04.19 jdlog://4240411370661/

[pspzockerscene]

Still working fine here.
Please re-read my latest post.

Most likely it will take some time until I can reproduce the problem here and work on a fix.

[travestree]

Trying to crawl an account doesn't work.

Switching login to different account doesn't work.

Pasting specific tweets does work.

23.01.24 16.22.57 <--> 23.01.24 16.47.32 jdlog://2340411370661/

[vesselsense]

Originally Posted by gabor

So, something changed, now instead of nothing I get Plugin error!media

**External links are only visible to Support Staff**...
23.01.24 19.01.11 <--> 23.01.24 19.04.19 jdlog://4240411370661/

I did a clean re-install and am now getting "Plugin error!media" error as well now.

[groinksan]

One more for "Plugin Defect!" errors on any X page. Account is active in JDownloader.

[pspzockerscene]

Which country are you guys from?

Here in Germany with a German IP it's working just fine.
Maybe Twitter has introduced country specific changes(?)
Does it work when you're using a german VPN?

[RickyPickles]

I'm in the US. Same issues. First it wouldn't crawl profiles, now I get a plugin defect error.

[mauriziowod]

same from italy

[DukeM]

Used it with several countries from different regions on my VPN and still getting the same result. Not sure if it matters but I also crawl by twitter.com/username/media and not by individual post.

Originally Posted by pspzockerscene

Either way I recommend moving away from Twitter/X...

Totally agree with this but it isn't so bad if you only use it to archive nudie accounts. Haha.

[pspzockerscene]

I understood the problem though what I've posted in post #14 remains:
For some reason it is working perfectly fine for me for this moment and yes, also with the example URLs/accounts you guys have provided in this thread.

I guess within the next few days I will be able to reproduce the problem and work on a fix.

Just a heads up: JD is not the only tool which is able to download from Twitter - you might want to use other tools in the meantime e.g.:
wiki.archiveteam.org/index.php/Twitter#Archives

[accelheim]

Same here, I get the error "Plugin defect!likes" when pasting the likes of someone or "Plugin defect!media" when I paste the media link.

24.01.24 15.53.59 <--> 24.01.24 23.17.56 jdlog://0440411370661/

[pspzockerscene]

Last update for now:
I still got zero problems here, even via US VPN.

You will need to wait until I'm able to reproduce the problem since I can't fix this one based on logs.

[rubicon1970]

I am getting a plugin defect error for X. 25.01.24 09.43.28 <--> 25.01.24 14.28.57 jdlog://1540411370661/ Thanks.

[vesselsense]

Originally Posted by pspzockerscene

Last update for now:
I still got zero problems here, even via US VPN.

You will need to wait until I'm able to reproduce the problem since I can't fix this one based on logs.

Have you tried a separate computer and wifi?

[pspzockerscene]

Yes I did try other IP addresses.

I know I'm repeating myself but here is a quick reminder:

1. JDownloader is not the only tool which is able to download from Twitter - you might want to use other tools in the meantime e.g.:
wiki.archiveteam.org/index.php/Twitter#Archives

2. Please be patient.
I see the high demand for twitter.com download automation but there is other things I'm working on and fixing the Twitter plugin doesn't have any high priority for me.

3. JDownloader is open source - anyone is welcome to help. Source code:
https://support.jdownloader.org/Know...up-ide-eclipse
Twitter plugins' sources are located here:

jd.plugins.hoster.TwitterCom

and:

jd.plugins.decrypter.TwitterComCrawler

[snoopyraw]

Still not working.... what alternative free programm can bulk download a twitter page?

[noradurst]

It's not working for me either, it's been a few days now. I'm from Brazil.

[pspzockerscene]

Originally Posted by snoopyraw

what alternative free programm can bulk download a twitter page?

Browse to github.com and search for "twitter downloader".
Random example:
github.com/mmpx12/twitter-media-downloader

Originally Posted by noradurst

it's been a few days now.

Keep in mind that JDownloader is a free application with free support. You can't expect any immediate fixes.

[noradurst]

Originally Posted by pspzockerscene

Browse to github.com and search for "twitter downloader".
Random example:
github.com/mmpx12/twitter-media-downloader


Keep in mind that JDownloader is a free application with free support. You can't expect any immediate fixes.

Sorry if it sounded like a criticism; it was actually just an informational comment since the functioning status in different countries was mentioned above. I just wanted to add to validate or invalidate the regional issue. I love the app and have only compliments.

[xped123]

Good weekend,

This may be due to twitter I found changing their /media page, though I could be wrong. However, I would not be surprised if this is due to my system being outdated (Win8.1/rarely update).

Please let me know if you need more info.

Attempted to plug a profile (x.com/account/media) and it yielded "Plugin Defect!accountname" or "Plugin Defect!media".

03.02.24 13.07.50 <--> 03.02.24 13.07.48 jdlog://0660411370661/

[Vanillo]

Seit einigen Tagen passiert nichts mehr, wenn ich von einem Twitter-Account die gesammelten Medien - also alle - downloaden will. JD findet keine Downloadlinks mehr.
Beispiellink: **External links are only visible to Support Staff****External links are only visible to Support Staff**

Klicke ich ein einzelnes Medium an und lade diese direkte URL in JD, dann klappts. Mein Twitter Account IN JD is aktuell und gültig. Das Twitter PlugIn is auch vorhanden und soweit ichs überblicke, alles richtig eingestellt.

Jemand eine Idee, worans liegt?

thx

PS:
Die Medienübersicht bei Twitter sieht auch anders aus seit einigen Tagen, man sieht dort wirklich nur noch die geposteten Medien und nicht mehr den gesamten Post an sich, so wie das vorher war. Ich vermute, durch diese Umstellung haben sich die darunter liegenden Linkverweise geändert und sind im PlugIn von JD noch nicht aktualisiert. Ist aber nur so eine Vermutung von mir, ohne das ich mich da auskennen würde.

Ein einzelner Medienlink sieht nun so aus:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

[Marco86]

Unfortunately Jdownloader2 is still the only viable option for twitter, other tools, if they work, only download with the original filenames and no txt file. It would be great if someone could make a threads plugin like the twitter plugin with the filename system like Original+2. More and more people are going to threads. I have the money and will definitely fund it.

For the twitter solution: You have to go into the Jdownloader2 plugins and enable crawl retweets.
But there is still an issue with the twitter plugin: If the poster himself replies to his tweet, the tweet is not recognized by Jdownloader, for example these 3 tweets: **External links are only visible to Support Staff****External links are only visible to Support Staff**
And this tweet: **External links are only visible to Support Staff****External links are only visible to Support Staff**
These tweets are not recognized by the profile crawler, but are working with the single link status crawler, so I hope there can be a fast fix for this.

[pspzockerscene]

Twitter Threads zusammengeführt.
EDIT

Originally Posted by noradurst

Sorry if it sounded like a criticism

No worries criticism is also absolutely fine.

[DukeM]

Originally Posted by Marco86

Unfortunately Jdownloader2 is still the only viable option for twitter, other tools, if they work, only download with the original filenames and no txt file. It would be great if someone could make a threads plugin like the twitter plugin with the filename system like Original+2. More and more people are going to threads. I have the money and will definitely fund it.

For the twitter solution: You have to go into the Jdownloader2 plugins and enable crawl retweets.
But there is still an issue with the twitter plugin: If the poster himself replies to his tweet, the tweet is not recognized by Jdownloader, for example these 3 tweets: **External links are only visible to Support Staff**...
And this tweet: **External links are only visible to Support Staff**...
These tweets are not recognized by the profile crawler, but are working with the single link status crawler, so I hope there can be a fast fix for this.

Thanks for sharing your temporary fix! Works great, at least for my use case.

Would love to see Threads supported as well. Unfortunately, I was only able to afford to donate twice in over five years since I started using JD2. Will definitely do so again when I'm not broke though since it's incredibly useful and the support has always been fantastic.

[Xuan]

When I try to grab a complete twitter page (like I used to do) it tells me, that there is a plugin defect .... I can grab solo posts from this exact page tho

[pspzockerscene]

Merged Twitter threads.

[mex]

Originally Posted by Marco86

go into the Jdownloader2 plugins and enable crawl retweets.

Works for me. Thanks.

[pspzockerscene]

I finally found time to look into this:
Twitter seems to have shut down another old API endpoint which resulted in these problems.
Retweets could only be crawled via the new/current one hence enabling said setting helped some users.

I've updated the plugin to always use the new endpoint and at the same time filter-out Re-Tweets if the user does not want them.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[JTK]

Thank you. It's working again.
But now it reached "rate limit" after crawling 50 files and can't crawl anymore.
I guess it's the new API restriction.

[pspzockerscene]

Thanks for your feedback.

Originally Posted by JTK

But now it reached "rate limit" after crawling 50 files and can't crawl anymore.

Check Settings -> Plugins -> twitter.com -> Scroll down -> See options for request limits -> Play around with higher values

[accouuntt]

thanks for fixing this, though even with crawl retweets disabled it's still downloading them. not sure if it's just on my end or not.

[pspzockerscene]

@accouuntt
Please provide example URLs and a log.

Please post your log-ID here
If your bugreport is about a specific website which JD supports via plugin, please also provide example URLs which can be used to reproduce the issue you are having.

Bitte poste deine Log-ID hier.
Falls dein gemeldetes Problem ein Problem mit einer Webseite ist, die per Plugin unterstützt wird, stelle bitte zusätzlich Testlinks zur Verfügung, mit denen sich dein Problem nachstellen lässt.

-psp-