JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 20.01.2024, 06:57
JTK JTK is offline
Super Loader
 
Join Date: Jun 2021
Posts: 26
Default X/Twitter all crawling not working

X/Twitter can't crawl any link today. It worked fine a couple of days ago.
My status show that Account status is OK.

When press crawling it starts and stop after a couple of seconds. That page works fine on the browser. I tried with 10+ links with no result.
Reply With Quote
  #2  
Old 20.01.2024, 13:43
DukeM's Avatar
DukeM DukeM is offline
Fibre Channel User
 
Join Date: Sep 2019
Posts: 118
Default

Just want to second this report. Haven't used JD2 in a few weeks but when I did today, it just shows the crawling icon on the lower right then disappears but nothing gets added to the LinkGrabber tab. Thanks!
Reply With Quote
  #3  
Old 20.01.2024, 15:11
gabor gabor is offline
Vacuum Cleaner
 
Join Date: Mar 2022
Posts: 16
Default

It was working fine yesterday(2024.01.19, 8-10PM GMT+1)

For me it doesn't even start the auto crawl, when I do it manually it shows the crawl windows for a few seconds but nothing happens. Other sites(youtube, pixeldrain) work fine.

Example links:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Log:
20.01.24 15.06.03 <--> 20.01.24 15.07.42 jdlog://3530411370661/

Last edited by gabor; 20.01.2024 at 15:15.
Reply With Quote
  #4  
Old 21.01.2024, 03:17
NatSpectre NatSpectre is offline
Baby Loader
 
Join Date: Mar 2023
Posts: 5
Default

I concur.
Is this a new setting in X/Twitter or some glitch on the latest JDownloader2 update?
I can download individual links in X/Twitter i.e. Right Click Link & Copy and JDownloader2 adds it.
Please address this ASAP
Thanks
~ NaTe

Last edited by NatSpectre; 21.01.2024 at 04:13. Reason: Update/Clarity
Reply With Quote
  #5  
Old 22.01.2024, 01:20
mex mex is offline
JD Alpha
 
Join Date: Dec 2020
Posts: 22
Default

Same issue here.
Reply With Quote
  #6  
Old 23.01.2024, 01:41
vesselsense vesselsense is offline
Modem User
 
Join Date: Jan 2024
Posts: 3
Default Twitter Not Being Scraped

I'm pasting a twitter link into the LinkGrabber, the crawling box shows up for a second or two, and nothing happens. This problem started last Friday on January 19, 2024 and has not been resolved.
Reply With Quote
  #7  
Old 23.01.2024, 06:26
NatSpectre NatSpectre is offline
Baby Loader
 
Join Date: Mar 2023
Posts: 5
Default

I concur!
Reply With Quote
  #8  
Old 23.01.2024, 15:06
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

Merged Twitter threads.

The plugin is fully operational here.
My guess is that Twitter is doing some A/B testing or they are about to re-new their website and/or internal API.

According of the logs of some of you, their API just responded with a blank page while it is working fine here.

I'm unable to help with this at this moment.
Either way I recommend moving away from Twitter/X...
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
  #9  
Old 23.01.2024, 19:06
gabor gabor is offline
Vacuum Cleaner
 
Join Date: Mar 2022
Posts: 16
Default

So, something changed, now instead of nothing I get Plugin error!media

**External links are only visible to Support Staff****External links are only visible to Support Staff**
23.01.24 19.01.11 <--> 23.01.24 19.04.19 jdlog://4240411370661/
Reply With Quote
  #10  
Old 23.01.2024, 19:24
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

Still working fine here.
Please re-read my latest post.

Most likely it will take some time until I can reproduce the problem here and work on a fix.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
  #11  
Old 24.01.2024, 01:48
travestree travestree is offline
Giga Loader
 
Join Date: Oct 2013
Posts: 96
Default

Trying to crawl an account doesn't work.

Switching login to different account doesn't work.

Pasting specific tweets does work.

23.01.24 16.22.57 <--> 23.01.24 16.47.32 jdlog://2340411370661/
Reply With Quote
  #12  
Old 24.01.2024, 02:33
vesselsense vesselsense is offline
Modem User
 
Join Date: Jan 2024
Posts: 3
Default

Quote:
Originally Posted by gabor View Post
So, something changed, now instead of nothing I get Plugin error!media

**External links are only visible to Support Staff**...
23.01.24 19.01.11 <--> 23.01.24 19.04.19 jdlog://4240411370661/
I did a clean re-install and am now getting "Plugin error!media" error as well now.
Reply With Quote
  #13  
Old 24.01.2024, 06:27
groinksan groinksan is offline
Wind Gust
 
Join Date: Dec 2018
Posts: 42
Default

One more for "Plugin Defect!" errors on any X page. Account is active in JDownloader.
Reply With Quote
  #14  
Old 24.01.2024, 11:06
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

Which country are you guys from?

Here in Germany with a German IP it's working just fine.
Maybe Twitter has introduced country specific changes(?)
Does it work when you're using a german VPN?
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
  #15  
Old 24.01.2024, 11:45
RickyPickles RickyPickles is offline
Baby Loader
 
Join Date: Oct 2020
Posts: 7
Default

I'm in the US. Same issues. First it wouldn't crawl profiles, now I get a plugin defect error.
Reply With Quote
  #16  
Old 24.01.2024, 14:54
mauriziowod mauriziowod is offline
Baby Loader
 
Join Date: May 2020
Posts: 6
Default

same from italy
Reply With Quote
  #17  
Old 24.01.2024, 15:22
DukeM's Avatar
DukeM DukeM is offline
Fibre Channel User
 
Join Date: Sep 2019
Posts: 118
Default

Used it with several countries from different regions on my VPN and still getting the same result. Not sure if it matters but I also crawl by twitter.com/username/media and not by individual post.

Quote:
Originally Posted by pspzockerscene View Post
Either way I recommend moving away from Twitter/X...
Totally agree with this but it isn't so bad if you only use it to archive nudie accounts. Haha.
Reply With Quote
  #18  
Old 24.01.2024, 15:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

I understood the problem though what I've posted in post #14 remains:
For some reason it is working perfectly fine for me for this moment and yes, also with the example URLs/accounts you guys have provided in this thread.

I guess within the next few days I will be able to reproduce the problem and work on a fix.

Just a heads up: JD is not the only tool which is able to download from Twitter - you might want to use other tools in the meantime e.g.:
wiki.archiveteam.org/index.php/Twitter#Archives
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
  #19  
Old 24.01.2024, 23:19
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 163
Default

Same here, I get the error "Plugin defect!likes" when pasting the likes of someone or "Plugin defect!media" when I paste the media link.

24.01.24 15.53.59 <--> 24.01.24 23.17.56 jdlog://0440411370661/
Reply With Quote
  #20  
Old 25.01.2024, 11:27
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

Last update for now:
I still got zero problems here, even via US VPN.

You will need to wait until I'm able to reproduce the problem since I can't fix this one based on logs.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
  #21  
Old 25.01.2024, 20:29
rubicon1970 rubicon1970 is offline
Mega Loader
 
Join Date: Sep 2019
Posts: 61
Default plugin defect for twitter

I am getting a plugin defect error for X. 25.01.24 09.43.28 <--> 25.01.24 14.28.57 jdlog://1540411370661/ Thanks.
Reply With Quote
  #22  
Old 25.01.2024, 22:38
vesselsense vesselsense is offline
Modem User
 
Join Date: Jan 2024
Posts: 3
Default

Quote:
Originally Posted by pspzockerscene View Post
Last update for now:
I still got zero problems here, even via US VPN.

You will need to wait until I'm able to reproduce the problem since I can't fix this one based on logs.
Have you tried a separate computer and wifi?
Reply With Quote
  #23  
Old 26.01.2024, 11:24
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

Yes I did try other IP addresses.

I know I'm repeating myself but here is a quick reminder:

1. JDownloader is not the only tool which is able to download from Twitter - you might want to use other tools in the meantime e.g.:
wiki.archiveteam.org/index.php/Twitter#Archives

2. Please be patient.
I see the high demand for twitter.com download automation but there is other things I'm working on and fixing the Twitter plugin doesn't have any high priority for me.

3. JDownloader is open source - anyone is welcome to help. Source code:
https://support.jdownloader.org/Know...up-ide-eclipse
Twitter plugins' sources are located here:
Code:
jd.plugins.hoster.TwitterCom
and:
Code:
jd.plugins.decrypter.TwitterComCrawler
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
  #24  
Old 01.02.2024, 04:11
snoopyraw snoopyraw is offline
Modem User
 
Join Date: Jan 2024
Posts: 1
Default

Still not working.... what alternative free programm can bulk download a twitter page?
Reply With Quote
  #25  
Old 02.02.2024, 01:15
noradurst noradurst is offline
Modem User
 
Join Date: Feb 2024
Posts: 2
Default

It's not working for me either, it's been a few days now. I'm from Brazil.
Reply With Quote
  #26  
Old 02.02.2024, 13:05
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

Quote:
Originally Posted by snoopyraw View Post
what alternative free programm can bulk download a twitter page?
Browse to github.com and search for "twitter downloader".
Random example:
github.com/mmpx12/twitter-media-downloader

Quote:
Originally Posted by noradurst View Post
it's been a few days now.
Keep in mind that JDownloader is a free application with free support. You can't expect any immediate fixes.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
  #27  
Old 03.02.2024, 17:42
noradurst noradurst is offline
Modem User
 
Join Date: Feb 2024
Posts: 2
Default

Quote:
Originally Posted by pspzockerscene View Post
Browse to github.com and search for "twitter downloader".
Random example:
github.com/mmpx12/twitter-media-downloader


Keep in mind that JDownloader is a free application with free support. You can't expect any immediate fixes.
Sorry if it sounded like a criticism; it was actually just an informational comment since the functioning status in different countries was mentioned above. I just wanted to add to validate or invalidate the regional issue. I love the app and have only compliments.
Reply With Quote
  #28  
Old 03.02.2024, 21:21
xped123's Avatar
xped123 xped123 is offline
Mega Loader
 
Join Date: Nov 2011
Posts: 61
Default Twitter "Plugin Defect"

Good weekend,

This may be due to twitter I found changing their /media page, though I could be wrong. However, I would not be surprised if this is due to my system being outdated (Win8.1/rarely update).

Please let me know if you need more info.

Attempted to plug a profile (x.com/account/media) and it yielded "Plugin Defect!accountname" or "Plugin Defect!media".

03.02.24 13.07.50 <--> 03.02.24 13.07.48 jdlog://0660411370661/
Reply With Quote
  #29  
Old 04.02.2024, 00:52
Vanillo Vanillo is offline
Junior Loader
 
Join Date: Nov 2017
Posts: 13
Default Twitter Medien werden nicht mehr in JD geladen

Seit einigen Tagen passiert nichts mehr, wenn ich von einem Twitter-Account die gesammelten Medien - also alle - downloaden will. JD findet keine Downloadlinks mehr.
Beispiellink: **External links are only visible to Support Staff****External links are only visible to Support Staff**

Klicke ich ein einzelnes Medium an und lade diese direkte URL in JD, dann klappts. Mein Twitter Account IN JD is aktuell und gültig. Das Twitter PlugIn is auch vorhanden und soweit ichs überblicke, alles richtig eingestellt.

Jemand eine Idee, worans liegt?

thx

PS:
Die Medienübersicht bei Twitter sieht auch anders aus seit einigen Tagen, man sieht dort wirklich nur noch die geposteten Medien und nicht mehr den gesamten Post an sich, so wie das vorher war. Ich vermute, durch diese Umstellung haben sich die darunter liegenden Linkverweise geändert und sind im PlugIn von JD noch nicht aktualisiert. Ist aber nur so eine Vermutung von mir, ohne das ich mich da auskennen würde.

Ein einzelner Medienlink sieht nun so aus:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #30  
Old 05.02.2024, 03:29
Marco86 Marco86 is offline
DSL Light User
 
Join Date: Aug 2022
Posts: 34
Default

Unfortunately Jdownloader2 is still the only viable option for twitter, other tools, if they work, only download with the original filenames and no txt file. It would be great if someone could make a threads plugin like the twitter plugin with the filename system like Original+2. More and more people are going to threads. I have the money and will definitely fund it.

For the twitter solution: You have to go into the Jdownloader2 plugins and enable crawl retweets.
But there is still an issue with the twitter plugin: If the poster himself replies to his tweet, the tweet is not recognized by Jdownloader, for example these 3 tweets: **External links are only visible to Support Staff****External links are only visible to Support Staff**
And this tweet: **External links are only visible to Support Staff****External links are only visible to Support Staff**
These tweets are not recognized by the profile crawler, but are working with the single link status crawler, so I hope there can be a fast fix for this.

Last edited by Marco86; 05.02.2024 at 04:56.
Reply With Quote
  #31  
Old 05.02.2024, 12:15
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

Twitter Threads zusammengeführt.
EDIT

Quote:
Originally Posted by noradurst View Post
Sorry if it sounded like a criticism
No worries criticism is also absolutely fine.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA


Last edited by pspzockerscene; 05.02.2024 at 12:16. Reason: EDIT
Reply With Quote
  #32  
Old 05.02.2024, 14:46
DukeM's Avatar
DukeM DukeM is offline
Fibre Channel User
 
Join Date: Sep 2019
Posts: 118
Default

Quote:
Originally Posted by Marco86 View Post
Unfortunately Jdownloader2 is still the only viable option for twitter, other tools, if they work, only download with the original filenames and no txt file. It would be great if someone could make a threads plugin like the twitter plugin with the filename system like Original+2. More and more people are going to threads. I have the money and will definitely fund it.

For the twitter solution: You have to go into the Jdownloader2 plugins and enable crawl retweets.
But there is still an issue with the twitter plugin: If the poster himself replies to his tweet, the tweet is not recognized by Jdownloader, for example these 3 tweets: **External links are only visible to Support Staff**...
And this tweet: **External links are only visible to Support Staff**...
These tweets are not recognized by the profile crawler, but are working with the single link status crawler, so I hope there can be a fast fix for this.
Thanks for sharing your temporary fix! Works great, at least for my use case.

Would love to see Threads supported as well. Unfortunately, I was only able to afford to donate twice in over five years since I started using JD2. Will definitely do so again when I'm not broke though since it's incredibly useful and the support has always been fantastic.
Reply With Quote
  #33  
Old 06.02.2024, 03:35
Xuan Xuan is offline
Modem User
 
Join Date: Dec 2023
Posts: 1
Default Problems with X/Twitter

When I try to grab a complete twitter page (like I used to do) it tells me, that there is a plugin defect .... I can grab solo posts from this exact page tho
Reply With Quote
  #34  
Old 06.02.2024, 11:15
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

Merged Twitter threads.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
  #35  
Old 07.02.2024, 16:02
mex mex is offline
JD Alpha
 
Join Date: Dec 2020
Posts: 22
Default

Quote:
Originally Posted by Marco86 View Post
go into the Jdownloader2 plugins and enable crawl retweets.
Works for me. Thanks.
Reply With Quote
  #36  
Old 07.02.2024, 17:04
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

I finally found time to look into this:
Twitter seems to have shut down another old API endpoint which resulted in these problems.
Retweets could only be crawled via the new/current one hence enabling said setting helped some users.

I've updated the plugin to always use the new endpoint and at the same time filter-out Re-Tweets if the user does not want them.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
  #37  
Old 07.02.2024, 18:13
JTK JTK is offline
Super Loader
 
Join Date: Jun 2021
Posts: 26
Default

Thank you. It's working again.
But now it reached "rate limit" after crawling 50 files and can't crawl anymore.
I guess it's the new API restriction.
Reply With Quote
  #38  
Old 07.02.2024, 18:16
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

Thanks for your feedback.

Quote:
Originally Posted by JTK View Post
But now it reached "rate limit" after crawling 50 files and can't crawl anymore.
Check Settings -> Plugins -> twitter.com -> Scroll down -> See options for request limits -> Play around with higher values
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
  #39  
Old 08.02.2024, 18:19
accouuntt accouuntt is offline
Baby Loader
 
Join Date: Jul 2023
Posts: 5
Default

thanks for fixing this, though even with crawl retweets disabled it's still downloading them. not sure if it's just on my end or not.
Reply With Quote
  #40  
Old 09.02.2024, 13:50
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

@accouuntt
Please provide example URLs and a log.

Please post your log-ID here
If your bugreport is about a specific website which JD supports via plugin, please also provide example URLs which can be used to reproduce the issue you are having.

Bitte poste deine Log-ID hier.
Falls dein gemeldetes Problem ein Problem mit einer Webseite ist, die per Plugin unterstützt wird, stelle bitte zusätzlich Testlinks zur Verfügung, mit denen sich dein Problem nachstellen lässt.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 00:58.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.