[RickyPickles]

Originally Posted by pspzockerscene

Check Settings -> Plugins -> twitter.com -> Scroll down -> See options for request limits -> Play around with higher values

Has anyone figured out the right values to deal with the rate limit?

I'm only able to partially crawl a profile before it hits, then I just get the same error over and over.

[JTK]

Can confirm that it's sometime crawl retweet even though the Crawl retweets is uncheck.

Another problem is that the ?max_date is broken too. I set it to 2024-01-16 but it's only go back to around 2024-02-02 but if crawl without max_date it's go back all the way to January without problem

10.02.24 15.32.42 <--> 10.02.24 15.32.40 jdlog://9080411370661/

[pspzockerscene]

@JTK
Please provide example URLs.
EDIT
List of upcoming changes:
- Fixed Re-Tweets (or in general Tweets of other users) being crawled when deselected by user
- Fixed stopping too early when "max_date" parameter is given and an unwanted Re-Tweet item was added which had a date < "max_date"

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[JTK]

Originally Posted by pspzockerscene

@JTK
Please provide example URLs.
EDIT
List of upcoming changes:
- Fixed Re-Tweets (or in general Tweets of other users) being crawled when deselected by user
- Fixed stopping too early when "max_date" parameter is given and an unwanted Re-Tweet item was added which had a date < "max_date"

This is the one that I have problem last time

**External links are only visible to Support Staff****External links are only visible to Support Staff**

This one with the latest update just now. At first I thought it's fixed because there is no "other user" in the file name.
But actually the retweet still show up but the file name is now changed to "user" instead of "other user" like before.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

13.02.24 00.21.35 <--> 13.02.24 00.21.33 jdlog://7580411370661/

Is twitter changed retweet to repost have something to do with it?

btw, the "External links are only visible to Support Staff" censor is not working if added "?max_date=2024-01-16" in the link.
Edit: maybe because I didn't add **External links are only visible to Support Staff**

[az0]

Originally Posted by pspzockerscene

Twitter plugin doesn't have any high priority for me.

So who has the highest priority to fix the Twitter plugin So give it to someone who has the highest priority Instead of fixing one thing and breaking the other
It was previously working due to your willful negligence

[pspzockerscene]

@JTK
I'll check it later.

Originally Posted by az0

So who has the highest priority to fix the Twitter plugin So give it to someone who has the highest priority Instead of fixing one thing and breaking the other
It was previously working due to your willful negligence

Please provide a detailed problem description and logs.
Also you're free to work on it yourself as JD is open source.
You can get the source code here:
https://support.jdownloader.org/Know...up-ide-eclipse

[pspzockerscene]

Originally Posted by JTK

At first I thought it's fixed because there is no "other user" in the file name.
But actually the retweet still show up but the file name is now changed to "user" instead of "other user" like before.

What do you mean by this?
Reposts still get added but under the wrong username?
Please post an example of such an item that shouldn't get added.

Originally Posted by JTK

Is twitter changed retweet to repost have something to do with it?

More context please. I'am not a Twitter user myself.

Originally Posted by JTK

btw, the "External links are only visible to Support Staff" censor is not working if added "?max_date=2024-01-16" in the link.

Yap, the URL-hiding does only work for URLs which get recognized as such, meaning ones which start with a protocol such as "http" or "ftp".

[accouuntt]

"More context please. I'am not a Twitter user myself."
twitter has changed "reweets" to "reposts", but it seems to be just a name change rather than anything more complex (from a user perspective at least), either way they are still being crawled for me as well, the only exception was where the profile only had retweets and no posted media of its own, so no links were grabbed at all (which is the correct behaviour of course)

[pspzockerscene]

Ah okay I understand so that's nothing I need to change.
I also didn't take over their rebranding from Twitter to X in our plugins.

[JTK]

Originally Posted by pspzockerscene

What do you mean by this?
Reposts still get added but under the wrong username?
Please post an example of such an item that shouldn't get added.


More context please. I'am not a Twitter user myself.


Yap, the URL-hiding does only work for URLs which get recognized as such, meaning ones which start with a protocol such as "http" or "ftp".

For example I crawl this link
**External links are only visible to Support Staff****External links are only visible to Support Staff**
and got this pic
2024-02-13_[crawl username]_1757366357455495638.jpg

but that pic is a repost from
**External links are only visible to Support Staff****External links are only visible to Support Staff**
which is a different user.

The post number is not the same so I guess it's not the case of wrong rename in the code.

Before, if it's crawling repost, filename would be the username and post number of the original post like
2024-02-13_[repost from username]_1757348038925005298.jpg

13.02.24 22.46.34 <--> 13.02.24 22.49.25 jdlog://5980411370661/

[pspzockerscene]

Fixed detection of Retweets failing in some cases.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Wryyyong]

Originally Posted by pspzockerscene

Fixed detection of Retweets failing in some cases.

I'm assuming it hasn't been pushed yet, since I'm still running into that issue, but ehh.

For what it's worth, I spent a bit of time earlier on today writing a quick patch for it in the hopes that it'd do the trick. I'll leave it attached in case it ends up providing further help/fixes that were missed the first time.

I wanted to share it earlier, but I had to register and wait through the activation process first, so I'm a bit late. Ah well.

Bit of forewarning, I haven't explicitly tested the patch out, but it's very light and doesn't alter all that much, and I made a point of making sure everything was in the right place. Removed a small snippet of duplicated code as well.

Here's hoping.

[pspzockerscene]

Originally Posted by Wryyyong

I'm assuming it hasn't been pushed yet, since I'm still running into that issue, but ehh.

Yes but no need to do assumptions - you can find that information in our update-FAQ:
https://board.jdownloader.org/showthread.php?t=42523

Originally Posted by Wryyyong

I wanted to share it earlier, but I had to register and wait through the activation process first, so I'm a bit late. Ah well.

No worries.

I won't make use of it now since I'm about to push my changes but if you find your changes more useful, let me know and I'll happily merge them.

[Wryyyong]

Originally Posted by pspzockerscene

I won't make use of it now since I'm about to push my changes but if you find your changes more useful, let me know and I'll happily merge them.

Just checked the revision, it does something similar to what I did in my patch, so I don't think merging would be too necessary at this point.

Gave the latest build a run as well, and it does seem to be filtering out retweets properly now.

[pspzockerscene]

Thanks for your feedback!

[RickyPickles]

I'm still receiving rate limit errors after only like 50 tweets crawled. Others are experiencing this, too, right?

I don't think any adjustments have been made to these numbers in the recent updates, and the higher values I've tried manually don't seem to make a difference. Has anyone figured out a solution?

global request limit for api.twitter.com (500)
global request limit for twimg.com (500)
profile crawler: wait time between pagination requests (3000)

[accouuntt]

Originally Posted by RickyPickles

I'm still receiving rate limit errors after only like 50 tweets crawled. Others are experiencing this, too, right?

I don't think any adjustments have been made to these numbers in the recent updates, and the higher values I've tried manually don't seem to make a difference. Has anyone figured out a solution?

global request limit for api.twitter.com (500)
global request limit for twimg.com (500)
profile crawler: wait time between pagination requests (3000)

i also have this problem, i set pagination requests to 10,000 and it doesn't seem to make a difference. (i edited the other 2 as well to 5000 but i think those are for downloading rather than profile crawling)

[JTK]

Originally Posted by RickyPickles

I'm still receiving rate limit errors after only like 50 tweets crawled. Others are experiencing this, too, right?

I don't think any adjustments have been made to these numbers in the recent updates, and the higher values I've tried manually don't seem to make a difference. Has anyone figured out a solution?

global request limit for api.twitter.com (500)
global request limit for twimg.com (500)
profile crawler: wait time between pagination requests (3000)

I think it's Twitter new API limit. It's 50requests/24hours for free tier or you have to pay for it.

developer.twitter.com/en/docs/twitter-api/rate-limits

Before, we're using old API so we're not being enforced those limit. I guess it' s the end of mass crawling if no other loopholes are found.

[pspzockerscene]

I've just reviewed all 3 existing Twitter plugin "anti rate-limit" settings:

- "Define global request limit for api.twitter.com in milliseconds (0 = no limit)" -> Currently has no effect anymore since this points to the old Twitter API which is not in use anymore. The setting will remain for now and will only be removed once I'm 100% sure that it won't be needed anymore.
- "Define global request limit for twimg.com in milliseconds (0 = no limit)" -> Still working but this basically only affects downloading Twitter media so it is not relevant for crawling Twitter profiles.
- "Profile crawler: Wait time between pagination requests in milliseconds" -> That has been forgotten to be implemented in the "new profile crawler" since Twitter switched to the newer API. I've just re-added it.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

Important notice:
Our ticket system & knowledgebase are currently under maintenance.
If this is still the case while you are reading my posts and you can't access help articles linked by our staff, use the Internet archive/wayback machine to be able to view those articles:
archive.org/web/
Enter the URL -> Click on "Browse history" -> Select the latest date available

Wichtig:
Unser Supportportal befindet sich zum Zeitpunkt dieses Posts eventuell noch im Wartungsmodus.
Du kannst die verlinkten Supportartikel trotzdem anschauen!
Nimm dafür die "Wayback Machine" vom "Internet Archive":
archive.org/web/
Gib dort die URL ein -> Klicke auf "Browser history" -> Klicke auf den aktuellsten Zeitpunkt, um die die letzte Version des Artikels anzeigen zu lassen

-psp-

[Marco86]

Thanks for the good work, I have found other issues that hopefully can be fixed as well.

I created a test account to make it easier for developers to test for the different scenarios.
Please use https ://twitter.com/XHashimoto84921 with the profile crawler.

Restricted retweets have wrong name & Retweets being crawled even when selecting crawl without retweets:
This "Accounts @ username follows or mentioned can reply" restricted repost is crawled (XHashimoto84921 profile crawler with retweets enabled), but has the wrong username and a wrong tweetID.
2024-02-20_XHashimoto84921_1759615460683702595_GGov6LIWkAA7Yyh should be 2024-02-19_Kawaka08_1759271288571396442_GGov6LIWkAA7Yyh
https ://twitter.com/Kawaka08/status/1759271288571396442 ("Accounts @ username follows or mentioned can reply" restricted post) is not recognised by the single link crawler, but loads correctly when using the profile crawler.
Same applies for the "Accounts @ username mentioned can reply" restricted repost 2024-02-20_XHashimoto84921_1759615590400967142_GGowAXiXkAAyiOR should be 2024-02-19_Kawaka08_1759271457056588136_GGowAXiXkAAyiOR.
https ://twitter.com/Kawaka08/status/1759271457056588136 ("Accounts @ username mentioned can reply" restricted post) is not recognised by the single link crawler, but loads correctly when using the profile crawler.
Even when crawling without retweets, these 2 restricted reposts will be loaded.

Restricted pinned posts not crawled:
This "Accounts @ username mentioned can reply" restricted pinned post is not crawled by the profile crawler:
https ://twitter.com/XHashimoto84921/status/1759230723788038431
Same applies to "Accounts @ username follows or mentioned can reply" restricted pinned posts.
All restricted posts are not crawled by the single link crawler, but by the profile crawler, except if they are pinned.
Normal posts that are pinned are crawled, but jdownloader crawls forever until the "Hoster problem rate limit reached!" is displayed. Check: https ://twitter.com/Kawaka08

Restricted quoted posts not crawled:
Note: There is a difference between reposts and quoted posts, a quoted post is also a repost, but the one who quotes the post is commenting on the post with text and/or media.
Quoted posts that are restricted are not crawled, even when enabling crawl with retweets.
These quoted posts are not crawled:
https ://twitter.com/Kawaka08/status/1759272571269243279
https ://twitter.com/Kawaka08/status/1759272805378449707

Restricted posts are not crawled by the profile crawler if the author himself replies directly to his post.
https ://twitter.com/XHashimoto84921/status/1759211808764653649
https ://twitter.com/XHashimoto84921/status/1759706156103270876
https ://twitter.com/XHashimoto84921/status/1759712890133881191

Now all the text only replies and posts are crawled by the profile crawler, but they should not.

[pspzockerscene]

Originally Posted by Marco86

Thanks for the good work, I have found other issues that hopefully can be fixed as well.

Thanks for your feedback!

Originally Posted by Marco86

Restricted retweets have wrong name & Retweets being crawled even when selecting crawl without retweets:

Fixed.

Originally Posted by Marco86

https ://twitter.com/Kawaka08/status/1759271288571396442 ("Accounts @ username follows or mentioned can reply" restricted post) is not recognised by the single link crawler, but loads correctly when using the profile crawler.

Fixed.

Originally Posted by Marco86

All restricted posts are not crawled by the single link crawler, but by the profile crawler, except if they are pinned.

Fixed.

Originally Posted by Marco86

Restricted quoted posts not crawled:

Fixed.

Originally Posted by Marco86

Restricted posts are not crawled by the profile crawler if the author himself replies directly to his post.

Fixed.

Originally Posted by Marco86

Now all the text only replies and posts are crawled by the profile crawler, but they should not.

You got examples for that?

Twitter plugin changes for the next update:
- fixed re-tweet detection in some cases (also fixes wrong filenames for Tweets crawled with wrong username and/or tweetID)
- profile crawler: fixed some pinned Tweets not being crawled
- profile crawler: added another fail-safe to prevent infinite loop
- single Tweet crawler: fixed some single Tweets not being crawled
- added simple BubbleNotification for when rate-limit is reached; currently there are no auto-retries when the limit is reached

Regarding all code changes I mentioned the following information applies:
Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


Important notice:
Our ticket system & knowledgebase are currently under maintenance.
If this is still the case while you are reading my posts and you can't access help articles linked by our staff, use the Internet archive/wayback machine to be able to view those articles:
archive.org/web/
Enter the URL -> Click on "Browse history" -> Select one of the latest dates available
If the date you selected does not lead you to the support article try the next oldest one.

Wichtig:
Unser Supportportal befindet sich zum Zeitpunkt dieses Posts eventuell noch im Wartungsmodus.
Du kannst die verlinkten Supportartikel trotzdem anschauen!
Nimm dafür die "Wayback Machine" vom "Internet Archive":
archive.org/web/
Gib dort die URL ein -> Klicke auf "Browser history" -> Klicke auf einen der aktuellen Zeitpunkte, um die die letzte Version des Artikels anzeigen zu lassen.
Sollte ein Zeitpunkt nicht funktionieren, versuche es mit dem nächst älteren.

[Marco86]

Thanks for fixing the issues.

Please check again if these quoted posts are crawled by the profile crawler with retweets enabled for XHashimoto84921:
https ://twitter.com/Retrogamebrews/status/1691616145491513759
https ://twitter.com/Kawaka08/status/1759272571269243279
https ://twitter.com/Kawaka08/status/1759272805378449707

I found another issue:
Posts are not crawled by the profile crawler when the user replies with media to a reply or post from another account. The same applies to text only replies to other accounts, but they should never be crawled.
These posts are not crawled:
https ://twitter.com/XHashimoto84921/status/1760045000229032256
https ://twitter.com/XHashimoto84921/status/1760049092321751451
https ://twitter.com/XHashimoto84921/status/1760051962328797524

regarding the text only posts:
normal text only posts from the user:
https ://twitter.com/XHashimoto84921/status/1759233343160275368
user replies to himself:
https ://twitter.com/XHashimoto84921/status/1759269654852251760
https ://twitter.com/XHashimoto84921/status/1759212068241096915
https ://twitter.com/XHashimoto84921/status/1759226319924089065

These posts were not crawled in the past, but I have already found them useful, so instead of removing them, add another plugin option for text crawl mode: Only if media or link is available. This would filter out text files which do not contain media or a link.
For accounts that post a lot of text only posts it would be beneficial to have a way to distinguish between text only posts and text posts that contain a link, maybe add another plugin option: Append '_textonly' to filenames of text files that contain no media or link.

Text only posts that are currently not being crawled are replies to other users, and it should stay this way:
https ://twitter.com/XHashimoto84921/status/1759228943364759955

[pspzockerscene]

Originally Posted by Marco86

Please check again if these quoted posts are crawled by the profile crawler with retweets enabled for XHashimoto84921:

Updated crawler to return only the quoted posts and not the quoted text though in my opinion the quoted text is also a status which is like a Tweet and should be returned too.
What do you think?

Also, quoted Tweets are technically no Retweets but when the Retweet crawler is disabled, they will be skipped too atm.

Originally Posted by Marco86

The same applies to text only replies to other accounts, but they should never be crawled.

Why do you think they shouldn't be crawled?

Originally Posted by Marco86

Posts are not crawled by the profile crawler when the user replies with media to a reply or post from another account. The same applies to text only replies to other accounts, but they should never be crawled.

Looks like such items are not part of the normal user timeline and/or they only get loaded once you've scrolled all the way down.
Usually those seem to be here:
twitter.com/XHashimoto84921/media
Do you know why are they not part of the normal timeline?
I mean timeline posts can also contain media or is the "media" tab about items which contain only media and no text at all?

Originally Posted by Marco86

add another plugin option for text crawl mode: Only if media or link is available. This would filter out text files which do not contain media or a link.
For accounts that post a lot of text only posts it would be beneficial to have a way to distinguish between text only posts and text posts that contain a link, maybe add another plugin option: Append '_textonly' to filenames of text files that contain no media or link.

I'am very unsure about that, also since it includes looking into the content itself.
I think that this is out of the boundaries of JDownloader and you should use external tools to evaluate the inside of profile-post-texts.

Also I see myself currently putting a lot of time into this Twitter plugin - a plugin for a website which might be about to die some time this year (who knows). I think that the current set of JDownloader Twitter crawler features already is more than good enough for 99% of all users.

Originally Posted by Marco86

Text only posts that are currently not being crawled are replies to other users, and it should stay this way:

I disagree:
Technically, by default everything that can be crawled should be crawled and the user shall be able to customize that using the given settings so maybe a "Crawl replies" setting would be the way to go.
Though then again I need to say that I can't find your example-status "1759228943364759955" in the timeline of that profile so then it might be okay that it is currently not crawled.

Changes for the next update:
- Added bubble notification for whenever less tweets / status-items than max available are found (they can be missing e.g. because: Tweet was deleted, or items were skipped based on user-settings)
- Updated crawler to crawl Tweet which was replied to instead of the reply


For all mentioned code changes, the following information applies:

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

Important notice:
Our ticket system & knowledgebase are currently under maintenance.
If this is still the case while you are reading my posts and you can't access help articles linked by our staff, use the Internet archive/wayback machine to be able to view those articles:
archive.org/web/
Enter the URL -> Click on "Browse history" -> Select one of the latest dates available
If the date you selected does not lead you to the support article try the next oldest one.

[pspzockerscene]

Changes for the next update:
- fixed caching bug which slowed down the crawl process a bit
- added support for crawling media items
- added bubble notification for when media items can't be crawled because no Twitter account is present
- do not display 'missing Tweet items' bubble notification if any items were skipped based on user settings

@Marco86
Now what I didn't understand is that according to the Twitter API, your profile contains 9 media items (field "media_count") but the Twitter website shows only 6.
EDIT

I will not release this update today because it could potentially break stuff and I'm not available during the weekends. Those changes will go live next week.

[RedViper]

Hi

Twitter X does not detect some images and downloads some videos without sound.

Regarding images, I have not been able to download this image with JDownloader2:

**External links are only visible to Support Staff****External links are only visible to Support Staff**



23.02.24 11.19.57 <--> 23.02.24 11.22.54 jdlog://9011411370661/



And the problem of videos without sound is very strange. Sometimes I download videos that don't have the audio, then I download them a second time and they already have the audio.

It is not very difficult for this to happen. It's just a matter of downloading around 5 and 10 videos so that this happens with around 3 videos [more or less].

[pspzockerscene]

Merged Twitter threads.

[pspzockerscene]

Changes for the next Twitter plugin-update:
- Added quoted Tweets support: Whenever a Tweet is a response/quote to another Tweet, both te Tweet and the quoted items get added

@RedViper
The mentioned changes will fix your problem of that un-downloadable Tweet/image.

Originally Posted by RedViper

And the problem of videos without sound is very strange. Sometimes I download videos that don't have the audio, then I download them a second time and they already have the audio.

This can happen when the video is downloaded as HLS stream.

Try this:
1. Delete the previously added video items.
2. Go to Settings -> Plugins -> twitter.com -> "Videos: Prefer HLS over http download?" -> Disable this
3. Try again.

For all code-changes mentioned in this post, the following information applies:

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

Important notice:
Our ticket system & knowledgebase are currently under maintenance.
If this is still the case while you are reading my posts and you can't access help articles linked by our staff, use the Internet archive/wayback machine to be able to view those articles:
archive.org/web/
Enter the URL -> Click on "Browse history" -> Select one of the latest dates available
If the date you selected does not lead you to the support article try the next oldest one.

[BMW]

broken
twitter.com/lydiagh0st?maxitems=10

[RedViper]

Originally Posted by pspzockerscene

Changes for the next Twitter plugin-update:
- Added quoted Tweets support: Whenever a Tweet is a response/quote to another Tweet, both te Tweet and the quoted items get added

@RedViper
The mentioned changes will fix your problem of that un-downloadable Tweet/image.


This can happen when the video is downloaded as HLS stream.

Try this:
1. Delete the previously added video items.
2. Go to Settings -> Plugins -> twitter.com -> "Videos: Prefer HLS over http download?" -> Disable this
3. Try again.

For all code-changes mentioned in this post, the following information applies:

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!

Important notice:
Our ticket system & knowledgebase are currently under maintenance.
If this is still the case while you are reading my posts and you can't access help articles linked by our staff, use the Internet archive/wayback machine to be able to view those articles:
archive.org/web/
Enter the URL -> Click on "Browse history" -> Select one of the latest dates available
If the date you selected does not lead you to the support article try the next oldest one.

Again thank you very much!

I have confirmed that it works. I just need to try the video options you suggest

[pspzockerscene]

@RedViper
Thanks for your feedback.

@BMW
Works fine for me.
Please provide a debug-log.

Please post your log-ID here
If your bugreport is about a specific website which JD supports via plugin, please also provide example URLs which can be used to reproduce the issue you are having.

Bitte poste deine Log-ID hier.
Falls dein gemeldetes Problem ein Problem mit einer Webseite ist, die per Plugin unterstützt wird, stelle bitte zusätzlich Testlinks zur Verfügung, mit denen sich dein Problem nachstellen lässt.

Important notice:
Our ticket system & knowledgebase are currently under maintenance.
If this is still the case while you are reading my posts and you can't access help articles linked by our staff, use the Internet archive/wayback machine to be able to view those articles:
archive.org/web/
Enter the URL -> Click on "Browse history" -> Select one of the latest dates available
If the date you selected does not lead you to the support article try the next oldest one.

[BMW]

No need to debug-log.
It crawls every profile
It does not crawl ten tweets
Check for yourself

[Marco86]

@BMW
You are crawling without retweets, the maxitems setting seems to only work at the moment when crawling with retweets.

@pspzockerscene
When you look at the media tab on the website one media can contain 4 media files. The new media tab layout is horrible but can be changed back with a browser extension called Old Twitter Layout (2024).

Why do you think they shouldn't be crawled?

These text only replies to other accounts are often just short meaningless text, also some accounts post an endless amount of text posts, this would spam the linkgrabber.
The only necessary text files are the ones included with media posts and ones that include a link, everything else should, if it is crawled, be optional. But these options are not implemented at the moment.
Elon certainly did more harm than good, but twitter will be here for many years to come, I see absolutely no alternative.

Do you know why are they not part of the normal timeline?

That might be because they are replies and only listed under twitter.com/XHashimoto84921/with_replies and twitter.com/XHashimoto84921/media
The media tab only contains posts with media. In the past it was easy to crawl only the media tab. A new API endpoint seems necessary.

Only since twitter changed their API in January are now all text only posts mandatorily included with every profile crawl. This was not the case before, only text files from media posts and text files from posts that contained links were crawled. Maybe this behaviour can be reintroduced by looking at how it was implemented before.


There seems to still be a few profiles where all media posts are not crawled if the posts are replies to other accounts.
When crawling this profile: **External links are only visible to Support Staff****External links are only visible to Support Staff**
These media posts are not crawled:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

highest priority:
Also now the media posts are not crawled if the post quoted another post.
When crawling this profile without retweets: twitter.com/XHashimoto84921
This media post is not crawled: twitter.com/XHashimoto84921/status/1763015886850400362
Note: This media post quoted a normal post without reply restriction, when the quoted post is reply restricted the media post is crawled correctly when crawling the profile without retweets.
These are crawled correctly:
twitter.com/XHashimoto84921/status/1763018475121868968
twitter.com/XHashimoto84921/status/1763019764505403728

This is an ongoing issue with lower priority as these posts are not that common:
When crawling this profile with retweets: twitter.com/XHashimoto84921
These reply restricted quoted posts are not crawled:
twitter.com/Kawaka08/status/1759272571269243279 (reply restricted) quoted by twitter.com/XHashimoto84921/status/1759699672837599663
twitter.com/Kawaka08/status/1759272805378449707 (reply restricted) quoted by twitter.com/XHashimoto84921/status/1759700293955293548
twitter.com/Kawaka08/status/1763010245993832622 (reply restricted) quoted by twitter.com/XHashimoto84921/status/1763018475121868968
twitter.com/Kawaka08/status/1763012200313008184 (reply restricted) quoted by twitter.com/XHashimoto84921/status/1763019764505403728
Quoted posts that are not reply restricted are already crawled correctly:
twitter.com/Kawaka08/status/1763009958440796523 (normal post) quoted by twitter.com/XHashimoto84921/status/1763015886850400362


regarding the missing audio (with an easy workaround):
Here is a test account with videos: twitter.com/LIALUZ16
This has been going on since at least the beginning of February.
When downloading a video for the first time, jdownloader will almost always download the video without audio, I only encountered a few instances where the video was downloaded with audio on the first try.
You download the video without audio first. After download you can reset and redownload the files and it will have audio.
I recommend just resetting the whole folder and redownloading everything. After the video has been downloaded once without audio, it does not matter if the files are deleted from the download list. The video will always download with audio. Even when closing and reopening the program.

You can not enable "Prefer HLS over http download" in the plugin settings, if this is enabled the videos will always have no audio. You also do not need to delete previously added links, this plugin option applies immediately.

[pspzockerscene]

Originally Posted by Marco86

You are crawling without retweets, the maxitems setting seems to only work at the moment when crawling with retweets.

Fixed.

Originally Posted by Marco86

When you look at the media tab on the website one media can contain 4 media files. The new media tab layout is horrible but can be changed back with a browser extension called Old Twitter Layout (2024).

Thanks for the information.

Originally Posted by Marco86

These text only replies to other accounts are often just short meaningless text, also some accounts post an endless amount of text posts, this would spam the linkgrabber.

Okay. I'm all for letting the user decide what is meaningless and what not.
The user "RedViper" did actually link a reply with text + image file in post #65 but I can't link it here due to privacy concerns.
As explained, if needed, I will add a setting for this behavior.

Originally Posted by Marco86

Maybe this behaviour can be reintroduced by looking at how it was implemented before.

Done, see changelog down below.

Originally Posted by Marco86

There seems to still be a few profiles where all media posts are not crawled if the posts are replies to other accounts.
When crawling this profile: http...
These media posts are not crawled:

Fixed.

Originally Posted by Marco86

Also now the media posts are not crawled if the post quoted another post.

Fixed.

Originally Posted by Marco86

highest priority:
Also now the media posts are not crawled if the post quoted another post.

Fixed.

Originally Posted by Marco86

This has been going on since at least the beginning of February.

No one has reported it. I can't fix bugs that I'm not aware of.

Originally Posted by Marco86

When downloading a video for the first time, jdownloader will almost always download the video without audio, I only encountered a few instances where the video was downloaded with audio on the first try.

Fixed. This issue happened due to the combination of two factors:
1. Part of the errorhandling thought the video was broken thus triggering a fallback:
2. The twitter.com hosterplugin has an auto fallback for broken video streams so if a progressive stream appears to be broken, HLS will be tried as fallback, regardless of the user-preference.
Since some time, HLS streams come without audio which explains this issue.
I've updated the plugin and disabled said auto-fallback. Instead you will be presented with an error message and auto-retry after some time.
I've also fixed the wrong detection of "looks like a broken video".
In the best case this just fixes the issue for you, in the worst case JD will download broken videos.
I've tested it with over 500 items so I do not expect the "broken video detetion" to fail.

By the way the Twitter plugin is doing the same for images: If the download of the "original" image fails, the "normal" one will be downloaded instead. I haven't touched this handling.

Internal changelog:
Twitter: RE forum 95054
- Fixed 'maxitems' limit not working as expected when some of the found results are skipped
- Added errorhandling for when user is trying to crawl all media items of a profile which contains none
- Added errorhandling for when user is trying to crawl all liked items of a profile which contains none
- Re-added feature to only grab all media items of profile when added link matches "/username/media"
- Fixed profile crawler not finding any media items due to duplicate avoidance handling
- Fixed quoted posts being skipped as Retweets which resulted in missing quoted items when user had Retweet crawling disabled
- Fixed some quoted Tweets not being found
- Disabled video streaming auto fallback: Auto fallback from progressive video download to HLS download will lead to video items without sound
- Fixed errorhandling wrongfully detecting progressive video downloads as broken videos thus triggering a non-needed fallback
- Removed un-needed login-request which may make download-starts quicker and result in less overall http requests
- Updated settings text of "Video: prefer HLS" setting: added warning that such videos will not have sound [this change will not yet be delivered to our users as we are not releasing CORE-updates at this moment]
- fixed image filenames being crippled on next download attempt when link is reset

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

Important notice:
Our ticket system & knowledgebase are currently under maintenance.
If this is still the case while you are reading my posts and you can't access help articles linked by our staff, use the Internet archive/wayback machine to be able to view those articles:
archive.org/web/
Enter the URL -> Click on "Browse history" -> Select one of the latest dates available
If the date you selected does not lead you to the support article try the next oldest one.

[RedViper]

Originally Posted by pspzockerscene

This can happen when the video is downloaded as HLS stream.

Try this:
1. Delete the previously added video items.
2. Go to Settings -> Plugins -> twitter.com -> "Videos: Prefer HLS over http download?" -> Disable this
3. Try again.

I checked and I have always had that option unchecked

It happened to me several times today that I downloaded videos and they didn't have audio on the first try.

[pspzockerscene]

Originally Posted by RedViper

I checked and I have always had that option unchecked

Please re-read my last post. It explains how the problem could happen even with disabled HLS video stream downloads.

[Coldblackice]

JDownloader's Twitter host plugin, when crawling copied Twitter image URLs (example below), linkgrabs the image but names it as "null" or "nullnull" (and without any extension).

Because of this, image URLs all get crawled as the same name, which then results in only one of them being downloaded. Obviously manually renaming each of them and adding an extension overcomes this.

This appears to be a new happening, as I don't recall ever experiencing this before.

Example URL:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

[Marco86]

Thanks for the fixes.

Now I only need the Litecoin donation address:)

If someone else complains again about quoted posts being loaded, tell them to filter out the profilename in the linkgrabber.

Another issue:
The textfiles are cutting of the text at a certain point. Premium accounts can post longer text. Maybe it's possible to crawl the text completely.
**External links are only visible to Support Staff**...

[pspzockerscene]

Merged Twitter threads.

Originally Posted by Marco86

Thanks for the fixes.

Thanks for your feedback

Originally Posted by Marco86

Now I only need the Litecoin donation address:)

All available donation ptions can be found here:
https://my.jdownloader.org/contribut...2F%3Fref=board

Originally Posted by Marco86

The textfiles are cutting of the text at a certain point. Premium accounts can post longer text. Maybe it's possible to crawl the text completely.

The link you've posted has been crippled by a rare forum bug.
Please re-provide that link and I will look into this possible issue.

@Coldblackice
Fixed - this was a regression of the changes I pushed last friday.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

Important notice:
Our ticket system & knowledgebase are currently under maintenance.
If this is still the case while you are reading my posts and you can't access help articles linked by our staff, use the Internet archive/wayback machine to be able to view those articles:
archive.org/web/
Enter the URL -> Click on "Browse history" -> Select one of the latest dates available
If the date you selected does not lead you to the support article try the next oldest one.

[Marco86]

Here is the link again:
twitter.com/intangiblecoins/status/1762865996447035600

And another issue:
& in posts will be translated as &amp; in the textfile
twitter.com/XHashimoto84921/status/1764875711696044521

[pspzockerscene]

Originally Posted by Marco86

& in posts will be translated as &amp; in the textfile

Added missing "html entity decode" for Twitter text items.
These changes will only be applied to items added after the update!
EDIT Next update will also "fix" existing text items. Just reset them and "re-download" them.

Regarding the problem with the super long Tweet-text being cut: I will look into this later.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-