[pspzockerscene]

Originally Posted by Marco86

I'm getting some inconsistent behaviour with some profiles.

Just so that we're talking about the same things:
1. When adding a profile URL ending with "/media", only media items will be crawled which may be less than when adding the "normal" profile URL.

2. I'm counting this by using the number of crawled text items in JD when "Text crawl mode" is set to "Always".
When I'm adding the "/media" URL, I'm getting 69 results consistently (retweets disabled).

Originally Posted by Marco86

There are accounts where the media posts are not all crawled, even if there are only a few media posts.

Fixed bug where the crawled crashed when a quoted Tweet was deleted (internally called "TweetTombstone").

Originally Posted by Marco86

ads are crawled as well

Please provide examples for crawled ads.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Marco86]

Thanks, the issues are fixed now.

You can see the ads by adding the links to the downloadlist, then crawling the profile again, all links that are not already in the downloadlist are ads.
twitter.com/WandaFPHOTO1/status/1768059544587079754
twitter.com/27mirror/status/1768212579908800875
twitter.com/karma_shopping/status/1768212579908800875

One thing that is missing is the information about the tagged accounts for media posts. This belongs in the textfile.

Example:
twitter.com/Immutable/status/1768441202280649164 There are 2 tagged accounts, displayed by clicking on the text right below the image.
This information could be added to the textfiles, for example:
tagged - Skies Verse @skies_verse
tagged - Polygon | Aggregated @0xPolygon

[pspzockerscene]

Originally Posted by Marco86

Thanks, the issues are fixed now.

Thanks for your feedback.

Originally Posted by Marco86

You can see the ads by adding the links to the downloadlist, then crawling the profile again, all links that are not already in the downloadlist are ads.

Which profile did you use for testing?
How do the ads look on the Twitter.com website?
Please provide a screenshot of such Twitter ads.
Are the ads account-specific like are Twitter account owners able to turn them off?

I didn't get any ads with all accounts I've tested with.
Also in browser after disabling my adblocker I wasn't able to get any ads.

Originally Posted by Marco86

One thing that is missing is the information about the tagged accounts for media posts. This belongs in the textfile.

I disagree since that information is not part of the post-text.
If you want to have that information, I can set it as a plugin-property (documentation) on the result items and you can save it into a text-file e.g. via EventScripter script.

EventScripter forum thread:
https://board.jdownloader.org/showthread.php?t=70525
EventScripter help article:
https://support.jdownloader.org/Know...event-scripter

[Marco86]

These accounts have ads:
twitter.com/BallietBran
twitter.com/JustJamad
It only affects a few accounts.

Another issue:
Crawling twitter.com/BallietBran or twitter.com/BallietBran/media crawls 209 items, sometimes 410 items.
209 items is incomplete until 2022-07-25_BallietBran_1551292983646883840_FYdMv3gUYAIch4s
410 items is complete until 2022-01-23_BallietBran_1485198445883232257_FJx8EfxUYAEKMt0
All available media that jdownloader should be able to crawl can be displayed with the Old Twitter Layout (2024) extension by scrolling to the bottom of the media tab, without this extension not all available media is displayed, at least for this account.

[pspzockerscene]

Originally Posted by Marco86

These accounts have ads:

Fixed.

Originally Posted by Marco86

Another issue:
Crawling twitter.com/BallietBran or twitter.com/BallietBran/media crawls 209 items, sometimes 410 items.

I can reproduce that but I'm very unsure if this is a bug.
You can search through your own Twitter logs and look for the text "Stopping because:".
Although it says "Stopping because: Failed to find any new Tweets on current page" from the last http response I can see that no Tweets were returned -> Serverside end has been reached.

Originally Posted by Marco86

All available media that jdownloader should be able to crawl can be displayed with the Old Twitter Layout (2024) extension by scrolling to the bottom of the media tab

I'm not using said browser addon so I have no insights on what it is doing.
The current Twitter plugin is using "the current version of the Twitter.com website".
Tbh I do not see me being able to constantly put so much time into this plugin.
I still recommend you using other applications which specify on downloading from Twitter.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Marco86]

There is an error in how the max setting handles reposts, twitter.com/Kawaka08?max_date=2024-03-15 should load the repost twitter.com/ComputerBase/status/1623061771425812480 as the repost was made today.

[pspzockerscene]

Fixed.

For the next update:
- refactored Retweet handling
- prefer date of source Tweet for finding latest Tweet timestamp for 'max_date' handling
- set some plugin properties of source Tweet on Retweet results as "source_<propertyname>" so from Retweet results you can now e.g. find out what the TweeID of the source Tweet was, property for this would be 'source_tweetid'

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Marco86]

There is an error with the max_date setting when the poster has quoted his own post and the quoted post has a date that is older than the max_date, the post will not be crawled and the crawler seems to abort. This is true for with and without retweets and the media tab.

twitter.com/Kawaka08?max_date=2024-02-25
twitter.com/Kawaka08/media?max_date=2024-02-25

This is the post in question:
twitter.com/Kawaka08/status/1772652445077823998

[Jiaz]

@Marco86: Thanks, pspzockerscene will look into this as soon as he finds time

[pspzockerscene]

Updated logic which finds the "current last Tweet date":
- If Tweet is a Retweet, date of the source Tweet will be used
- If Tweet contains a quoted Tweet, date of source Tweet will be used

This returns ~50 results for your example URLs (Retweets enabled).

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[DeusExBestia]

Thank you for all your hard work pspzockerscene.

Is there a way to prevent the downloader from appending " - media" to the folder of the user I'm liberating uploads from? I don't think it started happening until one of the recent updates.

[Jiaz]

@DeusExBestia: You can use packagizer, see https://support.jdownloader.org/de/k...the-packagizer to remove the " - media" from package name.likes/media and others have this suffix in package name

[pspzockerscene]

@DeusExBestia
Thanks for your feedback.
The text "media" is applied for good reasons.
If you don't want it, please use the feature, Jiaz mentioned to remove it from your package names.

[RedAero]

I'm getting some strange behavior... Most of the time if I paste an account URL (e.g. twitter.com/Kawaka08), nothing happens, nothing is crawled, but *sometimes*, a few posts do get crawled, and not even consistently between different devices (i.e. computer A will crawl nothing, computer B gets 3 posts). On the other hand, copying the /media link seems to work fine.

[Jiaz]

@RedAero: have you added your twitter/x account to JDownloader via cookies method?
Please provide a debug log, see https://support.jdownloader.org/de/k...d-session-logs
Enable debug mode and restart JDownloader. Now reproduce the issue and then create log and post logID here

[pspzockerscene]

Crawling of quoted Tweets during a profile crawl process can now be deactivated in the Twitter/X.com plugin settings.