[RedViper]

Hi

In this tweet with 3 photos and a video, only one photo is detected:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Log:

16.04.23 22.53.38 <--> 16.04.23 22.58.06 jdlog://8874311370661/

[pspzockerscene]

Hi,
this will take some more time as it seems like the one API endpoint we're using is either limited or we're using the wrong parameters.

Most likely you will be able to crawl all items when you're crawling the complete profile behind it as we're using another endpoint for profile crawling.

[RedViper]

OK pspzockerscene

Thank you very much for you support!

[Marco86]

I have more info on that issue. I am using the latest version released 1 day ago, when multiple videos or videos and images are together in a tweet, JD2 is not downloading all images or videos, this is true for the individual link and for the profile link, so crawling the complete profile does not help.

My solution for now: I'm using an old version #47140 from November 26, 2022, every video and image is recognized. If anyone wants it dm me.

Out of 27 video image tweets I have checked only 3 were working correctly. This is a serious bug now, and it's solvable because it's already working on an older version.

[pspzockerscene]

@Marco86
Please provide some example URLs which are working in the old version according to your tests.
EDIT

Using that old revision, the example URL of post #1 in this thread doesn't work at all.

[Marco86]

I can now confirm that there are indeed tweets that don't work with the status link but work when crawling the profile link, same with old and new JD2 version.
Here is one link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

However I have also found many tweets that work with the old version and not with the new one:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

These tweets work with the old version on the status link and profile link, on the new version they don't work on status link nor profile link.

Here is the link of the version I am using in case it is needed:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

@Marco86
Wow good catch!

Fixed regression which has been introduced when thumbnail setting was added.
The issue which was reported in post 1 of this thread still remains open for now as it requires further changes of this plugin.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[pspzockerscene]

@RedViper
Next CORE-Update will contain a new "Crawl Mode" setting.
Set that to "New API" and JD will be able to find all images of such tweets.

I won't switch to that new method yet as I didn't have enough time to test it.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[pspzockerscene]

CORE-Updates have been released!
All announced bugfixes and features are live!
Please update your JDownloader and report any issues you find asap.
If this thread gets marked as "[Solved]" by our forum staff you can still post in it and we will read- and reply to it!

CORE-Updates wurden released!
Alle angekündigten Bugfixes/Features sind nun verfügbar!
Bitte JDownloader updaten und eventuelle Bugs schnellstmöglich an uns melden.
Falls dieser Thread vom Team als "[Erledigt]" markiert wird, kannst du weiterhin darin antworten und wir lesen/beantworten auch solche Threads!

-psp-

[Marco86]

Problem with the New API:
When the media status link that is pasted into JD2 is a response to anything, JD2 will also parse all tweets that came before including all textfiles for text only tweets.
Fortunately that only affects single links, not profile links, and the Old API does not have this issue while fixing all links that could previously only be downloaded by the old program version, but I hope the New API issue can be fixed as certain links can only be downloaded with the New API.

There is another problem:
The thumbnail pictures are still being downloaded when the video is a gif, same with Old and New API and status and profile links.
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Also the names of the textfiles were changed to be the same as the name of the video thumbnail, even when video thumbnail is disabled, hopefully this can be changed back. Tweets without videos are not affected.

[pspzockerscene]

Fixed.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Marco86]

The thumbnails of gifs are now not downloaded anymore and the text filenames are now matching, but unfortunately there was another undesired change: The format of the twitter original filenames has been changed.
For example the following tweet **External links are only visible to Support Staff****External links are only visible to Support Staff** gives the filename 8FlpmYUlEf1fa1qq.mp4, where before the filename was 1672269387750051840_8FlpmYUlEf1fa1qq.mp4.

I hope this can be changed back, as I am using a packagizer rule to get the best twitter filenames:
Downloadurl contains (with regex enabled): https?://twitter.com/([^/]+)/.+
Filename: <jd:prop:date>_<jd:hoster:1>_<jd:orgfilename>

Using this packagizer rule gives the filename 2023-06-24_SpaceX_1672269387750051840_8FlpmYUlEf1fa1qq.mp4, which I would want to keep using.

[pspzockerscene]

The "original" filenames were never ment to be "<tweetID>_<original[video]FilenameWithoutExt>.ext".
The change you are talking about was done after this report.

If you need other filenames, please continue using the Packagizer as that is what the Packagizer has been designed for

[Marco86]

I am just not able to get the previous filename format 2023-02-06_username_tweetID_orgfilename.ext.
I'm still using the JD2 version from November 2022 to get the correct filenames
Can someone tell me how to change my packagizer rule to include the tweetID that was removed from the original filename?
@pspzockerscene I read the report you linked, there is someone complaining there as well that the names do not match anymore. Would it be possible to include a new twitter plugin setting like a custom filename field, like present in the instagram and youtube plugin, where everyone can choose the filename they want with the 4 variables: *date*, *uploader*, *tweetID* and *orgfilename*?

The current plugin filenames are good, but do not include the orgfilename at the end, instead it's 0, 1 ,2, 3. This is not ideal because if you downloaded a twitter image in the past with the original filename from the website, and now want to use the windows search to see which account the image belonged to and which date it was posted, that is not possible with the current plugin filenames because the original filename is not included.

[erarata]

Originally Posted by pspzockerscene

The "original" filenames were never ment to be "<tweetID>_<original[video]FilenameWithoutExt>.ext".
The change you are talking about was done after **External links are only visible to Support Staff**....

If you need other filenames, please continue using the Packagizer as that is what the Packagizer has been designed for

How would one restore "<tweetID>_<FilenameWithoutExt>.ext" using packagizer? My research so far has shown that we're not able to pull in parts of the url itself as the filename and using "mediaid" via the twitter plugin is not the same as the tweet id.

[pspzockerscene]

Originally Posted by Marco86

I'm still using the JD2 version from November 2022 to get the correct filenames

"Correct" is a definition-thing.

Originally Posted by Marco86

Can someone tell me how to change my packagizer rule to include the tweetID that was removed from the original filename?

Sure.
Screenshot:

Originally Posted by Marco86

I read the report you linked, there is someone complaining there as well that the names do not match anymore.

I read that. Double posting in two separate threads won't make me answer faster and if there is demand, I see it without a separate hint to it

Originally Posted by Marco86

Would it be possible to include a new twitter plugin setting like a custom filename field, like present in the instagram and youtube plugin, where everyone can choose the filename they want with the 4 variables: *date*, *uploader*, *tweetID* and *orgfilename*?

That is already possible using Packagizer rules.
We've explained this multiple times in this forums - short version here:
- Currently only some plugins for popular websites have their own settings for naming schemes
- The Packagizer is the tool for custom filenames and yes it is limited in featutrs but enough for the majority of the needs of our users
- More detailed explanations can be found throughout our forums. Also there is a thread in which we are collecting ideas regarding new/netter naming schemes to make the existing Packagizer more powerful - especially this one: https://board.jdownloader.org/showthread.php?t=90744

Originally Posted by Marco86

he current plugin filenames are good, but do not include the orgfilename at the end, instead it's 0, 1 ,2, 3. This is not ideal because if you downloaded a twitter image in the past with the original filename from the website, and now want to use the windows search to see which account the image belonged to and which date it was posted, that is not possible with the current plugin filenames because the original filename is not included.

As explained the old filenames were not "real original filenames" and this was a mistake, not an intentional feature.

Originally Posted by erarata

How would one restore "<tweetID>_<FilenameWithoutExt>.ext" using packagizer?

See beginning of this post (screenshot).

Originally Posted by erarata

My research so far has shown that we're not able to pull in parts of the url itself as the filename[...]

Sure - that is even included in our examples:
https://support.jdownloader.org/Know...to-filename/20

Originally Posted by erarata

and using "mediaid" via the twitter plugin is not the same as the tweet id.

Correct. The property you want to use is "tweetid".
Properties are explained a bit more in depth here:
https://support.jdownloader.org/Know...-packagizer/20
--> Search for "jd:prop" and "Plugin specific properties"

[pspzockerscene]

Again regarding those filenames:

Please don't misunderstand me I will not be this kind of dev who fixes a bug or adds a new feature which makes things worse for some of the users and then clings on to that for no reason.
I'm not using the twitter plugin myself so it's hard for me to empathize with the user.
However I do like options and customizability so this is what I will add as a feature.

Even though, as described, what you want is possible using a single Packagizer rule, I will add a plugin setting to be able to get back those filenames in an easy way.
This requires some refactoring which will require some time.
I will come back to this thread once I got something working here.

[erarata]

Originally Posted by pspzockerscene

Correct. The property you want to use is "tweetid".
Properties are explained a bit more in depth here:
**External links are only visible to Support Staff**...
--> Search for "jd:prop" and "Plugin specific properties"

Oh, great! That makes it simple, then. Part of my confusion in this case is that "tweetid" does not appear to be in the table here:

https://support.jdownloader.org/Know...ed-features/24

[pspzockerscene]

Originally Posted by erarata

Part of my confusion in this case is that "tweetid" does not appear to be in the table here

I need to admit I completely forgot about that article.
As stated in the main Packagizer article we do usually not provide any lists of available plugin properties at all so far - source:
https://support.jdownloader.org/Know...-packagizer/20

I've updated the twitter support article- and added more missing properties.

You can find all properties in the JD source code under jd.plugins.decrypter.TwitterComCrawler -> Mostly the variable names starting with "PROPERTY_".

[pspzockerscene]

Screenshot of the upcoming new twitter plugin settings for custom filenames:


Short explanation:
- Filenames of items added before won't change
- Filenames of items added after these changes can change "live" so if you change the setting, filenames will change as soon as you run a manual availablecheck or start downloading
- Default = Auto = "Original+" -> "Original plus", resembling the old format you guys wanted back
--> I've called that one "Auto" so I can change it at any time without needing to think about it. Set it to "Original+" to be sure that the scheme won't change automatically in the future.
- I've also added "Original+2" which is similar to the old format but it also includes the name of the uploader
- Feel free to suggest more filename schemes for twitter but if you want something exotic, I will still point you towards our Packagizer feature

For all changes mentioned here, the following information applies:

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-