#1
|
|||
|
|||
Twitter - text filenames need fixing
log : 08.07.23 00.38.25 <--> 08.07.23 01.12.11 jdlog://8036311370661/
I set naming to auto and it's work fine couple of days ago. After today update that add more naming option for twitter, the new name is changed from [date][tweet_id][originalFilenameWithoutExt][ext] to [date][originalFilenameWithoutExt][some random string][ext] I tried changing the option from auto to Original+ and delete file and recrawling again but the name is still the same without tweetid. edit: I try changing it to Original+2 and Plugin and it's show the result of Original+ setting. Here's the result of select setting. Auto, Original+ [date][originalFilenameWithoutExt][some random string][ext] Original+2 [date][tweet_id][originalFilenameWithoutExt][some random string][ext] Plugin [date][tweet_id][originalFilenameWithoutExt][ext] So it's work out for me at the moment to choose "Plugin" for "original+" option. Btw there're [some random string] add at the end of every files now after the [originalFilename] for some option. Is it a bug or twitter changed how they naming their files? Last edited by JTK; 07.07.2023 at 23:12. |
#2
|
|||
|
|||
Everything is working for me, check if you have a packagizer rule enabled. Try downloading and installing the new jdownloader version in a new folder and see if the fresh installation has the same problem.
|
#3
|
|||
|
|||
Twitter - text filenames need fixing
All tweets without media have only the tweetID as the text filename, this is true for profile links and single status links, no matter which API or filename scheme is chosen (except the plugin filename).
The filename scheme Plugin uses the text filename format date_username_tweetID.ext, this is also how the text filenames were handled in all previous versions with original and plugin filenames. I think the text filename format should stay the same for filename scheme original and plugin, and be changed for filename scheme Original+ and Original+2. The text filename format for Original+ should be date_tweetID.ext. The text filename format for Original+2 should be date_username_tweetID.ext, same as the plugin filename scheme. Example links: **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** Also the text filenames have been changed, the textfiles always took the name of the last image or video in a tweet, now it's random. This was a recent change since May 2023 or sooner. Please change this back, as I'm skipping files based on already existing filenames. Example 1: **External links are only visible to Support Staff****External links are only visible to Support Staff** text filename is: FvOaW4yaYAAbZ3k.txt should be: 5IQArLM9GvBLz9eA.txt Example 2: **External links are only visible to Support Staff****External links are only visible to Support Staff** text filename is: E2nK9eJe8dQFMMx_.txt should be: FuzOo3FaIAAHM2g.txt |
#4
|
|||
|
|||
I don't know if it's the same but I was going to create a new topic that the date is repeated twice, I don't know if it's my fault.
For example: **External links are only visible to Support Staff****External links are only visible to Support Staff** The name of the file comes out like: 2023-06-30-2023-06-30_1674810336149381121_OpHLMpClNOB2P31h.mp4.mp4 :confused: It had been a few months since I updated JDownloader 2, that's why I hadn't reported it before. I don't know if it's the same problem as this topic. Log: 09.07.23 18.51.47 <--> 10.07.23 00.51.20 jdlog://7636311370661/
__________________
- Ramón V M |
#5
|
|||
|
|||
@RedViper I get the name 2023-07-01_1674810336149381121_OpHLMpClNOB2P31h.mp4, I think there is something wrong with your jdownloader installation, check if you have any packagizer rules enabled, if not try a fresh jdownloader installation in another folder.
I'm still using the Jdownloader version #47140 from November 26, 2022 if you are able to use packagizer rules you can get the filenames you want. If you want that version as a backup write me a pm, even extmatrix plugin is working with that old version while not working in the new one. |
#6
|
||||
|
||||
Merged related twitter threads.
@RedViper The last update changed the filename behavior once again. Go to Settings -> Plugins -> twitter.com -> Filename scheme and change it to your desired value. Changes of next twitter plugin-update: - fixed fallback-name (= only tweet_id) being used for single .txt downloads even when FilenameScheme is set to AUTO - fixed "original filename without ext" in .txt files coming from the first crawled media item and not the last Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#7
|
|||
|
|||
Thank you pspzockerscene!
I didn't know this, I always named the images with the original file name and the videos with the tweet ID. I hope I don't make a mess with files that I have already downloaded, but I will start downloading the images and videos with the original name
__________________
- Ramón V M |
#8
|
||||
|
||||
You can choose your preferred filenaming scheme.
By selecting the one which was used before, the filenames will be identical to the ones of files you've downloaded before.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#9
|
|||
|
|||
Quote:
Thanks again pspzockerscene! But I think it will be for the better like this because when there was more than one video in a Tweet I always had to list them because they all ended up having the same name. Apparently putting the original file name in the videos as well will be better, I'll just have to be careful not to download a video I already have
__________________
- Ramón V M |
#10
|
|||
|
|||
Please do not mark this thread as solved, as the issues have not been solved yet.
Textfilenames: There are still a lot of tweets remaining where the textfile name is not the last picture or video name. Here are examples: 4 pictures, taking name of 1. (should be: FLr0B5UVkAAyrlK.txt, is: FLr0B5OVQAoQFZ4.txt) **External links are only visible to Support Staff**... 4 pictures, taking name of 2. (should be: FKYVmlrWQAUrNrJ.txt, is: FKYU5nSXwAErv6x.txt) **External links are only visible to Support Staff**... 4 pictures, taking name of 3. (should be: FJT96m3XMAgENaW.txt, is: FJT96m0WYAMqxE7.txt) **External links are only visible to Support Staff**... 3 pictures, taking name of 1. (should be: FSayaraXEAEyFNR.txt, is: FSayUzMWUAAcQP1.txt) **External links are only visible to Support Staff**... 3 pictures, taking name of 2. (should be: FSHjm39WUAEf7Po.txt, is: FSHjm3-WYAU_OGA.txt) **External links are only visible to Support Staff**... 2 pictures, taking name of 1. (should be: FGCXhvAXwAYy91B.txt, is: FGCXhujXIAEm6ei.txt) **External links are only visible to Support Staff**... 3 videos, taking name of 1. (should be: JNypQxxZJUIgxCmO.txt, is: O1Xgw7-M51klh19u.txt) **External links are only visible to Support Staff**... 1 video and 2 pictures, taking name of 2. (should be: Fve_1TtWwAEHUUD.txt, is: Fve_1TzXwAECw7Y.txt) **External links are only visible to Support Staff**... 4 gifs, taking name of 1. (works with New API only when parsing single link, profile link will work with Old and New API) (should be: Fw76AyVWIAA3P-1.txt, is: Fw75_bIWAAAzomq.txt) **External links are only visible to Support Staff**... 4 gifs, taking name of 3. (should be: Fw7-uXKX0AAdvTb.txt, is: Fw7-uW2XsAEsapA.txt) **External links are only visible to Support Staff**... 3 gifs, taking name of 1. (should be: Fw2kkYYXgAE5aF6.txt, is: Fw2kjlWXwAIjfLR.txt) **External links are only visible to Support Staff**... 2 gifs, taking name of 1. (should be: FxlHcwIWcAEwo4r.txt, is: FxlHcNBXsAEPL_G.txt) **External links are only visible to Support Staff**... 1 gif and 1 picture, taking name of 1. (should be: FyOiGq7XsAE_oJ5.txt, is: FyOiF2OWYAAo31M.txt) **External links are only visible to Support Staff**... These tweets' textfiles are taking the name of the last media when using previous JD2 versions, so this should be fixable. Textfile format: The text filename format was now changed to date_username_tweetid.ext for the AUTO setting, which is Original+, but Original+ has the format date_tweetid_originalfilename.ext, which does not include the username, so if someone would chose to download with the Original+ setting in the format date_tweetid_originalfilename.ext, they likely do not want to have the username in the filenames. But the current change would be perfect for the Original+2 setting. And this is what I was asking for, to have the Original+2 textfiles (of tweets without media) in the same format as the current Plugin filename scheme textfiles. But choosing Original+2 still gives only the filename 1678098028849143809.txt for the following tweet **External links are only visible to Support Staff**..., while giving the correct filename 2023-07-10_elonmusk_1678098028849143809.txt when choosing Plugin filename scheme, but I do not use the Plugin filename scheme as the original filenames are not included. I also think the AUTO settings should not include some hidden features. If the AUTO setting is the Original+ setting, then it should be exactly the same as choosing Original+ manually. Also I think it is very important to have the accountname in the filename, because if you load a profile link, jdownloader will also (rightfully so) download the tweets from other accounts that were quoted, without having the accountname in the filename you wouldn't know. |
#11
|
|||
|
|||
Marco86 you should definitely add the Log as well, so they can catch those issues.
__________________
- Ramón V M |
#12
|
||||||||
|
||||||||
Quote:
I've changed that prefix once again. Quote:
Please either post them again (1-2 examples should be enough) or put them into a .txt file and attach that to your next reply. Quote:
Also there is no such thing as a "text-filename format". The text filename format orientates itself on the other given filenames. Example: twitter.com/50cent/status/1679127118603509762 Filenames in AUTO mode: Code:
2023-07-12_1679127118603509762_F011NYSWAAUXNgt.jpg Code:
2023-07-12_1679127118603509762_F011NYSWAAUXNgt.txt - Prefer FilenameScheme.ORIGINAL_PLUS if possible - Use FilenameScheme.PLUGIN if ORIGINAL_PLUS is not possible EDIT Now I get it - tweets without media but only text. I will re-check this but please also provide separate example URLs so I got your test-cases. I want to mention again that we're usually not adding extra filename settings for single plugins - I urge you to read the following thread and be part of the discussion about the new filename replacer feature: https://board.jdownloader.org/showthread.php?t=90744 Quote:
--> But this is exactly what FilenameScheme.ORIGINAL_PLUS_2 does or am I missing something? Quote:
Quote:
Also I've added further explanation of it in this post - AUTO does not simply select any of the other pre-selectable FilenameSchemes. Quote:
Quote:
I'm asking because re-tweets are not crawled by default. If an issue is 100% reproducable with example links, I do not need any logs.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 13.07.2023 at 14:39. |
#13
|
||||
|
||||
[Internal] Changelog for the next update:
TwitterComCrawler: - updated filename handling: for schemes with 'relatedOriginalFilename': allow usage of them even if relatedOriginalFilename is not given RE forum 93871 - updated text crawl setting to ENUM based setting: Adds ability to completely disable tweet text results RE forum 93874 - fixed 'related_original_filename' not necessarily being the name of the last media item due to usage of HashMap vs LinkedHashMap RE forum 93871 Btw. we're open source - you can view the code here: jd.plugins.decrypter.TwitterComCrawler -> Function 'setFormattedFilename' (line 817 and below) Bitte auf das nächste CORE-Update warten! Please wait for the next CORE-Update! Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 14.07.2023 at 12:33. Reason: Updated text-block regarding CORE-updates |
#14
|
||||
|
||||
CORE-Updates have been released!
All announced bugfixes and features are live! Please update your JDownloader and report any issues you find asap. If this thread gets marked as "[Solved]" by our forum staff you can still post in it and we will read- and reply to it! CORE-Updates wurden released! Alle angekündigten Bugfixes/Features sind nun verfügbar! Bitte JDownloader updaten und eventuelle Bugs schnellstmöglich an uns melden. Falls dieser Thread vom Team als "[Erledigt]" markiert wird, kannst du weiterhin darin antworten und wir lesen/beantworten auch solche Threads! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#15
|
||||
|
||||
@Marco86
Just another example why a generic filename formatting system is better than e.g. what is implemented in the twitter plugin now: Different users will want different filename schemes which may result in a lot of pre-sets: Click Again: In this case that's fine as the twitter plugin is a very popular plugin and we do not have a good generic filename formatting system yet. Either way I hope it's finally working as intended and in the way you wanted it to work.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#16
|
|||
|
|||
Quote:
So much so that when the initial update to the filename scheme hit, I was really besides myself on whether to just pop in here and inquire about getting the old naming scheme back. I'm throwing my cents here on the matter more than anything, since I think I have my problem handled as is (hopefully) Because as a long time user, I have a terabyte of scrapes with the convention of <tweet_id>_<filename.ext>, which I have problems with, but this is one of those things when scraping for archival purposes I have to bear the burden of at this point. So any changes down the line in an attempt to improve the naming scheme is a bother, especially since when these updates hit, I never know what is in them because I just hope and assume the software will function as it did yesterday, just as the sun rises in the morning as it did the day before. Heck, even the introduction of the text files with the tweets was enough to cause a pause for concern. Cause on one hand, yeah that's a really useful thing to have and is certainly something I wish I'd had maybe four or five years ago. On the other, it seems really weird to suddenly scrape. But also on the other, its naming scheme is close to that of the images/videos and they are text files in the kilobytes. I think personally down the line, it may just be better to do something where a user has the ability to declare in the plugin itself the filenaming scheme like they would with the packagizer. Right now, I have the plugin set to the "original" scheme in the plugin and then in the packagizer I have it so that the files are named with this scheme in the packagizer rules I have for twitter "<jd:prop:tweetid>_<jd:orgfilename>" to somewhat match how the plugin handled scrapes. I say somewhat because I don't know if there's an edge case I'm unaware of. |
#17
|
|||
|
|||
@pspzockerscene Thank you very much, the issues have been fixed now.
I have found another issue: When loading single links with the New API, the link is not recognized if the uploader has restricted who can reply. **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** @Akasen You should really use one of the new formats, why wouldn't you want to have the date included, especially if you are a scraper. I also have a lot of files in the useless format <tweet_id>_<filename.ext>, but only because JD2 forced this format without me knowing that packagizer rules or plugin settings exist. Maybe some notifications can be added to the program. |
#18
|
|||
|
|||
Quote:
So what I don't need is to have a folder for one account now have duplicates files between <post-date>_<tweetid>_<imageid> and <tweetid>_<imageid> Were I to go back in time and setup my own rules, I probably would have done <post-date>_<tweetid>_<imageid>_<mediaindex>. But I'm multiple thousands of folders and accounts in too deep now. I'm just lucky that the tweet ID works well enough to organize items by when they were posted |
#19
|
||||
|
||||
Quote:
Quote:
https://board.jdownloader.org/showthread.php?t=90744 Short summary:
Quote:
https://board.jdownloader.org/showthread.php?t=93929 Please keep in mind that the placeholder hich is titled "Original Filename" is simply using the last filename set by the plugin as in that context, that is the original filename. Here is a simplified explanation of how filenames in JD work atm:
@Marco86 I will reply to your post #17 and Akasen to your post #18 soon in a separate reply.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#20
|
||||
|
||||
Now responding to posts #17 and #18:
Quote:
Quote:
The fix will be released with our next set of CORE-updates. As explained to Akasen, feel free to test changes before official release using the dev version: https://support.jdownloader.org/Know...up-ide-eclipse --> While some general software knowledge is good to be able to follow these instructions, you do not need to be a developer to be able to follow them! I [partially] disagree. I'm counting this as an advanced set of features which the user will usually have to find on his own. JDownloader is a quite advanced application so in order to find such functionality, I do expect the user to browser the applications' settings in order to find such features. At least that's the way I'm doing it with software I'm using. I'm even going through all the settings of a new software when I'm not searching for something in particular though that's just my way of handling it. Quite often if you think "there must be a setting for this", investing these 5 minutes to find it will reward you. Just a sidenote: We do have a support category dedicated to plugin settings: https://support.jdownloader.org/Know...dex/24/plugins While modern apps often contain some kind of instructions when using them the first time, I do not consider JDownloader a 'modern' application and I also do not see us as the current team ever finding the time to modernize it. Quote:
This way you could use external applications to change filenames to your needs afterwards - at any point of time. We do have tickets/threads about putting tags in files e.g.: https://board.jdownloader.org/showthread.php?t=84246 --> They're mostly about audio files but the concept for others is the same. That is a feature which could be addressed whenever someone will be working on the other feature - the generic filename builder/replacer.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#21
|
|||
|
|||
Yeah I get that overall this is very much a tricky situation here having had to come back and forth to the forums either to read over various posts or to report on something.
Quote:
So it came to me as the overall more logical answer to that once I found myself coming to. I can't really respond to most other points here, mostly because the tagging thing I'd have to look more into with relation to images and video. I'm fairly married to Jdownloader at this point as a tool. I do use various other specialized tools, and it probably would just be in my better interest to use a CLI program and save a script to run through and do the same job I have Jdownloader do. Maybe this is also a good time for me to consider doing that. |
#22
|
|||
|
|||
I tested the Original with tweetID option (#48037), the names for textfiles without media are tweetID.ext only, this should be date_user_tweetID.ext, at least this is how these names were always handled.
|
#23
|
||||
|
||||
Quote:
Quote:
The new logic for all schemes with original filename parts is: Build filename segment by segment with underscore in between. If the "original filename part" is not given, just don't include it. If you need a solution for that, I may add an option like "Fallback filename Scheme" though at this moment I don't know what would be the best logic here because maybe you'd want to have a different fallback scheme for each item or each target-scheme. ...or call it "preferred filename scheme for single textfiles witout media-items in related tweeet"?
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 24.07.2023 at 14:35. Reason: Fixed typo |
Thread Tools | |
Display Modes | |
|
|