JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 16.11.2020, 17:23
RickyPickles RickyPickles is offline
Modem User
 
Join Date: Oct 2020
Posts: 4
Default 4chan file names

When media files are posted in 4chan.org threads, the file names get changed to a random series of numbers. But the original title is what's shown as the URL text linking to the file.

Would it be at all possible to have LinkGrabber automatically use the URL text as the file name...?

The URL text also includes the file extension, so the switch would be seamless.

I have a Chrome extension called '4chan X' that does this via a little download icon next to the URL text. Unfortunately though there's no option to scrape the whole thread, so you have to go one by one and that gets pretty tiresome. Would be awesome to just use JDownloader instead!
Reply With Quote
  #2  
Old 16.11.2020, 17:36
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 57,372
Default

Hi,
please post example URLs.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3  
Old 17.11.2020, 00:13
RickyPickles RickyPickles is offline
Modem User
 
Join Date: Oct 2020
Posts: 4
Default

SFW examples
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

NSFW example
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #4  
Old 17.11.2020, 00:24
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 57,372
Default

Hm seems like I don't understand the issue you are having.
If I take this as an example:
bla.4chan.org/wsg/1601234567416.webm
--> Filename would be:
1601234567416.webm

Where is the problem / how/where are other filenames supposed to come from?

-psp-
EDIT

Fo you mean the "file:blabla.mp4" nems in their html code?
We could try to crawl- and set those filenames but at this moment, serverside filenames are chosen which is not per-se considerable as a bug.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #5  
Old 17.11.2020, 09:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 70,264
Default

@pspzockerscene: yes, that is what he means
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 17.11.2020, 20:38
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 57,372
Default

Done.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #7  
Old 18.11.2020, 19:14
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 314
Default

Please make this feature optional.
Reply With Quote
  #8  
Old 18.11.2020, 19:46
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 57,372
Default

Quote:
Originally Posted by Derp View Post
Please make this feature optional.
I will create a Packagizer rule for you tomorrow to get the old names back.

If more users want the same, I may add a plugin setting for that.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #9  
Old 18.11.2020, 19:59
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 70,264
Default

I've updated ticket, we will add plugin setting for this
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 18.11.2020, 21:51
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 314
Default

Quote:
Originally Posted by pspzockerscene View Post
I will create a Packagizer rule for you tomorrow to get the old names back.

If more users want the same, I may add a plugin setting for that.

-psp-
Tried that, but didn't work, see attached pic of the rule. JD2 was still using those new filenames.

Quote:
Originally Posted by Jiaz View Post
I've updated ticket, we will add plugin setting for this
Great!
Attached Images
File Type: png 4chan_packagizer.png (70.1 KB, 2 views)
Reply With Quote
  #11  
Old 19.11.2020, 08:54
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 70,264
Default

@Derp: that rule won't work because orgfilename is the one set by plugin. pspzocker was talking about a rule that uses the old names but I (and you) prefer to have plugin setting for this He will work on this
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 19.11.2020, 13:20
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 57,372
Default

@Derp
It won't work this way - you'd have to get the names out of the direct-URLs crawled by the crawler.
Anyways - I've added a setting for the next update.
See Settings -> Plugins -> boards.4chan.org

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #13  
Old 19.11.2020, 14:32
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 57,372
Default

I've just prepared a huge update for our 4chan plugin.
This includes:
- Faster crawling
- Can now crawl up to 20 pages of a category (see plugin settings - default = limited to 1)

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #14  
Old 19.11.2020, 19:09
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 314
Default

Build Thu Nov 19 18:44:33 CET 2020

The plugin is completely broken now.
Regardless of the server-filename-setting, JD2 is adding every file as a .webm and checks it as such, which of course results in all the files which aren't webm-files to not be found.

Also, the default packagename just uses the 4chan-board-acronym and not the board-name, i.e.
"4chan org - sp - 103946012" instead of
"4chan org - Sports - 103946012".

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff** <-- this thread happens to have at least one file of every type (jpg, png, gif, webm) and two webms for the auto-packaging.


------------------------------
Quote:
Originally Posted by Jiaz View Post
@Derp: that rule won't work because orgfilename is the one set by plugin.
Quote:
Originally Posted by pspzockerscene View Post
@Derp
It won't work this way - you'd have to get the names out of the direct-URLs crawled by the crawler.
No idea how that would work, was the first time I used it.
In the "Dynamic Variables" there's no entry for that, but "packagename" is listed twice.

Maybe I also found a bug:
When sourceurl is checked, and the user checks "rename", the checkbox is cleared and disabled.
After unchecking "rename", the sourceurl-checkbox is checked again, but not re-enabled.

Last edited by Derp; 19.11.2020 at 19:55. Reason: Added build-info
Reply With Quote
  #15  
Old 19.11.2020, 21:50
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 57,372
Default

Quote:
Originally Posted by Derp View Post
No idea how that would work, was the first time I used it.
In the "Dynamic Variables" there's no entry for that, but "packagename" is listed twice.
It works totally different - see board search for more examples.

Quote:
Originally Posted by Derp View Post
Maybe I also found a bug:
When sourceurl is checked, and the user checks "rename", the checkbox is cleared and disabled.
After unchecking "rename", the sourceurl-checkbox is checked again, but not re-enabled.
Jiaz will check this tomorrow.

I've fixed this issue - I made a simple but silly mistake with that wrong file-extension and I didn't know that the long/full board title was so important.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #16  
Old 19.11.2020, 22:37
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 314
Default

Quote:
Originally Posted by pspzockerscene View Post
It works totally different - see board search for more examples.
I'll pass, thanks.
Re: "Packagename" being available twice: It's because of line 284 or 301 in PackagizerFilterRuleDialog.java

Quote:
Originally Posted by pspzockerscene View Post
I've fixed this issue - I made a simple but silly mistake with that wrong file-extension and I didn't know that the long/full board title was so important.
Bug seems to be fixed and the filename-feature seems to work in both states, too.

Thank you very much.

EDIT2:
Just had an interesting case:
Two different files in a thread used the same upload-filename, and JD2 treats them as mirrors of each other.
Because of that only one of them is downloaded
With server-filenames, both files are downloaded.
**External links are only visible to Support Staff****External links are only visible to Support Staff**
filename is "file.png"

EDIT3:
Considering how slow the image-server already is, a fast linkcheck option would make sense, too.

Last edited by Derp; 20.11.2020 at 00:47.
Reply With Quote
  #17  
Old 20.11.2020, 15:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 70,264
Default

@Derp: Thanks for the info, those two Packagename are different ones. I will change description
__________________
JD-Dev & Server-Admin
Reply With Quote
  #18  
Old 20.11.2020, 15:07
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 57,372
Default

Quote:
Originally Posted by Derp View Post
Considering how slow the image-server already is, a fast linkcheck option would make sense, too.
"Fast linkcheck" is already always used.

You can still see the filesizes right away because this information is right away for 4chan content.
Where/how do you think that a linkcheck process is active- and slow?
I don't see how we could speed anything up here.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #19  
Old 20.11.2020, 15:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 70,264
Default

@pspzockerscene. maybe he's taking about the request limit
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 20.11.2020, 15:16
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 57,372
Default

Quote:
Originally Posted by Jiaz View Post
@pspzockerscene. maybe he's taking about the request limit
Yeah but we should respect that.

Afaik they do not enforce that rate-limit.
We could add a setting to prefer using the website without rate-limit but that makes no sense as we'd have to maintain the website code too then.

The rate-limit shouldn't be too restrictive in the way it is atm.

-psp-
EDIT

Okay by request-limiting only that API subdomain it will be "faster".
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 20.11.2020 at 15:36.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 18:54.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.