JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 16.11.2020, 18:23
RickyPickles RickyPickles is offline
Modem User
 
Join Date: Oct 2020
Posts: 4
Default 4chan file names

When media files are posted in 4chan.org threads, the file names get changed to a random series of numbers. But the original title is what's shown as the URL text linking to the file.

Would it be at all possible to have LinkGrabber automatically use the URL text as the file name...?

The URL text also includes the file extension, so the switch would be seamless.

I have a Chrome extension called '4chan X' that does this via a little download icon next to the URL text. Unfortunately though there's no option to scrape the whole thread, so you have to go one by one and that gets pretty tiresome. Would be awesome to just use JDownloader instead!
Reply With Quote
  #2  
Old 16.11.2020, 18:36
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 56,640
Default

Hi,
please post example URLs.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3  
Old 17.11.2020, 01:13
RickyPickles RickyPickles is offline
Modem User
 
Join Date: Oct 2020
Posts: 4
Default

SFW examples
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

NSFW example
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #4  
Old 17.11.2020, 01:24
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 56,640
Default

Hm seems like I don't understand the issue you are having.
If I take this as an example:
bla.4chan.org/wsg/1601234567416.webm
--> Filename would be:
1601234567416.webm

Where is the problem / how/where are other filenames supposed to come from?

-psp-
EDIT

Fo you mean the "file:blabla.mp4" nems in their html code?
We could try to crawl- and set those filenames but at this moment, serverside filenames are chosen which is not per-se considerable as a bug.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #5  
Old 17.11.2020, 10:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

@pspzockerscene: yes, that is what he means
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 17.11.2020, 21:38
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 56,640
Default

Done.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #7  
Old 18.11.2020, 20:14
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 312
Default

Please make this feature optional.
Reply With Quote
  #8  
Old 18.11.2020, 20:46
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 56,640
Default

Quote:
Originally Posted by Derp View Post
Please make this feature optional.
I will create a Packagizer rule for you tomorrow to get the old names back.

If more users want the same, I may add a plugin setting for that.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #9  
Old 18.11.2020, 22:51
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 312
Default

Quote:
Originally Posted by pspzockerscene View Post
I will create a Packagizer rule for you tomorrow to get the old names back.

If more users want the same, I may add a plugin setting for that.

-psp-
Tried that, but didn't work, see attached pic of the rule. JD2 was still using those new filenames.

Quote:
Originally Posted by Jiaz View Post
I've updated ticket, we will add plugin setting for this
Great!
Attached Images
File Type: png 4chan_packagizer.png (70.1 KB, 2 views)
Reply With Quote
  #10  
Old 18.11.2020, 20:59
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

I've updated ticket, we will add plugin setting for this
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 19.11.2020, 09:54
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

@Derp: that rule won't work because orgfilename is the one set by plugin. pspzocker was talking about a rule that uses the old names but I (and you) prefer to have plugin setting for this He will work on this
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 19.11.2020, 14:20
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 56,640
Default

@Derp
It won't work this way - you'd have to get the names out of the direct-URLs crawled by the crawler.
Anyways - I've added a setting for the next update.
See Settings -> Plugins -> boards.4chan.org

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #13  
Old 19.11.2020, 15:32
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 56,640
Default

I've just prepared a huge update for our 4chan plugin.
This includes:
- Faster crawling
- Can now crawl up to 20 pages of a category (see plugin settings - default = limited to 1)

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #14  
Old 19.11.2020, 20:09
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 312
Default

Build Thu Nov 19 18:44:33 CET 2020

The plugin is completely broken now.
Regardless of the server-filename-setting, JD2 is adding every file as a .webm and checks it as such, which of course results in all the files which aren't webm-files to not be found.

Also, the default packagename just uses the 4chan-board-acronym and not the board-name, i.e.
"4chan org - sp - 103946012" instead of
"4chan org - Sports - 103946012".

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff** <-- this thread happens to have at least one file of every type (jpg, png, gif, webm) and two webms for the auto-packaging.


------------------------------
Quote:
Originally Posted by Jiaz View Post
@Derp: that rule won't work because orgfilename is the one set by plugin.
Quote:
Originally Posted by pspzockerscene View Post
@Derp
It won't work this way - you'd have to get the names out of the direct-URLs crawled by the crawler.
No idea how that would work, was the first time I used it.
In the "Dynamic Variables" there's no entry for that, but "packagename" is listed twice.

Maybe I also found a bug:
When sourceurl is checked, and the user checks "rename", the checkbox is cleared and disabled.
After unchecking "rename", the sourceurl-checkbox is checked again, but not re-enabled.

Last edited by Derp; 19.11.2020 at 20:55. Reason: Added build-info
Reply With Quote
  #15  
Old 19.11.2020, 22:50
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 56,640
Default

Quote:
Originally Posted by Derp View Post
No idea how that would work, was the first time I used it.
In the "Dynamic Variables" there's no entry for that, but "packagename" is listed twice.
It works totally different - see board search for more examples.

Quote:
Originally Posted by Derp View Post
Maybe I also found a bug:
When sourceurl is checked, and the user checks "rename", the checkbox is cleared and disabled.
After unchecking "rename", the sourceurl-checkbox is checked again, but not re-enabled.
Jiaz will check this tomorrow.

I've fixed this issue - I made a simple but silly mistake with that wrong file-extension and I didn't know that the long/full board title was so important.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #16  
Old 19.11.2020, 23:37
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 312
Default

Quote:
Originally Posted by pspzockerscene View Post
It works totally different - see board search for more examples.
I'll pass, thanks.
Re: "Packagename" being available twice: It's because of line 284 or 301 in PackagizerFilterRuleDialog.java

Quote:
Originally Posted by pspzockerscene View Post
I've fixed this issue - I made a simple but silly mistake with that wrong file-extension and I didn't know that the long/full board title was so important.
Bug seems to be fixed and the filename-feature seems to work in both states, too.

Thank you very much.

EDIT2:
Just had an interesting case:
Two different files in a thread used the same upload-filename, and JD2 treats them as mirrors of each other.
Because of that only one of them is downloaded
With server-filenames, both files are downloaded.
**External links are only visible to Support Staff****External links are only visible to Support Staff**
filename is "file.png"

EDIT3:
Considering how slow the image-server already is, a fast linkcheck option would make sense, too.

Last edited by Derp; 20.11.2020 at 01:47.
Reply With Quote
  #17  
Old 20.11.2020, 16:07
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 56,640
Default

Quote:
Originally Posted by Derp View Post
Considering how slow the image-server already is, a fast linkcheck option would make sense, too.
"Fast linkcheck" is already always used.

You can still see the filesizes right away because this information is right away for 4chan content.
Where/how do you think that a linkcheck process is active- and slow?
I don't see how we could speed anything up here.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #18  
Old 22.11.2020, 14:20
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 312
Default

Quote:
Originally Posted by pspzockerscene View Post
"Fast linkcheck" is already always used.

You can still see the filesizes right away because this information is right away for 4chan content.
Oh right, forgot about that.

Quote:
Originally Posted by Jiaz View Post
Limiting the request interval to api only will be available with next core update next week. I'm not finished yet
Several years ago the authors of the 4chanX-userscript were asked by 4chan-admins to set the minimum interval of hitting the API to 30 seconds or so, the same as the native inline-extension. But that was before it was cloudflare'd, IIRC.
Considering the JSON already contains the filesizes and JD2 doesn't need to hit the image-server to get that info, it shouldn't be a problem at all.

-----
Another question: Is JD2 somehow limiting the concurrent downloads from cloudflare-protected links or something?
When I start to download a package with ~30 images (~11MiB total) from 4chan with 5 concurrent downloads, it takes about 3 minutes for it to finish, with most of them being stuck at "Starting..." for 10-15 seconds.
Or are you seeing the same behavior and it's being limited server-side? I think it changed to this about a week ago.
EDIT:
Another try yesterday: 140 files, 75 MiB, 3 concurrent downloads, took 8 minutes.

Last edited by Derp; 23.11.2020 at 09:29.
Reply With Quote
  #19  
Old 20.11.2020, 16:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

@Derp: Thanks for the info, those two Packagename are different ones. I will change description
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 20.11.2020, 16:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

@pspzockerscene. maybe he's taking about the request limit
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 06:22.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.