JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 07.03.2024, 23:43
Tock Tock is offline
Modem User
 
Join Date: Sep 2023
Posts: 2
Default kemono.party kemono.su Organize downloaded files by post

Hello,

I'm trying to figure out how to download images from Kemono.su while organizing them by post, such that the folder structure is something like:

JDownloads\KemonoArtist\Post1\Image1.jpg ... Image5.jpg
JDownloads\KemonoArtist\Post2\Image1.jpg ... Image5.jpg
JDownloads\KemonoArtist\Post3\Image1.jpg ... Image5.jpg

Unfortunately the 'Adopt Folder Structure' rule doesn't seem to apply to Kemono, since they're posts and not folders I suppose, though that's essentially what I'm trying to do.

This is about as far as I've gotten from hours of googling: **External links are only visible to Support Staff****External links are only visible to Support Staff**

URL in question (NSFW): **External links are only visible to Support Staff****External links are only visible to Support Staff**

Thanks

edit: Ideally I'd prefer to use the Page title rather than the Post ID, but couldn't figure out how to get there.

Last edited by Tock; 07.03.2024 at 23:45. Reason: added info
Reply With Quote
  #2  
Old 08.03.2024, 11:28
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

Using Packagizer rules with some dynamic plugin properties, you can do something like:
If Sourceurl -> Contains -> kemono.su
...then set
Download Directory -> Downloads\00TestDLs\<jd:prop:userid>\<jd:prop:title>

Important notice:
Our ticket system & knowledgebase are currently under maintenance.
If this is still the case while you are reading my posts and you can't access help articles linked by our staff, use the Internet archive/wayback machine to be able to view those articles:
archive.org/web/
Enter the URL -> Click on "Browse history" -> Select one of the latest dates available
If the date you selected does not lead you to the support article try the next oldest one.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
  #3  
Old 08.03.2024, 17:38
Tock Tock is offline
Modem User
 
Join Date: Sep 2023
Posts: 2
Default

Ah hah! Perfect! Thank you!
Reply With Quote
  #4  
Old 21.03.2024, 22:06
VanTuz VanTuz is offline
Linkgrabbing Monster
 
Join Date: Oct 2018
Posts: 88
Default

When a link to some file sharing site like mega appears in a post, then such links are not sent to the post folder, but are dumped into one pile. How do I make such links fall into the post folder?
Reply With Quote
  #5  
Old 22.03.2024, 16:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,006
Default

@VanTuz: Please provide example links
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 22.03.2024, 21:13
VanTuz VanTuz is offline
Linkgrabbing Monster
 
Join Date: Oct 2018
Posts: 88
Default

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #7  
Old 23.03.2024, 21:53
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,006
Default

@VanTuz: Would be nice to get a post with included link? You provided user overview and checked a few but did not find those in question. Please help us to save time and provide direct example links and then we can check and provide better help/solutions.
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 23.03.2024 at 22:25.
Reply With Quote
  #8  
Old 24.03.2024, 12:58
VanTuz VanTuz is offline
Linkgrabbing Monster
 
Join Date: Oct 2018
Posts: 88
Default

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #9  
Old 25.03.2024, 15:27
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

Quote:
Originally Posted by VanTuz View Post
When a link to some file sharing site like mega appears in a post, then such links are not sent to the post folder, but are dumped into one pile. How do I make such links fall into the post folder?
That behavior is not a bug since MEGA links will typically need to go into the file/folder structure they provide, see:
https://support.jdownloader.org/de/k...lder-structure

If you really wanted them to go into the profile- or post-folder, you could maybe do this via Packagizer rule based on the "source URL".
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
  #10  
Old 25.03.2024, 16:27
VanTuz VanTuz is offline
Linkgrabbing Monster
 
Join Date: Oct 2018
Posts: 88
Default

I tried, but I can't make the files downloaded from mega or other file sharing sites go to the post folder, they end up in a separate folder.
Reply With Quote
  #11  
Old 25.03.2024, 16:28
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

Please provide all Packagizer rules and/or EventScripter scripts you've tried so far.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
  #12  
Old 25.03.2024, 16:53
VanTuz VanTuz is offline
Linkgrabbing Monster
 
Join Date: Oct 2018
Posts: 88
Default

Download url: kemono* (or did not use this item)
source url: mega*
download folder kemono\<jd:prop:postid>-<jd:prop:title>

I didn't use scripts because I don't know how to use them.
Reply With Quote
  #13  
Old 25.03.2024, 17:12
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,006
Default

@VanTuz: Any customization? for me it worked fine for your first example link. any custom packagizer rule or does this randomly work/not work? most likely examples with single links work but those with folder links do not work
issue comes up when the linked folder has subfolders and then placing everything in same/one package will mess up the files structure
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 25.03.2024, 17:13
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,006
Default

@VanTuz: source and download url are mixed, source = kemono and download is mega(as you download from mega).
but this will not work that way as the properties are not part of mega/dropbox folder
so for example you could create packagizer rule that parses postID from URL and then places everything into custom package
only kemono links do have access to those properties but cannot access them from other links (eg mega or dropbox)
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 25.03.2024 at 17:16.
Reply With Quote
  #15  
Old 25.03.2024, 17:36
VanTuz VanTuz is offline
Linkgrabbing Monster
 
Join Date: Oct 2018
Posts: 88
Default

Quote:
Originally Posted by Jiaz View Post
@VanTuz: Any customization? for me it worked fine for your first example link. any custom packagizer rule or does this randomly work/not work? most likely examples with single links work but those with folder links do not work
issue comes up when the linked folder has subfolders and then placing everything in same/one package will mess up the files structure
If you download a specific post, there are no problems, but if you start scanning the author's page, then the posts are sorted into folders, and files from external file sharing sites are added separately. That's why I gave links to the authors' pages for the first time.

Quote:
Originally Posted by Jiaz View Post
so for example you could create packagizer rule that parses postID from URL and then places everything into custom package
How?
Reply With Quote
  #16  
Old 25.03.2024, 18:09
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,260
Default

I've tested Jiaz' idea because it sounded good and it makes sense but it doesn't work.

Explanation:
When a kemono.su user profile is crawled, JDownloader does never "see" or process the single "/post/..." URLs.
The posts are crawled as part of the user crawler because this is how their API works (reference: kemono.su/api/schema).
This means that from non-kemono URLs crawled as part of a user crawl process, there is no "source URL" reference on single post kemono URLs.
This ultimately means that for all items resulting from a profile crawl process, all you golt as a source URL is the kemono profile URL ("kemono.su/portal/user/username").

@VanTuz what you want is either _impossible_ or only using the following tricks:
  • An EventScripter script (though to be honest, I still don't know how you want to match the results to find their origin)
  • or: Modify the JDownloader source code of the kemono crawler so that the kemono plugin puts all "plugin properties" on all results [also on non-kemono results] so that you can access them also on e.g. mega.nz items.
  • or: Modify the JDownloader soure code of the kemono crawler so that the user profile crawler will first return all single post URLs and then process them. This will make it waay slower and also puts an unnecessary load on the kemono API but in the end you will have the single post URLs as "sourcer URL" reference which is needed to accomplish your goal here.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

-----------------------------------

On Vacation / Im Urlaub
Start: 2024-02-08
End: TBA

Reply With Quote
  #17  
Old 25.03.2024, 18:37
VanTuz VanTuz is offline
Linkgrabbing Monster
 
Join Date: Oct 2018
Posts: 88
Default

It's clear. It's a pity that nothing can be done about it, because I'm not a programmer. It's just that when yiff. party existed, the program put posts in folders, plus it did this with pixiv, but for some reason it doesn't want to work with this site. And I hoped that the program sees that a link to a mega was found on such and such a post, so you can put it in a folder with the name of the post in which the link was found, but since the API of the site does not allow this, I will download as is.
Reply With Quote
  #18  
Old 25.03.2024, 18:59
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,006
Default

@VanTuz: we have one more idea and currently testing it
Update:
Good news, I have found a way! Bad news, it will take some time before the required change will be available in update.will further discuss this with pspzockerscene
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 25.03.2024 at 22:40.
Reply With Quote
  #19  
Old 06.04.2024, 12:54
VanTuz VanTuz is offline
Linkgrabbing Monster
 
Join Date: Oct 2018
Posts: 88
Default

There is a new problem - the program does not pick up links from the dropbox if you insert a link to a post that contains a link to the dropbox. If you just add a link to the dropbox, everything is fine.
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #20  
Old 06.04.2024, 13:33
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,006
Default

@VanTuz: works fine here, you have to Enable Settings->Plugins->kemono.su
"Crawl http links in post text"
__________________
JD-Dev & Server-Admin
Reply With Quote
  #21  
Old 07.04.2024, 00:15
VanTuz VanTuz is offline
Linkgrabbing Monster
 
Join Date: Oct 2018
Posts: 88
Default

And indeed, it was turned off. I just remember that this setting was turned on and I was afraid that something was broken.
Reply With Quote
  #22  
Old 07.04.2024, 12:46
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,006
Default

@VanTuz: Thanks for the feedback and you're welcome!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #23  
Old 07.04.2024, 16:53
VanTuz VanTuz is offline
Linkgrabbing Monster
 
Join Date: Oct 2018
Posts: 88
Default

I'm trying to download this author - **External links are only visible to Support Staff****External links are only visible to Support Staff** - but out of more than 700 files, the program finds 593.
If I give direct links to posts, it finds 160 files. In addition, the program really does not interfere with picking up links from a text file.
UPD. When I enable the Packagizer rule, the program finds fewer files.

Last edited by VanTuz; 07.04.2024 at 16:57.
Reply With Quote
  #24  
Old 07.04.2024, 21:26
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,006
Default

@VanTuz: sounds like you're running in sort of rate limit and/or crawl-/bot protection. Please provide a debug log for when this happens.

Happens for me too, first try I got around 880 links, next time only 300, next time only 450. the requests are blocking.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #25  
Old 07.04.2024, 21:29
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,006
Default

Quote:
Originally Posted by VanTuz View Post
In addition, the program really does not interfere with picking up links from a text file.
What do you mean by that? Adding an user url requires less requests and more likely to fetch/process more links than 800 inidividual links with minimum 800 requests resulting nearly guaranteed in rate limit or protection
__________________
JD-Dev & Server-Admin
Reply With Quote
  #26  
Old 07.04.2024, 23:57
VanTuz VanTuz is offline
Linkgrabbing Monster
 
Join Date: Oct 2018
Posts: 88
Default

08.04.24 00.55.19 <--> 08.04.24 00.57.55 jdlog://7202411370661/

Last edited by VanTuz; 08.04.2024 at 00:02.
Reply With Quote
  #27  
Old 08.04.2024, 00:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,006
Default

@VanTuz: As I thought/expected, it's caused by rate limit. Please try again with next plugin update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #28  
Old 08.04.2024, 10:49
VanTuz VanTuz is offline
Linkgrabbing Monster
 
Join Date: Oct 2018
Posts: 88
Default

Thanks a lot, everything is working. And this is very pleasing, because gallery-dl ran into that very limit, and jdownloader downloaded everything.
Reply With Quote
  #29  
Old 08.04.2024, 12:18
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,006
Default

@VanTuz: Thanks for the feedback. There was just some little thing missing by work of pspzockerscene but looks like the rate limit handling works good
__________________
JD-Dev & Server-Admin
Reply With Quote
  #30  
Old 25.05.2024, 11:48
VanTuz VanTuz is offline
Linkgrabbing Monster
 
Join Date: Oct 2018
Posts: 88
Default

Previously, when I downloaded individual posts by some author, the files that the author posted on mega fell into the same folder with the files from kemono, but now they fall into a separate folder, which is inconvenient. Is there any way to turn it off?
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 04:30.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.