[Developer Feedback required] Kemono.party and coomer.party support

[shinji2009]

hi. now kemono.party moves to kemono.su. is there a way to make jd handle kemono.su links as kemono.party links and not download what was already downloaded from kemono.party?

[pspzockerscene]

Domain kemono.su is already supported.
If it doesn't work, please provide example URLs and a debug-log.

Please post your log-ID here
If your bugreport is about a specific website which JD supports via plugin, please also provide example URLs which can be used to reproduce the issue you are having.

Bitte poste deine Log-ID hier.
Falls dein gemeldetes Problem ein Problem mit einer Webseite ist, die per Plugin unterstützt wird, stelle bitte zusätzlich Testlinks zur Verfügung, mit denen sich dein Problem nachstellen lässt.

-psp-

[shinji2009]

i know it is supported, my question is not about support

[pspzockerscene]

The duplicate detection should already work across domains if you mean that.
If it doesn't work for you, please provide example URLs.

[shinji2009]

basically every artist crawls anew. except txt files.
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

This time kemono has also changed the domain inside the data-URLs.
I've tried to add a better duplicate detection some time ago but that broke backward compatibility.
I did then revert those changes upon your request.
Reference:
https://board.jdownloader.org/showth...055#post525055

I will re-add the new method for the next set of CORE-updates.

You will still loose backward-compatibility for duplicate check on previously added items.
Here is what you will need to do after(!) said update:
1. Remove all previously added/downloaded items from your downloadlist.
2. Re-add those items to your downloadlist.
3. Now duplicate detection will also work against new items and independent from the domain used inside the URLs.

About text files:
Those didn't exist in beforehand so they're already using the new dupe-check IDs -> Text-files dupecheck will work cntinuously for old- and new items.

[shinji2009]

if backward-compatibility not working now and will be not working after re-adding the new method, is there a point of this re-adding? i mean. i still have to download everything if i don't want to lose time manually picking what to download. or by 1. and 2. you mean something but downloading it agan?

[pspzockerscene]

Depends on you:
If you got a lot of items in your downloadlist which you want to use for duplicate checking, you might want to refresh them.
Either way you can also just create a backup of your settings and downloadlist, wait for the update and then just see what fits you better.

I know that this is not ideal but please keep in mind that JDownloader was never ment to be a precise database-like archiving tool so problems like this can happen over time.

[shinji2009]

what do you mean by refresh. i edited my previous post, please check. if this can be done by not downloading everything again, please make it more clear how to do it

yes i understand about database and i am very grateful for your support

[pspzockerscene]

No you will not need to re-download the items, just delete them from your download list, re-add them and disable them so that the check "is already in downloadlist" in the linkgrabber will work fine for all newer items then.

[shinji2009]

so i need to
1. wait for core update
2. backup download list
3. empty download list
4. stop downloads
5. restore download list
6. disable kemono links in download list

why i have to disable links after restore, don't they backuped as already downloaded?

[pspzockerscene]

Originally Posted by shinji2009

why i have to disable links after restore, don't they backuped as already downloaded?

Becaue you probably do not want to re-download the files - you just want the duplicate-check to work.

Also, you can already create the backup now.

[shinji2009]

not working, all the same - only txt files are highlited red.
download list restored as downloaded, so jd didn't try to download files again.

[pspzockerscene]

I said that what I wrote counts once the next CORE-update is released.

That update and the changes haven't been released yet and I can't provide any ETA for the release of said changes.
Please be patient and wait.

[shinji2009]

oh sorry i was thinking "you can already create the backup now" means everything is done. ok.

[pspzockerscene]

The data on kemono.party / coomer.party is static, right?
Once a post is published, new items are not being added to it, right?
I'm looking for a good way for duplicate checking.
At this moment, I'd do it like this:
Example:

kemono.su/patreon/user/24010882/post/89769521

Unique ID for each file would be something like this:

kemono://patreon/user/24010882/post/89769521/file/1

(The "/1" at the end simply represents the position of the item.)
Alternatively I could also use the "f" parameter inside their URLs since it's mostly given e.g.:

c3.kemono.su/data/1d/c0/CENSORED_BY_PSP_DUE_TO_NSFW.jpg?f=4R4A0159.jpg

-> Unique ID:

kemono://patreon/user/24010882/post/89769521/file/4R4A0159.jpg

I'm waiting for your input before doing any code changes on the new dupe-IDs.
EDIT

This would work fine as long as a post will never have multiple files with the same name according to the parabeter "f" inside the URL.

[shinji2009]

i don't understand what is the question

[pspzockerscene]

Okay again:

You want JDownloade to be able to recognize duplicates of this website in a reliable way.
My question is: What's the best way to create the duplicate detection in this case.

I'm asking you as you are more familiar with this website.

[shinji2009]

best way will be probably more flexible one, which will not fail/will be easy adjustable in the future if they change domain again. or structure of links. or something.

also please remeber that without https://board.jdownloader.org/showpo...18&postcount=8 jd miss a lot of files from kemono sometimes. i don't know if this can affect to dupe checking

[pspzockerscene]

Originally Posted by shinji2009

best way will be probably more flexible one, which will not fail/will be easy adjustable in the future if they change domain again. or structure of links. or something.

I know and that's exactly why I'm asking you too.
Okay so...to be more precise:
Do you think that it is possible that one post contains two or more URLs with the same filename (leading to different files) according to the "f" parameter inside the URLs?

Originally Posted by shinji2009

also please remeber that without https://board.jdownloader.org/showpo...18&postcount=8 jd miss a lot of files from kemono sometimes. i don't know if this can affect to dupe checking

This will not be affected by my upcoming changes since I will not touch the filenames.

[shinji2009]

Originally Posted by pspzockerscene

Do you think that it is possible that one post contains two or more URLs with the same filename (leading to different files) according to the "f" parameter inside the URLs?

like in this post, for example? 1 and 3 attached picture
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Thanks this is one really important information to know before adding any kind of duplicate detection.

Additionally I found out that kemono.party is providing a public API.
I will implement this first before I'm gonna work on any kind of better dupe detection.
The API should make the existing crawler much more reliable and faster.
API reference:

kemono.su/api/schema

[pspzockerscene]

For the next update:
- refactoring
- implemented public API: kemono.su/api/schema
- data URLs contain sha256 filehash -> Make use of them
- additional dupe-check via URL-path as this looks to be unique

This means that JDownloader will be able to recognize duplicated files 100% regardless of the url, filename and origin.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[pspzockerscene]

Updates have been released.

Happy testing and feel free to provide feedback.

[olekingcole001]

Loving this plugin- but I'm noticing that files grabbed from Mega links within a post won't insert the post's properties into the filename. I'm fairly new to JD, so I apologize if there's an obvious fix for this that I'm missing.

Packagizer Filename rule:
<jd:prop:postid> - <jd:prop:date> - <jd:prop:postContentIndex> - <jd:orgfilename>

Example of successful Filename format:
38666680 - 2020-06-27T04_37_21 - 0 - Push_ups_Slow.gif

Example of failed Filename format:
- - - vr_test_long_8K.mp4

It looks like the rule is getting applied, but file is disconnected from the post it was linked through, leaving just the dashes. Is there a workaround or fix for this I could do on my end?


Side question- is there a date property that doesn't include the time stamp? Not worth adding if not, just curious since we can't see plugin-specific properties.

[pspzockerscene]

Originally Posted by olekingcole001

Loving this plugin- but I'm noticing that files grabbed from Mega links within a post won't insert the post's properties into the filename.

This is not a bug. This is how our internal property system works.

Originally Posted by olekingcole001

I'm fairly new to JD, so I apologize if there's an obvious fix for this that I'm missing.

There is no "fix". If you want mega files to have a similar filename structure to the source/"connected" files from kemono.party, you will either need to do this with external tools once the download is complete or you will need to write an EventScripter script to auto-obtain the properties from other kemono.party items and apply similar filenames to the mega.nz items.

EventScripter forum thread:
https://board.jdownloader.org/showthread.php?t=70525
EventScripter help article:
https://support.jdownloader.org/Know...event-scripter

Originally Posted by olekingcole001

Side question- is there a date property that doesn't include the time stamp? Not worth adding if not, just curious since we can't see plugin-specific properties.

No at this moment this is not possible but then again you can freely modify such filenames using EventScripter scripts.

If you want to join the discussions about a more granular custom filename system, be free to join the discussion in the following thread:
https://board.jdownloader.org/showthread.php?t=90744

Thanks for your detailed feedback!

[olekingcole001]

No worries, thanks for all of the information, and thanks for all the work you put into this!

[pspzockerscene]

Thanks for your understanding and your feedback

[Nimboid]

I have been using a customised version of the kemono.party plugin, and it looks like the new features might make this unnecessary.

But I looked at the source anyway, and from KemonoPartyConfig.java it seems there ought to be additional options in the plugin's Config UI which I'm not seeing. These should relate to offering to save post text into .txt files, etc.

I see this (have also attached file):



To confirm this isn't a problem with my local installation, I did a fresh download and install on a clean machine, with the same result.

[pspzockerscene]

Originally Posted by Nimboid

I have been using a customised version of the kemono.party plugin

Which features does that have?

Originally Posted by Nimboid

But I looked at the source anyway, and from KemonoPartyConfig.java it seems there ought to be additional options in the plugin's Config UI which I'm not seeing. These should relate to offering to save post text into .txt files, etc.

The two settings you see are all text related settings which are available.
What exactly are you looking for?
The source contains one more option but that is not related to text downloading.

[Nimboid]

Originally Posted by pspzockerscene

Which features does that have?

It makes the post text accessible to my Event Script. I requested the feature some months ago, but was dismissed.

Originally Posted by pspzockerscene

The two settings you see are all text related settings which are available.
What exactly are you looking for?
The source contains one more option but that is not related to text downloading.

I remain puzzled:
Code says:

@PluginHost(host = "kemono.party", type = Type.CRAWLER)

but the UI refers to "kemono.su". Where does it get the ".su" from?


Code says:

final String    text_CrawlHttpLinksFromPostContent = "Crawl http links in post text?";
final String    text_TextCrawlMode                 = "When to add post text content as .txt file:";

But only the first of those strings appears in the UI.

Code says:

    public static enum TextCrawlMode implements LabelInterface {
        ALWAYS {
            @Override
            public String getLabel() {
                return "Always if text is available";
            }
        },
        ONLY_IF_NO_MEDIA_ITEMS_ARE_FOUND {
            @Override
            public String getLabel() {
                return "Only if no media items are found and text is available";
            }
        },
        NEVER {
            @Override
            public String getLabel() {
                return "Never";
            }
        };
    }

But the wording in the drop-down control does not match; it shows

"Always if available"
"Only if no media items are found and text is available"
"Never"

Note that the first of those does not match the string in the code.

Maybe the build has picked up the wrong version of the KemonoPartyConfig.java source file?

[pspzockerscene]

Originally Posted by Nimboid

It makes the post text accessible to my Event Script. I requested the feature some months ago, but was dismissed.

Ah okay. Interesting.

Originally Posted by Nimboid

I remain puzzled:
Code says:

That was outdated.
I've just changed the domains inside "KemonoPartyConfig" and "KemonoPartyConfigCoomerParty".

Originally Posted by Nimboid

but the UI refers to "kemono.su". Where does it get the ".su" from?

jd.plugins.decrypter.KemonoPartyCrawler -> Function "getPluginDomains".

Originally Posted by Nimboid

Maybe the build has picked up the wrong version of the KemonoPartyConfig.java source file?

No.
The current build is a mix between "current revisions of all plugins" and "~3 months old core revision".
If you need a more detailed explanation, feel free to talk to me via Discord.
Either way - you can use the current SVN code as a base for tests and not(!) the current stable.
EDIT
The way our public builds work can be confusing.

[Nimboid]

So presumably it's just a matter of being patient for ~3 months and these config will appear?

I assumed that these changes were made when the crawler was updated to use the kemono API - which is included in my installation (evident from log files).

[pspzockerscene]

Originally Posted by Nimboid

So presumably it's just a matter of being patient for ~3 months and these config will appear?

I hope not but this is out of my hands.
Also - again:
If you use our current source code, you can _already_now_ have all current changes that were ever made to this plugin and the config.


Actually I still don't understand which part of the config is missing for you,
Screenshot of the current publicly available plugin config:

Spoiler:

Screenshot of the un-released new config:

Spoiler:

As you can see all text-download related config things only got cosmetic changes (the text in the UI was changed, the functionality remains the same).

[Nimboid]

The second screenshot is more like what I expected.

To this user, it looks more than a cosmetic change, since the wording communicates a very different functionality:

Previously, the dropdown was labelled "Select text crawl mode"

Now it will be labelled "When to add post text content as .txt file:"

At the moment, kemono.party.json file contains:

{"textcrawlmode":"ALWAYS","crawlhttplinksfrompostcontent":true}

Would it be possible to post an example of the new kemono.party.json file content?

[pspzockerscene]

Originally Posted by Nimboid

Would it be possible to post an example of the new kemono.party.json file content?

No.

If you want to do any tests with the unreleased code, grab the current source code and use that for playing around.
EDIT

Originally Posted by Nimboid

The second screenshot is more like what I expected.

To this user, it looks more than a cosmetic change, since the wording communicates a very different functionality:

Previously, the dropdown was labelled "Select text crawl mode"

Now it will be labelled "When to add post text content as .txt file:"

Yes but again: The functionality behind that setting is 100% the same.

[Nimboid]

So, in other words, the change of UI string is a correction to bring the UI into line with functionality?

[pspzockerscene]

Name it however you want.
Here is my perspective:
I've looked into this when I was refactoring the plugin and I didn't like the wording so I changed it - there is no other background to it.
Do you have further improvement-suggestions?

[AccomplishedStuff]

Looks like this Plugin has the same issue with Packagizer rule this:

https://board.jdownloader.org/showthread.php?t=90225

I tried the workaround suggested and it seemed to consistently scrape only around 50-55 links.

Example:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Updated KemonoPartyCrawler plugin pattern.
Old:

/[^/]+/user/[^/]+(/post/\\d+)?

New:

/[^/]+/user/([\\w\\-]+)(/post/\\d+)?

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-