JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #101  
Old 26.10.2023, 15:26
shinji2009 shinji2009 is offline
Ultra Loader
 
Join Date: Aug 2021
Posts: 49
Default

Quote:
Originally Posted by pspzockerscene View Post
Do you think that it is possible that one post contains two or more URLs with the same filename (leading to different files) according to the "f" parameter inside the URLs?
like in this post, for example? 1 and 3 attached picture
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #102  
Old 26.10.2023, 15:44
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,730
Default

Thanks this is one really important information to know before adding any kind of duplicate detection.

Additionally I found out that kemono.party is providing a public API.
I will implement this first before I'm gonna work on any kind of better dupe detection.
The API should make the existing crawler much more reliable and faster.
API reference:
Code:
kemono.su/api/schema
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #103  
Old 26.10.2023, 17:56
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,730
Default

For the next update:
- refactoring
- implemented public API: kemono.su/api/schema
- data URLs contain sha256 filehash -> Make use of them
- additional dupe-check via URL-path as this looks to be unique

This means that JDownloader will be able to recognize duplicated files 100% regardless of the url, filename and origin.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #104  
Old 26.10.2023, 18:17
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,730
Default

Updates have been released.

Happy testing and feel free to provide feedback.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 26.10.2023 at 18:26. Reason: Fixed typo
Reply With Quote
  #105  
Old 06.12.2023, 23:13
olekingcole001 olekingcole001 is offline
Modem User
 
Join Date: Dec 2023
Posts: 2
Default

Loving this plugin- but I'm noticing that files grabbed from Mega links within a post won't insert the post's properties into the filename. I'm fairly new to JD, so I apologize if there's an obvious fix for this that I'm missing.

Packagizer Filename rule:
<jd:prop:postid> - <jd:prop:date> - <jd:prop:postContentIndex> - <jd:orgfilename>

Example of successful Filename format:
38666680 - 2020-06-27T04_37_21 - 0 - Push_ups_Slow.gif

Example of failed Filename format:
- - - vr_test_long_8K.mp4

It looks like the rule is getting applied, but file is disconnected from the post it was linked through, leaving just the dashes. Is there a workaround or fix for this I could do on my end?


Side question- is there a date property that doesn't include the time stamp? Not worth adding if not, just curious since we can't see plugin-specific properties.
Reply With Quote
  #106  
Old 07.12.2023, 13:54
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,730
Default

Quote:
Originally Posted by olekingcole001 View Post
Loving this plugin- but I'm noticing that files grabbed from Mega links within a post won't insert the post's properties into the filename.
This is not a bug. This is how our internal property system works.

Quote:
Originally Posted by olekingcole001 View Post
I'm fairly new to JD, so I apologize if there's an obvious fix for this that I'm missing.
There is no "fix". If you want mega files to have a similar filename structure to the source/"connected" files from kemono.party, you will either need to do this with external tools once the download is complete or you will need to write an EventScripter script to auto-obtain the properties from other kemono.party items and apply similar filenames to the mega.nz items.

EventScripter forum thread:
https://board.jdownloader.org/showthread.php?t=70525
EventScripter help article:
https://support.jdownloader.org/Know...event-scripter

Quote:
Originally Posted by olekingcole001 View Post
Side question- is there a date property that doesn't include the time stamp? Not worth adding if not, just curious since we can't see plugin-specific properties.
No at this moment this is not possible but then again you can freely modify such filenames using EventScripter scripts.

If you want to join the discussions about a more granular custom filename system, be free to join the discussion in the following thread:
https://board.jdownloader.org/showthread.php?t=90744

Thanks for your detailed feedback!
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #107  
Old 08.12.2023, 08:24
olekingcole001 olekingcole001 is offline
Modem User
 
Join Date: Dec 2023
Posts: 2
Default

No worries, thanks for all of the information, and thanks for all the work you put into this!
Reply With Quote
  #108  
Old 08.12.2023, 14:57
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,730
Default

Thanks for your understanding and your feedback
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #109  
Old 19.12.2023, 19:26
Nimboid Nimboid is offline
Vacuum Cleaner
 
Join Date: Jan 2023
Location: UK
Posts: 18
Default

I have been using a customised version of the kemono.party plugin, and it looks like the new features might make this unnecessary.

But I looked at the source anyway, and from KemonoPartyConfig.java it seems there ought to be additional options in the plugin's Config UI which I'm not seeing. These should relate to offering to save post text into .txt files, etc.

I see this (have also attached file):



To confirm this isn't a problem with my local installation, I did a fresh download and install on a clean machine, with the same result.
Attached Thumbnails
KemonoConfig.jpg  
Reply With Quote
  #110  
Old 19.12.2023, 20:05
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,730
Default

Quote:
Originally Posted by Nimboid View Post
I have been using a customised version of the kemono.party plugin
Which features does that have?

Quote:
Originally Posted by Nimboid View Post
But I looked at the source anyway, and from KemonoPartyConfig.java it seems there ought to be additional options in the plugin's Config UI which I'm not seeing. These should relate to offering to save post text into .txt files, etc.
The two settings you see are all text related settings which are available.
What exactly are you looking for?
The source contains one more option but that is not related to text downloading.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #111  
Old 19.12.2023, 21:37
Nimboid Nimboid is offline
Vacuum Cleaner
 
Join Date: Jan 2023
Location: UK
Posts: 18
Default

Quote:
Originally Posted by pspzockerscene View Post
Which features does that have?
It makes the post text accessible to my Event Script. I requested the feature some months ago, but was dismissed.

Quote:
Originally Posted by pspzockerscene View Post
The two settings you see are all text related settings which are available.
What exactly are you looking for?
The source contains one more option but that is not related to text downloading.
I remain puzzled:
Code says:
Code:
@PluginHost(host = "kemono.party", type = Type.CRAWLER)
but the UI refers to "kemono.su". Where does it get the ".su" from?


Code says:
Code:
final String    text_CrawlHttpLinksFromPostContent = "Crawl http links in post text?";
final String    text_TextCrawlMode                 = "When to add post text content as .txt file:";
But only the first of those strings appears in the UI.

Code says:
Code:
    public static enum TextCrawlMode implements LabelInterface {
        ALWAYS {
            @Override
            public String getLabel() {
                return "Always if text is available";
            }
        },
        ONLY_IF_NO_MEDIA_ITEMS_ARE_FOUND {
            @Override
            public String getLabel() {
                return "Only if no media items are found and text is available";
            }
        },
        NEVER {
            @Override
            public String getLabel() {
                return "Never";
            }
        };
    }
But the wording in the drop-down control does not match; it shows

"Always if available"
"Only if no media items are found and text is available"
"Never"

Note that the first of those does not match the string in the code.

Maybe the build has picked up the wrong version of the KemonoPartyConfig.java source file?
Reply With Quote
  #112  
Old 19.12.2023, 21:49
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,730
Default

Quote:
Originally Posted by Nimboid View Post
It makes the post text accessible to my Event Script. I requested the feature some months ago, but was dismissed.
Ah okay. Interesting.

Quote:
Originally Posted by Nimboid View Post
I remain puzzled:
Code says:
That was outdated.
I've just changed the domains inside "KemonoPartyConfig" and "KemonoPartyConfigCoomerParty".

Quote:
Originally Posted by Nimboid View Post
but the UI refers to "kemono.su". Where does it get the ".su" from?
jd.plugins.decrypter.KemonoPartyCrawler -> Function "getPluginDomains".

Quote:
Originally Posted by Nimboid View Post
Maybe the build has picked up the wrong version of the KemonoPartyConfig.java source file?
No.
The current build is a mix between "current revisions of all plugins" and "~3 months old core revision".
If you need a more detailed explanation, feel free to talk to me via Discord.
Either way - you can use the current SVN code as a base for tests and not(!) the current stable.
EDIT
The way our public builds work can be confusing.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 19.12.2023 at 21:49. Reason: EDIT
Reply With Quote
  #113  
Old 19.12.2023, 22:16
Nimboid Nimboid is offline
Vacuum Cleaner
 
Join Date: Jan 2023
Location: UK
Posts: 18
Default

So presumably it's just a matter of being patient for ~3 months and these config will appear?

I assumed that these changes were made when the crawler was updated to use the kemono API - which is included in my installation (evident from log files).
Reply With Quote
  #114  
Old 19.12.2023, 22:22
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,730
Default

Quote:
Originally Posted by Nimboid View Post
So presumably it's just a matter of being patient for ~3 months and these config will appear?
I hope not but this is out of my hands.
Also - again:
If you use our current source code, you can _already_now_ have all current changes that were ever made to this plugin and the config.


Actually I still don't understand which part of the config is missing for you,
Screenshot of the current publicly available plugin config:
Spoiler:


Screenshot of the un-released new config:
Spoiler:


As you can see all text-download related config things only got cosmetic changes (the text in the UI was changed, the functionality remains the same).
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 19.12.2023 at 22:25. Reason: Fixed typo
Reply With Quote
  #115  
Old 19.12.2023, 22:56
Nimboid Nimboid is offline
Vacuum Cleaner
 
Join Date: Jan 2023
Location: UK
Posts: 18
Default

The second screenshot is more like what I expected.

To this user, it looks more than a cosmetic change, since the wording communicates a very different functionality:

Previously, the dropdown was labelled "Select text crawl mode"

Now it will be labelled "When to add post text content as .txt file:"

At the moment, kemono.party.json file contains:
Code:
{"textcrawlmode":"ALWAYS","crawlhttplinksfrompostcontent":true}

Would it be possible to post an example of the new kemono.party.json file content?
Reply With Quote
  #116  
Old 19.12.2023, 22:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,730
Default

Quote:
Originally Posted by Nimboid View Post
Would it be possible to post an example of the new kemono.party.json file content?
No.

If you want to do any tests with the unreleased code, grab the current source code and use that for playing around.
EDIT

Quote:
Originally Posted by Nimboid View Post
The second screenshot is more like what I expected.

To this user, it looks more than a cosmetic change, since the wording communicates a very different functionality:

Previously, the dropdown was labelled "Select text crawl mode"

Now it will be labelled "When to add post text content as .txt file:"
Yes but again: The functionality behind that setting is 100% the same.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 19.12.2023 at 22:58. Reason: EDIT
Reply With Quote
  #117  
Old 20.12.2023, 00:22
Nimboid Nimboid is offline
Vacuum Cleaner
 
Join Date: Jan 2023
Location: UK
Posts: 18
Default

So, in other words, the change of UI string is a correction to bring the UI into line with functionality?
Reply With Quote
  #118  
Old 20.12.2023, 11:10
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,730
Default

Name it however you want.
Here is my perspective:
I've looked into this when I was refactoring the plugin and I didn't like the wording so I changed it - there is no other background to it.
Do you have further improvement-suggestions?
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #119  
Old 25.01.2024, 12:44
AccomplishedStuff AccomplishedStuff is offline
Junior Loader
 
Join Date: May 2022
Posts: 13
Default

Looks like this Plugin has the same issue with Packagizer rule this:

https://board.jdownloader.org/showthread.php?t=90225

I tried the workaround suggested and it seemed to consistently scrape only around 50-55 links.

Example:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #120  
Old 25.01.2024, 13:36
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,730
Default

Updated KemonoPartyCrawler plugin pattern.
Old:
Code:
/[^/]+/user/[^/]+(/post/\\d+)?
New:
Code:
/[^/]+/user/([\\w\\-]+)(/post/\\d+)?
Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 25.01.2024 at 13:39. Reason: Fixed typo
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 20:09.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.