JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 09.09.2023, 14:05
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default Civitai.com support would be amazing

Hello, I would like to have support for Civitai gallery images.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

and

**External links are only visible to Support Staff****External links are only visible to Support Staff**

If you scroll down you can see the gallery. Would be highly appreciated if JD2 can download these images.
Reply With Quote
  #2  
Old 11.09.2023, 16:44
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,759
Default

This is the 2nd request for this website so I'm open for adding a plugin.
Reference:
https://board.jdownloader.org/showthread.php?t=93193

Afaik they do have a public API:
github.com/civitai/civitai/wiki/REST-API-Reference

...though I was unable to find an API call for "/images/<numbers>" links.

I've added a first version of the plugin which at this moment can only handle single image links in this format:
Code:
civitai.com/images/1234567...
Regarding those "models" links:
So I can see that every "model" can be available in multiple versions.
The versionID can also be specified in the URL.

Does each version always lead to exactly one file?
I'm asking because they do provide an API endpoint for "models" though the response can contain multiple download-URLs.
Reference:
github.com/civitai/civitai/wiki/REST-API-Reference#get-apiv1models-versionsmodelversionid
Random example:
Code:
civitai.com/api/v1/model-versions/13908
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #3  
Old 11.09.2023, 19:12
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Thanks for the reply. I do not know about the actual URLs for the model or LORAs. I can try to ask people who are knowledgable with stable diffusion if they know eventually. Could you add the plugin for image download first? That would be great! Right now it doesn't work, I found no plugin for Civitai.
Reply With Quote
  #4  
Old 27.02.2024, 19:04
Nobrumski Nobrumski is offline
Modem User
 
Join Date: Feb 2024
Posts: 3
Default

Quote:
Originally Posted by pspzockerscene View Post
This is the 2nd request for this website so I'm open for adding a plugin.
Reference:
**External links are only visible to Support Staff**...

Afaik they do have a public API:
github.com/civitai/civitai/wiki/REST-API-Reference

...though I was unable to find an API call for "/images/<numbers>" links.

I've added a first version of the plugin which at this moment can only handle single image links in this format:
Code:
civitai.com/images/1234567...
Regarding those "models" links:
So I can see that every "model" can be available in multiple versions.
The versionID can also be specified in the URL.

Does each version always lead to exactly one file?
I'm asking because they do provide an API endpoint for "models" though the response can contain multiple download-URLs.
Reference:
github.com/civitai/civitai/wiki/REST-API-Reference#get-apiv1models-versionsmodelversionid
Random example:
Code:
civitai.com/api/v1/model-versions/13908
Not sure if you still need an answer on this since I saw you made a plugin, though since I am brand new to JDownloader I need to figure out how to set it up for PNGs and Jpegs on Civitai so haven't tested it yet. I did find the file in the hoster folder so got that far at least. Working through archive.org is a bit of a pain so if there is a good thread or instructional link I am open to it. I would love to start automating grabbing some PNGs and Jpegs from the feeds.

With regards to models, they do sometimes have other files.

For example this model, **External links are only visible to Support Staff****External links are only visible to Support Staff**, has two attachments. They address this with type.

The first link is the Checkpoint in this case.
**External links are only visible to Support Staff****External links are only visible to Support Staff**

The second link is the Vae file.
**External links are only visible to Support Staff****External links are only visible to Support Staff**

On **External links are only visible to Support Staff****External links are only visible to Support Staff** they also have multiple versions addressed by the modelVersionID. **External links are only visible to Support Staff****External links are only visible to Support Staff**. Sometimes you want all the models and sometimes you don't because the model version can either be incremental updates or versions based off a different Stable Diffusion version such as SD 1.0, SD 1.5, etc. The page info has the only indicator what the model is for so for example **External links are only visible to Support Staff****External links are only visible to Support Staff** is for SD XL Turbo while 290640 is for Pony.
Reply With Quote
  #5  
Old 12.09.2023, 11:27
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,759
Default

Quote:
Originally Posted by accelheim View Post
I do not know about the actual URLs for the model or LORAs.
But you seem to be a user of this website?
To be more precise: One "model version" always leads to exactly one file - is that correct?

Quote:
Originally Posted by accelheim View Post
Could you add the plugin for image download first?
I've already added it yesterday.

Quote:
Originally Posted by accelheim View Post
I found no plugin for Civitai.
Where did you search for the plugin and which links did you try to add?
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #6  
Old 12.09.2023, 16:40
accelheim accelheim is offline
JD Addict
 
Join Date: Jan 2020
Posts: 158
Default

Quote:
Originally Posted by pspzockerscene View Post
But you seem to be a user of this website?
To be more precise: One "model version" always leads to exactly one file - is that correct?


I've already added it yesterday.


Where did you search for the plugin and which links did you try to add?
I'm a user but I do not use models yet, I only want to download the images gallery at the moment.

I searched in settings -> plugins. I tried to paste these links:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**

All of these only output "apple-splash" filenames in JD2. Log:

11.09.23 18.02.15 <--> 12.09.23 15.40.52 jdlog://8867311370661/
Reply With Quote
  #7  
Old 12.09.2023, 16:42
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,759
Default

Quote:
Originally Posted by accelheim View Post
I searched in settings -> plugins.
That is the wrong place to search if a plugin for a website exists.
Explanation:
https://support.jdownloader.org/Know...bsites-plugins

Quote:
Originally Posted by accelheim View Post
All of these only output "apple-splash" filenames in JD2. Log:
Fixed.
I've added a pattern-mistake in the last moment before I've released that plugin.

As explained, at this moment, it can only handle such URLs:
Code:
ht...civitai.com/images/1234...
Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #8  
Old 27.02.2024, 20:50
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,759
Default

Quote:
Originally Posted by Nobrumski View Post
Working through archive.org is a bit of a pain so if there is a good thread or instructional link I am open to it.
No extra instructions needed: Add [archive.org] links, wait until the items appear in the linkgrabber and download them.
Which part exactly is hard to understand?
If anything doesn't work as expected, please provide a problem description, example URLs and a log.

Quote:
Originally Posted by Nobrumski View Post
For example this model, htt...civitai.com/models/2...?modelVersionId=2..., has two attachments. They address this with type.
As explained, such links aren't supported.
I did only add support for links matching the following regular expression:
Code:
/images/[0-9]+.*
Important notice:
Our ticket system & knowledgebase are currently under maintenance.
If this is still the case while you are reading my posts and you can't access help articles linked by our staff, use the Internet archive/wayback machine to be able to view those articles:
archive.org/web/
Enter the URL -> Click on "Browse history" -> Select one of the latest dates available
If the date you selected does not lead you to the support article try the next oldest one.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #9  
Old 27.02.2024, 21:48
Nobrumski Nobrumski is offline
Modem User
 
Join Date: Feb 2024
Posts: 3
Default

Quote:
Originally Posted by pspzockerscene View Post
No extra instructions needed: Add [archive.org] links, wait until the items appear in the linkgrabber and download them.
Which part exactly is hard to understand?
If anything doesn't work as expected, please provide a problem description, example URLs and a **External links are only visible to Support Staff**....
I just followed your link above which didn't explain what to do with the file once I found it. I don't know if I needed to do anything, since it didn't work like I expected it to in linkgrabber as it was. I also saw mention to adding my login cookie details referenced elsewhere, but didn't see a way to do it since you can't see NSFW without the login.

I checked the link in linkgrabber, **External links are only visible to Support Staff****External links are only visible to Support Staff** as an example, and don't see it actually listing any of the images.

I really haven't sat down to play with it too much so figure I am missing something. I thought it would let me paste a link and then it would scrape all the images referenced on that page which isn't happening for me. I just see a list of the splash images and not the actual images.


[QUOTE=pspzockerscene;531079]
As explained, such links aren't supported.
I did only add support for links matching the following regular expression:
Code:
/images/[0-9]+.*
I was just explaining since I didn't see the question answered. I don't care about downloading the resource at **External links are only visible to Support Staff****External links are only visible to Support Staff** for example, but when you scroll down that page it does display images that are linked to that model. If Jdownloader can grab those image links, great and it is one of the things I am interested in. If not it is something that would be nice to have.

There are essentially two pages you frequent on the site.
This leads you to the resource but also the images created by the resource.
**External links are only visible to Support Staff****External links are only visible to Support Staff**
This is an example of the feed of images tied to a specific tag.
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #10  
Old 27.02.2024, 22:23
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,759
Default

Quote:
Originally Posted by Nobrumski View Post
I just followed your link above which didn't explain what to do with the file once I found it.
Nothing. That help article only explains how to check whether or not there is a plugin available for a specific website.
All available plugins are always delivered with JD so there is no need to do anything or "install" them.
The article was made solely for people who expected JD to show a list of supported websites inside the GUI which JD does not.

Quote:
Originally Posted by Nobrumski View Post
I also saw mention to adding my login cookie details referenced elsewhere, but didn't see a way to do it since you can't see NSFW without the login.
I didn't add account support for "civitai.com" (no one has requested it) so you can't add civitai.com accounts to JDownloader.

Quote:
Originally Posted by Nobrumski View Post
I checked the link in linkgrabber, **External links are only visible to Support Staff****External links are only visible to Support Staff** as an example, and don't see it actually listing any of the images.
Since that is public content, I'm going to make this link publicly visible for explanation.
civitai.com/images?tags=161967&view=feed
--> Navigate to that URL in your browser -> Rightclick on any image on that page -> Copy link address e.g.:
civitai.com/images/6843991
--> Those are the kind of links, I added support for so those are the ones, JD can handle.

Quote:
Originally Posted by Nobrumski View Post
I thought it would let me paste a link and then it would scrape all the images referenced on that page which isn't happening for me. I just see a list of the splash images and not the actual images.
No worries. Looks like you assumed that when something is supported via plugin, all website-links are automatically supported.
That is not the case.
Again - JD atm only handles such links automatically: civitai.com/images/1234567
You need to collect them manually from the upper pages where they're coming from e.g. using the following methods:
https://support.jdownloader.org/Know...orted-websites

Quote:
Originally Posted by Nobrumski View Post
I was just explaining since I didn't see the question answered. I don't care about downloading the resource at...
That resource is only visible for logged-in users plus the url-structure is not supported (again: supported links look like this: civitai.com/images/1234567).

Please read the article I linked.
If you need account support for this host, please send me your login credentials via PN.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #11  
Old 28.02.2024, 06:07
Nobrumski Nobrumski is offline
Modem User
 
Join Date: Feb 2024
Posts: 3
Default

Quote:
Originally Posted by pspzockerscene View Post
No worries. Looks like you assumed that when something is supported via plugin, all website-links are automatically supported.
That is not the case.
Again - JD atm only handles such links automatically: civitai.com/images/1234567
You need to collect them manually from the upper pages where they're coming from e.g. using the following methods:
**External links are only visible to Support Staff**...


That resource is only visible for logged-in users plus the url-structure is not supported (again: supported links look like this: civitai.com/images/1234567).
Thanks I completely misunderstood thinking it was something I could feed a link to and it would go so many layers deep to grab images or if I found a tag I liked just grab everything avoiding having to go one by one. My mind jumped to that when I read plugin for Civitai.

They actually do have an API for images I found though it looks somewhat limited. **External links are only visible to Support Staff****External links are only visible to Support Staff**

API keys are free and generated via the account, so could send you one if you were willing to poke around. Accounts don't even require passwords. They just send you a link in email that ties a user to that email address and it keeps you logged in for like 7 days.


I can see how just copying the link would be helpful, but I usually append their file name with the image number. I don't see a way to do that. One of the reasons why is the image sometimes has the workflow to recreate it and sometimes not. If it doesn't most of the info is stored on the image link. Having that number when the workflow isn't embedded, lets me go straight to it since I haven't found a better way to match the image to the site otherwise.
Reply With Quote
  #12  
Old 28.02.2024, 12:24
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,759
Default

Looks like we're still talking past each other.
Once again: JD does handle supported links automatically which you copy.
In this case, links matching the following structure are supported: civitai.com/images/1234567
Any others are not.

If you want to add/find/collect those supported "/images/1234567" links on other subpages of the mentioned website, use the following instructions to do so:
https://support.jdownloader.org/Know...orted-websites

Quote:
Originally Posted by Nobrumski View Post
API keys are free and generated via the account, so could send you one if you were willing to poke around. Accounts don't even require passwords. They just send you a link in email that ties a user to that email address and it keeps you logged in for like 7 days.
So far, all "/images/1234567" links I found worked without the need of an account but as said if you think an account is needed, feel free to send me login-credentials via PN.
EDIT

Added simple crawler support for such links: /models/123456?modelVersionId=123456

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 28.02.2024 at 13:02. Reason: EDIT
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 05:06.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.