JDownloader Community - Appwork GmbH
 

Closed Thread
 
Thread Tools Display Modes
  #1  
Old 11.04.2023, 04:18
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default [LinkCrawler rules] suggestion/request : add account for this page civitai.com

hi , may i ask for adding account on this site
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Thanks alot
  #2  
Old 11.04.2023, 14:50
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,047
Default

Hi,
the chances of this happening are super low.

Please read:
https://support.jdownloader.org/Know...orted-websites
and:
https://support.jdownloader.org/Know...aming-websites
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
  #3  
Old 11.04.2023, 17:28
notice notice is offline
JD Supporter
 
Join Date: Mar 2023
Posts: 505
Default

@navmesh: Should be possible with help of Linkcrawler Rules, see https://support.jdownloader.org/Know...kcrawler-rules
One DEEPDECRYPT rule that parses the /models/number/... link and finds the embedded download from /api/download/models/number. And another DIRECTHTTP rule that provides support for the embedded download links
Please try by yourself first and in case of questions, just ask here
  #4  
Old 11.04.2023, 19:49
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default

Quote:
Originally Posted by notice View Post
@navmesh: Should be possible with help of Linkcrawler Rules, see **External links are only visible to Support Staff**...
One DEEPDECRYPT rule that parses the /models/number/... link and finds the embedded download from /api/download/models/number. And another DIRECTHTTP rule that provides support for the embedded download links
Please try by yourself first and in case of questions, just ask here
Hi ,thank you for your help ,but that site require login and settings of account to show NFSW content, for example this link, if you dont have account or your account not set to "show NFSW" ,this content is hidden
**External links are only visible to Support Staff****External links are only visible to Support Staff**
if i 'm not wrong, this mean to access those contents i have to add my account to jdownloader

Last edited by navmesh; 11.04.2023 at 20:02.
  #5  
Old 11.04.2023, 20:05
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,047
Default

Quote:
Originally Posted by navmesh View Post
if i 'm not wrong, this mean to access those contents i have to add my account to jdownloader
No you can also provide your login cookies to JDownloader via the mentioned LinkCrawler rules.
It's all explained here:
https://support.jdownloader.org/Know...awler-rules/22
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
  #6  
Old 11.04.2023, 21:07
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default

Quote:
Originally Posted by pspzockerscene View Post
No you can also provide your login cookies to JDownloader via the mentioned LinkCrawler rules.
It's all explained here:
**External links are only visible to Support Staff**...
Thank you for the help, in the cookie section , is it possible to use cookie export in json format like this instruction ? (by using editthiscookie extension in chrome)

https://support.jdownloader.org/Know...your%20cookies.

Last edited by navmesh; 11.04.2023 at 21:45.
  #7  
Old 12.04.2023, 11:56
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,047
Default

No you need to use our proprietary key/value cookie format.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
  #8  
Old 12.04.2023, 12:04
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default

Quote:
Originally Posted by pspzockerscene View Post
No you need to use our proprietary key/value cookie format.
Thanks , can you please tell me how to get that site's cookie in proprietary key/value cookie format like you said?
  #9  
Old 12.04.2023, 12:06
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,047
Default

Well...get the keys and values of your cookies and put them in the format as displayed in the LinkCrawler article as an example:
Quoting the article:
Code:
"cookies": [
      [
        "key1",
        "value1"
      ],[
        "key2",
        "value2"
      ]
    ]
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 12.04.2023 at 12:07. Reason: Enhanced example json
  #10  
Old 13.04.2023, 10:06
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default

thank you very much , i will try it
  #11  
Old 13.04.2023, 10:15
notice notice is offline
JD Supporter
 
Join Date: Mar 2023
Posts: 505
Default

@navmesh: In case you need further help, just ask
  #12  
Old 13.04.2023, 13:25
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,047
Default

@navmesh
I've updated some examples in our LinkCrawler rules help articles and also included the new "cookies" json array example in them.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
  #13  
Old 15.04.2023, 17:41
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default

Thank you very much ,
im using "edit this cookie" extension to retrieve the key value of cookie
https://chrome.google.com/webstore/d...omclgfbg?hl=en
when i export cookie as json, i have the "value" field
if i 'm not wrong , we just copy the "value" content in the json box to "value" field of Jdownloader
but i dont know where i can get the key field
can you please help
Thanks
  #14  
Old 17.04.2023, 13:53
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,047
Default

Mapping of fields from EditThisCookie to JDownloader LinkCrawler rule "cookies" syntax:
name -> key
value -> value
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
  #15  
Old 17.04.2023, 19:42
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default

i see, thank you very much for the help
  #16  
Old 18.04.2023, 01:48
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,047
Default

No worries.
We got an internal ticket to add support for cookie import from editthiscookie and other sources for LinkCrawler Rules but at this moment that's not possible.
Here is that ticket:
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
  #17  
Old 23.04.2023, 16:32
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default

Hi again, i try entering link crawler rule with my cookie from editthiscookie
i check many times but i haven't found where my mistake is,
i think my cookie format is already in this format ,so i dont know what cause this error
HTML Code:
"cookies": [
      [
        "key1",
        "value1"
      ],[
        "key2",
        "value2"
      ]
    ]
it say ]' is not valid java.utils list , but i thought i enter the cookie into correct format list
because my link crawler rule contains cookie, may i send the deep crawler rule through your inbox ?
Thanks alot

Quote:
Originally Posted by pspzockerscene View Post
Well...get the keys and values of your cookies and put them in the format as displayed in the **External links are only visible to Support Staff**... as an example:
Quoting the article:
Code:
"cookies": [
      [
        "key1",
        "value1"
      ],[
        "key2",
        "value2"
      ]
    ]
  #18  
Old 23.04.2023, 19:44
notice notice is offline
JD Supporter
 
Join Date: Mar 2023
Posts: 505
Default

Quote:
Originally Posted by navmesh View Post
may i send the deep crawler rule through your inbox ?
Send it to support@jdownloader.org and we'll take a look at it and fix it/tell you where the mistake happens. most likely just some json formatting issue, but we'll see
You may contact us earlier if you're stuck! no need to spend hours/days/weeks! just ask for help/hint
  #19  
Old 23.04.2023, 19:55
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default

Quote:
Originally Posted by notice View Post
Send it to support@jdownloader.org and we'll take a look at it and fix it/tell you where the mistake happens. most likely just some json formatting issue, but we'll see
You may contact us earlier if you're stuck! no need to spend hours/days/weeks! just ask for help/hint
Thank you very much, i sent it to your mail
I have no idea why it cause list error, i check json format on jsoneditoronline and it still correct
Thanks alot
  #20  
Old 23.04.2023, 20:02
notice notice is offline
JD Supporter
 
Join Date: Mar 2023
Posts: 505
Default

@navmesh: I was able to copy/paste the rule from your text file as it is without any problem. Opened the txt file, select and copy and paste to JDownloader.
You paste it into Settings->Advanced Settings->LinkCrawler.linkcrawlerrules ?
JDownloader with GUI or MyJDownloader? Pay attention that othing else is copied to clipboard, maybe something invisible at the end, eg tabs or something like that
  #21  
Old 23.04.2023, 20:12
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default

Quote:
Originally Posted by notice View Post
@navmesh: I was able to copy/paste the rule from your text file as it is without any problem. Opened the txt file, select and copy and paste to JDownloader.
You paste it into Settings->Advanced Settings->LinkCrawler.linkcrawlerrules ?
JDownloader with GUI or MyJDownloader? Pay attention that othing else is copied to clipboard, maybe something invisible at the end, eg tabs or something like that
sorry i had sent the file without saving, i've just sent the correct one , can you please check again
Thanks

Last edited by navmesh; 23.04.2023 at 20:21.
  #22  
Old 23.04.2023, 20:22
notice notice is offline
JD Supporter
 
Join Date: Mar 2023
Posts: 505
Default

@navmesh: I recommend to use a json formatter/validator, for example
jsonformatter.org that will show you that the json is wrong formatted and contains errors and where the error might be. I've sent you the corrected ones by mail
  #23  
Old 23.04.2023, 20:24
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default

Quote:
Originally Posted by notice View Post
@navmesh: I recommend to use a json formatter/validator, for example
jsonformatter.org that will show you that the json is wrong formatted and contains errors and where the error might be. I've sent you the corrected ones by mail
i see it now, thank you very much for the help
  #24  
Old 23.04.2023, 20:28
notice notice is offline
JD Supporter
 
Join Date: Mar 2023
Posts: 505
Default

Quote:
Originally Posted by navmesh View Post
i see it now, thank you very much for the help
You're welcome
  #25  
Old 24.04.2023, 16:11
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default

Quote:
Originally Posted by notice View Post
@navmesh: Should be possible with help of Linkcrawler Rules, see **External links are only visible to Support Staff**...
One DEEPDECRYPT rule that parses the /models/number/... link and finds the embedded download from /api/download/models/number. And another DIRECTHTTP rule that provides support for the embedded download links
Please try by yourself first and in case of questions, just ask here
Hi again , actually i want to download the image that display on that site, not the model in button "download"
i sent the mail that describe more clearly and contains link grabber rule and cookie , which not works, can you please help
Thanks
  #26  
Old 24.04.2023, 16:34
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,047
Default

Here is an example rule which works for "civitai.com/images/...?..." and will grab single images:
Code:
[
  {
    "enabled": true,
    "logging": false,
    "maxDecryptDepth": 1,
    "name": "example rule for civitai.com single images",
    "pattern": "**External links are only visible to Support Staff**,
    "rule": "DEEPDECRYPT",
    "packageNamePattern": null,
    "passwordPattern": null,
    "deepPattern": "src=\"(**External links are only visible to Support Staff**]+)\"[^>]*style=\"max-height:100%;max-width:100%\" "
  }
]
Rule as plaintext:
pastebin.com/raw/qHHiLfGt

This rule will only work for public content - you will need to add your login cookies to it to be able to download content which can only be viewed when logged in!
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 24.04.2023 at 18:03. Reason: Added info that this rule wil only work for public content.
  #27  
Old 24.04.2023, 16:59
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default

Quote:
Originally Posted by pspzockerscene View Post
Here is an example rule which works for "civitai.com/images/...?..." and will grab single images:
Code:
[
  {
    "enabled": true,
    "logging": false,
    "maxDecryptDepth": 1,
    "name": "example rule for civitai.com single images",
    "pattern": "**External links are only visible to Support Staff**,
    "rule": "DEEPDECRYPT",
    "packageNamePattern": null,
    "passwordPattern": null,
    "deepPattern": "src=\"(**External links are only visible to Support Staff**]+)\" style=\"max-height:100%;max-width:100%\" "
  }
]
Rule as plaintext:
pastebin.com/raw/qHHiLfGt

This rule will only work for public content - you will need to add your login cookies to it to be able to download content which can only be viewed when logged in!
Thank you very much for the help,
I tried your rule with the link (which is public link) with your rule ,include and not include my cookie
**External links are only visible to Support Staff****External links are only visible to Support Staff**
i paste the link into "link grabber" tab but it cannot grab anything
i also try to disable my link grabber filter but its still not work
  #28  
Old 24.04.2023, 18:04
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,047
Default

You're right.
I've updated my previous post (also the pastebin text).
Only a minor change of the "deepPattern" was needed to make this work for other/all items.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
  #29  
Old 24.04.2023, 18:19
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default

Quote:
Originally Posted by pspzockerscene View Post
You're right.
I've updated my previous post (also the pastebin text).
Only a minor change of the "deepPattern" was needed to make this work for other/all items.
Thank you , but i dont know now why i cannot visit the pastebin.com and also the link you gave
it says
pastebin.com unexpectedly closed the connection.
i tried another computer (on other modem) and it have the same problem
can you please upload to another host ?
Thanks alot
  #30  
Old 24.04.2023, 18:24
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,047
Default

Maybe your ISP is dns-blocking pastebin.com?
Or a firewall/AV application is blocking?

Here it is as .txt file:
workupload.com/file/hYWNwfuadxJ
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
  #31  
Old 24.04.2023, 18:40
navmesh navmesh is offline
Linkgrabbing Monster
 
Join Date: Nov 2016
Posts: 82
Default

Thank you very much ,it works
  #32  
Old 24.04.2023, 18:48
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,047
Default

Thanks for your feedback.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
  #33  
Old 11.09.2023, 15:47
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,047
Default

Added simple plugin for civitai.com which at this moment is able to download single images.
More information:
https://board.jdownloader.org/showthread.php?t=94252

Thread closed.
Please continue in the other thread I linked.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 11.09.2023 at 15:48. Reason: Fixed typo
Closed Thread

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 12:11.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.