JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 22.10.2022, 06:33
hishtnik011 hishtnik011 is offline
JD Alpha
 
Join Date: Aug 2021
Posts: 20
Default Add links from Tampermonkey script?

Some gallery sites are tricky and I have my own Tampermonkey scripts that essentially go through all the pages, make a bunch of requests and fetch all the full-size image links.

When finished the scripts add a textarea element to the page, containing all the links and a text input element containing whatever title I have chosen to give that gallery.

I then need to copy the links to Jdownloader and set the package name as the title.

-------

Can this be automated? In other words, instead of copying the links and title to jdownloader, once my script is finished, it contacts Jdownloader with that information automatically.

-------

Note that the package name is extremely important here. In jdownloader I have a custom renaming rule, which names all the images based on the package name, followed by a number. This is obviously so that everything is organized and the original order of the gallery is preserved.
Reply With Quote
  #2  
Old 22.10.2022, 16:55
tarkett tarkett is offline
JD VIP
 
Join Date: Apr 2009
Location: Deutschland
Posts: 367
Default

The feature FolderWatch can be a possible solution for your request. Your script create crawljob files with Links, Packagename, ...
JD will look into the defined Folder and import the crawljobs automatic.



Info: https://support.jdownloader.org/Know...ch-basic-usage
Reply With Quote
  #3  
Old 23.10.2022, 14:42
hishtnik011 hishtnik011 is offline
JD Alpha
 
Join Date: Aug 2021
Posts: 20
Default

Quote:
Originally Posted by tarkett View Post
The feature FolderWatch can be a possible solution for your request. Your script create crawljob files with Links, Packagename, ...
JD will look into the defined Folder and import the crawljobs automatic.



Info: **External links are only visible to Support Staff**...
Ok thank you. Will look into that.
Reply With Quote
  #4  
Old 23.10.2022, 18:27
hishtnik011 hishtnik011 is offline
JD Alpha
 
Join Date: Aug 2021
Posts: 20
Default

IGNORE POSTS BELOW ABOUT THE PROBLEMS I HAD. IT WAS RESOLVED (THE SCRIPTS BELOW WERE FIXED)

Folder Watch works great. Here is the function I have btw for saving the .crawljob file if anyone finds this and needs it for some reason:

Code:
function save_as_crawljob_file(linksArr, galleryName)
{
    let obj =
        [
            {
                //autoStart: "TRUE",
                text: linksArr.join("<br>"),
                packageName: galleryName
            }
        ];

    let base64Encoded = btoa(JSON.stringify(obj));

    GM_download({
        url:`data:text/plain;base64,${base64Encoded}`,
        name:`${galleryName}.crawljob`,
        saveAs:false,
        onerror:()=>{alert(`FAILED SAVING FILE`)},
        onload:()=>{alert(`FILE SAVED with ${linksArr.length} links`)}
    });
}
your script in tampermonkey also needs to have the "GM_download" permission and in Tampermonkey settings > Advanced > Whitelisted File Extensions, you need to add .crawljob to the end.

----------------------------------------------------------------------------

Also if you want to use Click n Load instead, here is the function for that as well:

Code:
function contact_jdownloader(linksArr, galleryName)
{
    // used so that the form doesn't load a new page or open a new window
    try {document.querySelector("#jdFrame").remove()} catch(e) {}
    let frameHtml = `<iframe id="jdFrame" name="jdownloader-frame" style="width:100%;" hidden></iframe>`;
    document.querySelector("body").appendChild(html_to_node(frameHtml));

    try {document.querySelector("#jdForm").remove()} catch(e) {}
    // since jdownloader site doesn't allow links, set the form action below to 'http{colon,slash.slah}127.0.0.1:9666/flash/add'
    let formHtml = `<form id='jdForm' action='' method='POST' target='jdownloader-frame'><input type="hidden" name="source" value="tampermonkey"/><input type="hidden" name="package" value="${galleryName}"/><input id="jdUrlsInput" type="hidden" name="urls" value="${linksArr.join("<br>")}"/></form>`;
    document.querySelector("body").appendChild(html_to_node(formHtml));

    let formElem = document.querySelector("#jdForm");
    formElem.submit();
}

function html_to_node(code)
{
    let tempWrapper = document.createElement("div");
    tempWrapper.innerHTML = code;
    if (tempWrapper.childElementCount == 1) tempWrapper = tempWrapper.firstChild;
    return tempWrapper;
}

Last edited by hishtnik011; 02.11.2022 at 19:25.
Reply With Quote
  #5  
Old 24.10.2022, 00:24
tarkett tarkett is offline
JD VIP
 
Join Date: Apr 2009
Location: Deutschland
Posts: 367
Default

Perfect, that it works for you
Reply With Quote
  #6  
Old 24.10.2022, 18:55
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,143
Default

Thanks for helping out @tarkett.

@hishtnik011
What you could also use for this:
1. Click and load API.
2. Myjd API.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #7  
Old 25.10.2022, 03:31
hishtnik011 hishtnik011 is offline
JD Alpha
 
Join Date: Aug 2021
Posts: 20
Default

Quote:
Originally Posted by pspzockerscene View Post
Thanks for helping out @tarkett.

@hishtnik011
What you could also use for this:
1. **External links are only visible to Support Staff**... API.
2. **External links are only visible to Support Staff**....
Ok thank you! Will look at those too when I get the time.
Reply With Quote
  #8  
Old 26.10.2022, 11:38
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,343
Default

@hishtnik011: In case you've got questions about api usage, just contact us via support@jdownloader.org
I can also provide a basic pure js MyJDownloader api client for further testing/development, that way your script would also work from remote
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 30.10.2022, 17:42
hishtnik011 hishtnik011 is offline
JD Alpha
 
Join Date: Aug 2021
Posts: 20
Default

Quote:
Originally Posted by Jiaz View Post
@hishtnik011: In case you've got questions about api usage, just contact us via support@jdownloader.org
I can also provide a basic pure js MyJDownloader api client for further testing/development, that way your script would also work from remote
I do have one question/problem with both the Folder Watch plugin and Click n Load.

Despite of having the "LinkCollector: Do Link Check" option disabled in Jdownloader Advanced Settings, both Folder Watch and Click n Load seem to crawl the links I provide. This can take a long time if I feed it hundreds/thousands of links and especially with some hosts.

Can that be fixed?
Reply With Quote
  #10  
Old 31.10.2022, 14:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,143
Default

Please add more details e.g. which links are you adding.

The "do link check" option is for single links that go into host plugins.
Example where this will have an effect: uploaded.net, rapidgator.net and most of all other websites for which you're typically adding links that lead to single files.
Examples where this will have no effect: youtube, vimeo and other websites where links first get processed by a crawler as they will/can contain multiple results

--> That setting does not have any effect for links which need to be processed by a crawler plugin first.

If "fast linkcheck" disabled that too, JD just wouldn't do anything at all anymore when adding such links.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #11  
Old 31.10.2022, 16:37
hishtnik011 hishtnik011 is offline
JD Alpha
 
Join Date: Aug 2021
Posts: 20
Default

Quote:
Originally Posted by pspzockerscene View Post
Please add more details e.g. which links are you adding.

The "do link check" option is for single links that go into host plugins.
Example where this will have an effect: uploaded.net, rapidgator.net and most of all other websites for which you're typically adding links that lead to single files.
Examples where this will have no effect: youtube, vimeo and other websites where links first get processed by a crawler as they will/can contain multiple results

--> That setting does not have any effect for links which need to be processed by a crawler plugin first.

If "fast linkcheck" disabled that too, JD just wouldn't do anything at all anymore when adding such links.
I'm adding mainly image links. The links I'm adding are either links to full size images that I have fetched myself (the ones that Jdownloader can't find on its own) or the hrefs of the images from a gallery / forum page etc. (the ones Jdownloader can find the full size image on its own).

I have custom scripts for all the sites I download from that get me the image links, combined with some tricks to trigger my custom renaming rule in packagizer (#custnum= added at the end of each link). This is why when I download images I do not need the link check thing.

Anyway, by having the link check option disabled, if I were to add links manually through the Jdownloader interface, it does not check their online status, it just adds them and that's it.

For example, I click on "add links" (or Ctrl+L), enter a package name and provide however many links I want. It accepts them with a question mark beside each link and I can begin downloading right away.

Meanwhile with Folder Watch (.crawljob files) and Click n Load, in both cases it analyzes (crawls) the links before I can begin downloading them (that spinning thing at the bottom right corner).

It appears to me that both Click n Load and Folder Watch do not respect/know that I have disabled link check in advanced settings. I looked through the APIs of both of them and they don't seem to have anything related to this.
Reply With Quote
  #12  
Old 31.10.2022, 17:04
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,143
Default

Quote:
Originally Posted by hishtnik011 View Post
For example, I click on "add links" (or Ctrl+L), enter a package name and provide however many links I want. It accepts them with a question mark beside each link and I can begin downloading right away.

Meanwhile with Folder Watch (.crawljob files) and Click n Load, in both cases it analyzes (crawls) the links before I can begin downloading them (that spinning thing at the bottom right corner).

It appears to me that both Click n Load and Folder Watch do not respect/know that I have disabled link check in advanced settings. I looked through the APIs of both of them and they don't seem to have anything related to this.
Thanks for your explanation.
That would indeed be a bug.
We'll check this once we find the time.

Do you have any CNL/example URLs which can be used to reproduce this?
Otherwise we'll try with random example URLs.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #13  
Old 31.10.2022, 17:44
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,143
Default

I was unable to confirm your theory.
Here is the LinkCrawler rule I've used for checking:
Code:
extractPasswords=["Password1","Password2"]
downloadPassword=123456Test
enabled=TRUE
text=http://cdn8.appwork.org/speed.zip
packageName=MyPackageName
filename=NewFilename.zip
comment=SuperUsefulComment
autoConfirm=TRUE
autoStart=TRUE
extractAfterDownload=FALSE
forcedStart=FALSE
downloadFolder=C:\Users\test\Downloads
overwritePackagizerEnabled=false
URL used:
Code:
cdn8.appwork.org/speed.zip
When linkcheck is disabled, this one gets added with a question mark else it gets added as offline which is the expected behavior.
My theory is that you're adding links which JD needs to deep crawl e.g. direct-links without file-extension.
Without such testlinks I won't be able to provide help with this.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #14  
Old 01.11.2022, 12:04
hishtnik011 hishtnik011 is offline
JD Alpha
 
Join Date: Aug 2021
Posts: 20
Default

Honestly though I'm not really that concerned about this since I can set up my scripts so that i add the links from the few hosts that take a long time to process manually, and everything else is automatic.

--------------

Although, I think I'm done using Click n Load and Folder Watch due to another issue I encountered. It appears to be caused by my custom renaming rule and it happens regardless of whether I have link check on or off.

My custom renaming packagizer rule is as follows:
if
Sourceurl(s) contains *#custnum=*

then set Filename
<jd:packagename> - <jd:source:2>.<jd:orgfiletype>


Since it's hard to explain, I will provide you with a sample .crawljob file. So save it and load it in Jdownloader (you will need folder watch installed and enabled to be able to load the JSON-format)
WARNING: links are NSFW
Code:
[{"text":"**External links are only visible to Support Staff**,"packageName":"example"}]

What happens after I add this file is I get this:
example - 22_.jpg
example - 22_.jpg
example - 22_.jpg
....

AKA the same thing repeating over and over again. It also adds that weird underscore before .jpg which is not supposed to be there.

If I disable my custom renaming rule it works just fine, but the links are no longer named the way I want them to.

--------------

Obviously, when I add this package manually through the interface (Ctrl+L, copy paste) (links separated by new line), everything, including my custom renaming rule works:
package name: example
Code:
**External links are only visible to Support Staff**

Last edited by hishtnik011; 02.11.2022 at 09:42.
Reply With Quote
  #15  
Old 02.11.2022, 14:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,143
Default

Jiaz might be able to add a better description but that's all I can provide you with now:

Just in case you're already auto-generating those rules via a script:
If wou were to create one rule per URL, you'd be able to work around this problem.

The background roughly is this:
The complete content inside that "text" field of your crawljob is available for every single item as "source" this is why you will get the same "custnum" result for all items with that Packagizer rule even though each link is different.

Jiaz will correct me / edit this post if any of that information is wrong.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #16  
Old 02.11.2022, 16:48
hishtnik011 hishtnik011 is offline
JD Alpha
 
Join Date: Aug 2021
Posts: 20
Default

Quote:
Originally Posted by pspzockerscene View Post
Jiaz might be able to add a better description but that's all I can provide you with now:

Just in case you're already auto-generating those rules via a script:
If wou were to create one rule per URL, you'd be able to work around this problem.

The background roughly is this:
The complete content inside that "text" field of your crawljob is available for every single item as "source" this is why you will get the same "custnum" result for all items with that Packagizer rule even though each link is different.

Jiaz will correct me / edit this post if any of that information is wrong.
Yeah that makes sense. I did it with separate requests for each image, using Click n Load, and it works, but I'm not sure about the certainty of it. I had to set a delay between requests (200ms), otherwise some were ignored (the lower the delay). This seems to work, but what if I have 2 or 3 galleries and separate windows side by side and I start the process for each simultaneously. Gonna have to test that when I have the time.

Last edited by hishtnik011; 02.11.2022 at 19:09.
Reply With Quote
  #17  
Old 02.11.2022, 16:53
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,143
Default

As said Jiaz might be able to provide more/better information than me but for now that's all I can tell you about this topic^^
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #18  
Old 02.11.2022, 19:20
hishtnik011 hishtnik011 is offline
JD Alpha
 
Join Date: Aug 2021
Posts: 20
Default

Quote:
Originally Posted by pspzockerscene View Post
As said Jiaz might be able to provide more/better information than me but for now that's all I can tell you about this topic^^
I had someone send me a private message suggesting that I should separate links using "<br>" instead of "\\". That fixed all the issues I was having. I updated the functions I posted above and you can close this if you want.

Thanks for your help!

Last edited by hishtnik011; 03.11.2022 at 05:56.
Reply With Quote
  #19  
Old 03.11.2022, 13:03
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,143
Default

Thanks for your feedback.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 13:50.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.