JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #41  
Old 09.02.2020, 13:52
Fetter Biff Fetter Biff is offline
JD Legend
 
Join Date: Oct 2014
Posts: 945
Default

Many thanks!

Quote:
Once you comment out the message, JD will only skip it and not display any message. Skipping is required to prevent downloading the file again.
So deactivating (instead of skipping) the links would not work, I guess.

Quote:
To avoid dupe checking everytime you start JD, remove the "#duplicatelink" or "#duplicatefile" from the comment (if you are plannig to redownload the file)
So it is not possible to keep the "#duplicatefile" in the comments when JD closes / restarts, I assume.

How could one remove the "#duplicatelink" from all comments / for all duplicates with a single action / click or so instead of removing them maunally one by one?

Could the links be checked for duplicates in the link grabber already?

So if there is no connection to the server the files to be downloaded are stored on or there is a wainting time for downloads it is not possible to check for duplicates? So one had to wait for the waiting time to finish to let the script check for the dupllicats?

Although links (from Mega) just were downloaded and added to the history they were downloaded a second time after having added them to JD again. Why is this? In a next try they are skipped.

In this list
Code:
Code:
// Skip download link, if filename exists in the user-spcified list
// To download the file (prevent skipping), remove "#duplicatefile" from comment
// Trigger required: A Download Started

var fileNamesList = "I:\\jD-Downloads\\jD-Dummies.txt"; // < Set path to text file which contain the file names. Use "\\" or "/" as path separators.
var dupeFilecheck = link.getProperty("dupeFileCheck");
var linkName = link.getName();
var comment = link.getComment() || "";
var skipLink = function() {
there are trailing spaces in each line (after the name of the archivies already being download). Does that - the spaces - avoid the already downloaded archives to be recognized?

Last edited by Fetter Biff; 09.02.2020 at 13:56.
Reply With Quote
  #42  
Old 10.02.2020, 07:06
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 901
Default

Quote:
So deactivating (instead of skipping) the links would not work, I guess.
To disable the link instead of skipping it, replace link.setSkipped(true); with link.setEnabled(false); in both the scripts.

Quote:
So it is not possible to keep the "#duplicatefile" in the comments when JD closes / restarts, I assume.
The script will be present in the comment field till it is deleted by you. The links will however be unskipped automatically when the download is started, so the script will check the comment and skip it again.

Quote:
How could one remove the "#duplicatelink" from all comments / for all duplicates with a single action / click or so instead of removing them maunally one by one?
Code:
// Remove #duplicatelink and #dupliatefile from comment field for selected links
// Trigger: Downloadlist Contextmenu Button Pressed

if (name = "Clear dupe comment") {
    dlSelection.getLinks().forEach(function(link) {
        var myComments = ["#duplicatelink", "#duplicatefile"];
        myComments.forEach(function(myComment) {
            var comment = link.getComment() || "";
            if (comment.indexOf(myComment) > -1) link.setComment(comment.replace("#duplicatelink", "").replace("#duplicatefile", ""));
        })
    })
}

Quote:
Could the links be checked for duplicates in the link grabber already?

So if there is no connection to the server the files to be downloaded are stored on or there is a wainting time for downloads it is not possible to check for duplicates? So one had to wait for the waiting time to finish to let the script check for the dupllicats?
Script follows the same sequence as JD for dupe check. Dupe link check is done only when adding links to linkgrabber. Dupe file check is done only when the download starts.

The script does not wait for a server respons. It just checks the link or file name against the text file.

Quote:
Although links (from Mega) just were downloaded and added to the history they were downloaded a second time after having added them to JD again. Why is this? In a next try they are skipped.
The dupe check is performed only when adding the link to linkgrabber. I am assuming the it was present in the download list before it could be checked. To ensure all existing pending links in download list are dupe checked, you will have to remove and add them back again.

Quote:
In this list ... there are trailing spaces in each line (after the name of the archivies already being download). Does that - the spaces - avoid the already downloaded archives to be recognized?
Yes, leading and trailing spaces can prevent the filenames from matching. I have modified the script in Post #24, to remove leading/trailing spaces in filenames from your text file before matching them.
Reply With Quote
  #43  
Old 10.02.2020, 08:55
Fetter Biff Fetter Biff is offline
JD Legend
 
Join Date: Oct 2014
Posts: 945
Default

Quote:
To disable the link instead of skipping it, replace link.setSkipped(true); with link.setEnabled(false); in both the scripts.
Thank you!

Quote:
The script will be present in the comment field till it is deleted by you. The links will however be unskipped automatically when the download is started, so the script will check the comment and skip it again.
So an automatically / second / further check after restarting / starting JD cannot be avoid, I assume. But if the skip link message (of the script) is disabled the user will not notice it when skipping is enabled or disabling, I assume.

Thank you very much for the new script!

Quote:
Dupe link check is done only when adding links to linkgrabber. Dupe file check is done only when the download starts.
Could the links in the link grabber (when added to it) checked for dupe links (for file check it is not possible, I assume, as they are checked only at download start) already be marked (in another color as JD marks them) in the link grabber window to be able to remove them before starting the downloads / adding them to the download list?

Quote:
Yes, leading and trailing spaces can prevent the filenames from matching. I have modified the script in Post #24, to remove leading/trailing spaces in filenames from your text file before matching them.
Many thanks!
Reply With Quote
  #44  
Old 10.02.2020, 18:39
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 901
Default

Quote:
So an automatically / second / further check after restarting / starting JD cannot be avoid, I assume.
Can be set to check only once. But if there are no subsequent checks, it will result in a duplicate download.

Quote:
Could the links in the link grabber (when added to it) checked for dupe links ... marked (in another color as JD marks them) in the link grabber window to be able to remove them before starting the downloads / adding them to the download list?
Not possible using script. That is why the comment field is used to mark them as duplicate. Such a feature has to be implemented natively in JD GUI.
Reply With Quote
  #45  
Old 10.02.2020, 19:37
Fetter Biff Fetter Biff is offline
JD Legend
 
Join Date: Oct 2014
Posts: 945
Default

Quote:
Can be set to check only once. But if there are no subsequent checks, it will result in a duplicate download.
OK, so the result of the first check is not kept, not available anymore after a new start of JD.

Quote:
Could the links in the link grabber (when added to it) checked for dupe links ... marked (in another color as JD marks them) in the link grabber window to be able to remove them before starting the downloads / adding them to the download list?
Not possible using script. That is why the comment field is used to mark them as duplicate. Such a feature has to be implemented natively in JD GUI.
Or can they be marked in another way? For example as skipped or disabled like done in the download list window?
Reply With Quote
  #46  
Old 10.02.2020, 19:53
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 901
Default

Quote:
OK, so the result of the first check is not kept, not available anymore after a new start of JD.
Result will be kept (dupe comment will still be present), but because the script will run only once, the link will not be checked again and skipped.

Quote:
Or can they be marked in another way? For example as skipped or disabled like done in the download list window?
Links in linkgrabber tab cannot be skipped, but they can be disabled. In the script, below link.setComment(comment); add link.setEnabled(false);.

If you just want to find which links are marked as duplicate, use the search bar in the bottom toolbar. Select search by comment and add #duplicatelink in the search field.
Reply With Quote
  #47  
Old 10.02.2020, 20:02
Fetter Biff Fetter Biff is offline
JD Legend
 
Join Date: Oct 2014
Posts: 945
Default

Quote:
Can be disabled. In the script, below link.setComment(comment); add link.setEnabled(false);.
Thank you. Looks like this now:
Code:
// Skip download link, if filename exists in the user-spcified list
// To download the file (prevent skipping), remove "#duplicatefile" from comment
// Trigger required: A Download Started

var fileNamesList = "I:\\jD-Downloads\\jD-Dummies.txt"; // < Set path to text file which contain the file names. Use "\\" or "/" as path separators.
var dupeFilecheck = link.getProperty("dupeFileCheck");
var linkName = link.getName();
var comment = link.getComment() || "";
var skipLink = function() {
    link.setSkipped(true); //To disable the link instead of skipping it, replace link.setSkipped(true); with link.setEnabled(false); in both the scripts.
    alert("Download Skipped: File \"" + linkName + "\" is present in files list.");
}

if (dupeFilecheck) {
    if (comment.indexOf("#duplicatefile") > -1) skipLink();
} else {
    var fileNames = readFile(getPath(fileNamesList)).trim().split("\r\n");

    fileNames.some(function(fileName) {
        if (linkName == fileName.trim()) {
            var text = "#duplicatefile";
            comment = comment ? text + " " + comment : text;
            link.setComment(comment);
            link.setEnabled(false);
            skipLink();
            return true;
        }
    })

    link.setProperty("dupeFileCheck", true);
}
Quote:
If you just want to find which links are marked as duplicate, use the search bar in the bottom toolbar. Select search by comment and add #duplicatelink in the search field.
That is great, thank you!
Reply With Quote
  #48  
Old 10.02.2020, 20:17
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 901
Default

Quote:
Thank you. Looks like this now
That is the file check script which runs on download start. You can also use disable the link there, but I thought you wanted to disable links in linkgrabber.

skipLink(); is a invalid command. To skip link you have to use link.setSkipped(true);. Also, if you choose to disable the link it is not necessary to also skip it.

You will also have to add link.setEnabled(false); in the link check script, if you want to disble the links in linkgrabber tab.
Reply With Quote
  #49  
Old 10.02.2020, 20:53
Fetter Biff Fetter Biff is offline
JD Legend
 
Join Date: Oct 2014
Posts: 945
Default

Oh sorry, so many scripts, the wrong one. So I comment it out and could use it if need be. So I can use link.setEnabled(false); twice (if need be)(?) - where the lines are commentet out:


Quote:
skipLink(); is a invalid command. To skip link you have to use link.setSkipped(true);. Also, if you choose to disable the link it is not necessary to also skip it.
Oops, two times the invalid command, do not have any idea how these commands get there.
Very strange, the script (obviously) worked with an invalid command.

Quote:
Also, if you choose to disable the link it is not necessary to also skip it.
Yes, isn't it disabled (commented out)? But it would work? Both ones at the same time, I mean? I just would exchange the commands, if need be, comment out the other one.

Hope, it is correct now:
Code:
// Skip download link, if filename exists in the user-spcified list
// To download the file (prevent skipping), remove "#duplicatefile" from comment
// Trigger required: A Download Started

var fileNamesList = "I:\\jD-Downloads\\jD-Dummies.txt"; // < Set path to text file which contain the file names. Use "\\" or "/" as path separators.
var dupeFilecheck = link.getProperty("dupeFileCheck");
var linkName = link.getName();
var comment = link.getComment() || "";
var skipLink = function() {
    link.setSkipped(true); //To disable the link instead of skipping it, replace link.setSkipped(true); with link.setEnabled(false); in both the scripts.
    alert("Download Skipped: File \"" + linkName + "\" is present in files list.");
}

if (dupeFilecheck) {
    if (comment.indexOf("#duplicatefile") > -1) link.setSkipped(true);
} else {
    var fileNames = readFile(getPath(fileNamesList)).trim().split("\r\n");

    fileNames.some(function(fileName) {
        if (linkName == fileName.trim()) {
            var text = "#duplicatefile";
            comment = comment ? text + " " + comment : text;
            link.setComment(comment);
            //            link.setEnabled(false); falls Duplikate im Download-Fenster enabled werden sollen
            link.setSkipped(true);
            return true;
        }
    })

    link.setProperty("dupeFileCheck", true);
}
Quote:
You will also have to add link.setEnabled(false); in the link check script, if you want to disble the links in linkgrabber tab.
Hope, it is correct:
Code:
// If download link is present in history file, mark it as duplicae (add "#duplicatelink" to comment)
// Trigger required: Packagizer Hook

if (state == "AFTER") {
    var url = link.getURL();
    var historyFile = getPath(JD_HOME + "/cfg/history.txt");
    var history = historyFile.exists() ? readFile(historyFile) : "";

    if (history.indexOf(url) > -1) {
        var text = "#duplicatelink";
        var comment = link.getComment();
        comment = comment ? text + " " + comment : text;
        link.setComment(comment);
        link.setEnabled(false);
        
    }
}
Reply With Quote
  #50  
Old 10.02.2020, 21:14
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 901
Default

Quote:
Oops, two times the invalid command, do not have any idea how these commands get there.
Very strange, the script (obviously) worked with an invalid command.
Sorry, my bad.

skipLink(); is a valid command. It is a custom command which I had created in the script to skip the link and show the message, which I totally forgot.

You can use skipLink(); (to skip and show the message) or link.setSkpped(true); (to only skip the link).

You can disable the link in both file check and link check scripts if you wish. I used skip insted of disable, because it is easier to unskp the link. Only one click in status bar or toolbar.

Modification appears to be OK.
Reply With Quote
  #51  
Old 10.02.2020, 21:26
Fetter Biff Fetter Biff is offline
JD Legend
 
Join Date: Oct 2014
Posts: 945
Default

Quote:
Sorry, my bad.
No, no, no reason to say sorry, does not matter.

Quote:
You can use skipLink(); (to skip and show the message) or link.setSkpped(true); (to only skip the link).
So if I want the message(s) to be shown I just would exchange these commands. But that does not sound like an invalid command, but just one with another function, I would think.

Quote:
You can disable the link in both file check and link check scripts if you wish.
Disable the link check? Or what link? I would not turn any link check off.

Quote:
I used skip insted of disable, because it is easier to unskp the link. Only one click in status bar or toolbar.
A good point. But unskipping a duplicate link would not be necessary here, just remove it.

Quote:
Modification appears to be OK.
Thank you!
Reply With Quote
  #52  
Old 10.02.2020, 21:43
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 901
Default

Quote:
So if I want the message(s) to be shown I just would exchange these commands. But that does not sound like an invalid command, but just one with another function, I would think.
Yes skipLink function has two commands link.setSkipped and alert (to show message). You can use only skipLink. To active/deactivate message display, simply comment/uncomment the alert command.

Quote:
Disable the link check? Or what link? I would not turn any link check off.
Not link check. I was suggesting, if you wish, you can use link.setDisabled in both scripts, instead of skipLink.

Quote:
A good point. But unskipping a duplicate link would not be necessary here, just remove it.
Deleting duplicate link directly using the script is also possible. But, there is no UNDO action in JD. So, it is better to give the user a chance to review the links before deleting it.
Reply With Quote
  #53  
Old 10.02.2020, 21:55
Fetter Biff Fetter Biff is offline
JD Legend
 
Join Date: Oct 2014
Posts: 945
Default

Quote:
Yes skipLink function has two commands link.setSkipped and alert (to show message). You can use only skipLink. To active/deactivate message display, simply comment/uncomment the alert command.
OK. One can comment out / in entire lines only, not a part of a line? So to exchange two commands against each other the easiest way would to do it like this(?):
Code:
//skipLink();
link.setSkpped(true);
and
Code:
skipLink();
//link.setSkpped(true);
Quote:
Not link check. I was suggesting, if you wish, you can use link.setDisabled in both scripts, instead of skipLink.
Sorry.

Quote:
Deleting duplicate link directly using the script is also possible. But, there is no UNDO action in JD. So, it is better to give the user a chance to review the links before deleting it.
Yes, that's true. I will see now how it is using the great scripts in allday life and if need be may be I would try to aks for an instant removing option.
Reply With Quote
  #54  
Old 10.02.2020, 22:26
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 901
Default

Quote:
One can comment out / in entire lines only, not a part of a line? So to exchange two commands against each other the easiest way would to do it like this(?):
Yes that is the correct way to comment out the entire line to prevent the command from being executed.

It is also possible to comment part of the line.

Code:
// Entire line is commented

This is part is not commented // This is part is commented

This is not commented /* This is commented */ This is not commented

/*
This is commented
This is commented
This is commented
*/
Reply With Quote
  #55  
Old 10.02.2020, 22:47
Fetter Biff Fetter Biff is offline
JD Legend
 
Join Date: Oct 2014
Posts: 945
Default

Ah yes, very understandable! Thank you very much!
Reply With Quote
  #56  
Old 10.06.2020, 13:46
Dockel Dockel is offline
Giga Loader
 
Join Date: Feb 2020
Posts: 93
Default

An unbelievably great script.

It seems with YouTube links / videos it does not work perfectly because somehow the YouTube links of (the same) videos are changing? And because one can choose different variants (of the same video) with different compression. Is there a way to treat all of the different variations of the same videos as duplicates? So once a video is downloaded all of the other smaller or bigger sizes of a video would count as duplicate by the script?
__________________
Aktuelles Windows
Reply With Quote
  #57  
Old 10.06.2020, 17:42
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 901
Default

Quote:
Originally Posted by Dockel View Post
... with YouTube links ... Is there a way to treat all of the different variations of the same videos as duplicates? ...
In the script, replace:
Code:
var url = link.getContentURL() || link.getPluginURL();

with:
Code:
var url = link.getHost() == "youtube.com" ? link.getContainerURL() : link.getContentURL() || link.getPluginURL();

You can manually strip the variant string (following the video ID) from the urls which already exist in your 'history' file. (Keep a backup of the file, just in case).

Example string to remove from url:
Code:
#variant=ew0KICAiaWQiIDogIk1QNF9IMjY0XzM2MFBfMzBGUFNfQUFDXzk2S0JJVCIsDQogICJkYXRhIiA6ICJ7XHJcbiAgXCJhQml0cmF0ZVwiIDogLTEsXHJcbiAgXCJoZWlnaHRcIiA6IDM1MixcclxuICBcIndpZHRoXCIgOiA2NDAsXHJcbiAgXCJmcHNcIiA6IDI5LFxyXG4gIFwicHJvamVjdGlvblwiIDogXCJOT1JNQUxcIlxyXG59Ig0KfQ%3D%3D
Reply With Quote
  #58  
Old 10.06.2020, 19:28
Dockel Dockel is offline
Giga Loader
 
Join Date: Feb 2020
Posts: 93
Default

Thank you very much!

I have changed one of the scripts like this:

Code:
// Skip link if it present in download history (has "#duplicatelink" in comment)
// To download the file (prevent skipping), remove "#duplicatelink" from comment
// Trigger required: A Download Started

var comment = link.getComment() || "";

if (comment.indexOf("#duplicatelink") > -1) {
    // var url = link.getContentURL() || link.getPluginURL(); // Nur tatsächlich runtergeladene Varianten von YouTube werden als Duplikate behandelt
    var url = link.getHost() == "youtube.com" ? link.getContainerURL() : link.getContentURL() || link.getPluginURL(); // alle Varianten auf YouTube werden als Duplikate behandelt
    link.setSkipped(true); //To disable the link instead of skipping it, replace link.setSkipped(true); with link.setEnabled(false); in both the scripts
    alert("Download Skipped: \"" + url + "\" is present in history file.");
}ff
Quote:
You can manually strip the variant string (following the video ID) from the urls which already exist in your 'history' file. (Keep a backup of the file, just in case).
Sorry, I do not understand what this means. I can remove the code in the history.txt that causes the downloaded links / files be treaten as duplicates only (instead of all the variants of a video)?
What do I have to do? Search a part of the example string in the history file?
__________________
Aktuelles Windows
Reply With Quote
  #59  
Old 10.06.2020, 22:25
Dockel Dockel is offline
Giga Loader
 
Join Date: Feb 2020
Posts: 93
Default

Obviously I maed a mistake, the script does not seem to properly work anymore:


Code:
net.sourceforge.htmlunit.corejs.javascript.EcmaError: ReferenceError: "ff" is not defined. (#12)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3629)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3613)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.notFoundError(ScriptRuntime.java:3683)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.name(ScriptRuntime.java:1690)
	at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1622)
	at script(:12)
	at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
	at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
	at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411)
	at org.jdownloader.scripting.JSHtmlUnitPermissionRestricter$SandboxContextFactory.doTopCall(JSHtmlUnitPermissionRestricter.java:119)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3057)
	at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
	at net.sourceforge.htmlunit.corejs.javascript.Context.evaluateString(Context.java:1212)
	at org.jdownloader.extensions.eventscripter.ScriptThread.evalUNtrusted(ScriptThread.java:288)
	at org.jdownloader.extensions.eventscripter.ScriptThread.executeScipt(ScriptThread.java:180)
	at org.jdownloader.extensions.eventscripter.ScriptThread.run(ScriptThread.java:160)
__________________
Aktuelles Windows

Last edited by Dockel; 10.06.2020 at 22:32.
Reply With Quote
  #60  
Old 14.06.2020, 23:16
tarkett tarkett is offline
Storm
 
Join Date: Apr 2009
Location: Deutschland
Posts: 224
Default

Change the last line
Code:
}ff
to
Code:
}
and test it again.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 10:06.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.