Many thanks!

Once you comment out the message, JD will only skip it and not display any message. Skipping is required to prevent downloading the file again.

So deactivating (instead of skipping) the links would not work, I guess.

To avoid dupe checking everytime you start JD, remove the "#duplicatelink" or "#duplicatefile" from the comment (if you are plannig to redownload the file)

So it is not possible to keep the "#duplicatefile" in the comments when JD closes / restarts, I assume.

How could one remove the "#duplicatelink" from all comments / for all duplicates with a single action / click or so instead of removing them maunally one by one?

Could the links be checked for duplicates in the link grabber already?

So if there is no connection to the server the files to be downloaded are stored on or there is a wainting time for downloads it is not possible to check for duplicates? So one had to wait for the waiting time to finish to let the script check for the dupllicats?

Although links (from Mega) just were downloaded and added to the history they were downloaded a second time after having added them to JD again. Why is this? In a next try they are skipped.

In this list

	
	
// Skip download link, if filename exists in the user-spcified list
// To download the file (prevent skipping), remove "#duplicatefile" from comment
// Trigger required: A Download Started

var fileNamesList = "I:\\jD-Downloads\\jD-Dummies.txt"; // < Set path to text file which contain the file names. Use "\\" or "/" as path separators.
var dupeFilecheck = link.getProperty("dupeFileCheck");
var linkName = link.getName();
var comment = link.getComment() || "";
var skipLink = function() {

there are trailing spaces in each line (after the name of the archivies already being download). Does that - the spaces - avoid the already downloaded archives to be recognized?

[mgpai]

So deactivating (instead of skipping) the links would not work, I guess.

To disable the link instead of skipping it, replace link.setSkipped(true); with link.setEnabled(false); in both the scripts.

So it is not possible to keep the "#duplicatefile" in the comments when JD closes / restarts, I assume.

The script will be present in the comment field till it is deleted by you. The links will however be unskipped automatically when the download is started, so the script will check the comment and skip it again.

How could one remove the "#duplicatelink" from all comments / for all duplicates with a single action / click or so instead of removing them maunally one by one?

// Remove #duplicatelink and #dupliatefile from comment field for selected links
// Trigger: Downloadlist Contextmenu Button Pressed

if (name = "Clear dupe comment") {
    dlSelection.getLinks().forEach(function(link) {
        var myComments = ["#duplicatelink", "#duplicatefile"];
        myComments.forEach(function(myComment) {
            var comment = link.getComment() || "";
            if (comment.indexOf(myComment) > -1) link.setComment(comment.replace("#duplicatelink", "").replace("#duplicatefile", ""));
        })
    })
}

Could the links be checked for duplicates in the link grabber already?

So if there is no connection to the server the files to be downloaded are stored on or there is a wainting time for downloads it is not possible to check for duplicates? So one had to wait for the waiting time to finish to let the script check for the dupllicats?

Script follows the same sequence as JD for dupe check. Dupe link check is done only when adding links to linkgrabber. Dupe file check is done only when the download starts.

The script does not wait for a server respons. It just checks the link or file name against the text file.

Although links (from Mega) just were downloaded and added to the history they were downloaded a second time after having added them to JD again. Why is this? In a next try they are skipped.

The dupe check is performed only when adding the link to linkgrabber. I am assuming the it was present in the download list before it could be checked. To ensure all existing pending links in download list are dupe checked, you will have to remove and add them back again.

In this list ... there are trailing spaces in each line (after the name of the archivies already being download). Does that - the spaces - avoid the already downloaded archives to be recognized?

Yes, leading and trailing spaces can prevent the filenames from matching. I have modified the script in Post #24, to remove leading/trailing spaces in filenames from your text file before matching them.

To disable the link instead of skipping it, replace link.setSkipped(true); with link.setEnabled(false); in both the scripts.

Thank you!

The script will be present in the comment field till it is deleted by you. The links will however be unskipped automatically when the download is started, so the script will check the comment and skip it again.

So an automatically / second / further check after restarting / starting JD cannot be avoid, I assume. But if the skip link message (of the script) is disabled the user will not notice it when skipping is enabled or disabling, I assume.

Thank you very much for the new script!

Dupe link check is done only when adding links to linkgrabber. Dupe file check is done only when the download starts.

Could the links in the link grabber (when added to it) checked for dupe links (for file check it is not possible, I assume, as they are checked only at download start) already be marked (in another color as JD marks them) in the link grabber window to be able to remove them before starting the downloads / adding them to the download list?

Yes, leading and trailing spaces can prevent the filenames from matching. I have modified the script in Post #24, to remove leading/trailing spaces in filenames from your text file before matching them.

Many thanks!

[mgpai]

So an automatically / second / further check after restarting / starting JD cannot be avoid, I assume.

Can be set to check only once. But if there are no subsequent checks, it will result in a duplicate download.

Could the links in the link grabber (when added to it) checked for dupe links ... marked (in another color as JD marks them) in the link grabber window to be able to remove them before starting the downloads / adding them to the download list?

Not possible using script. That is why the comment field is used to mark them as duplicate. Such a feature has to be implemented natively in JD GUI.

Can be set to check only once. But if there are no subsequent checks, it will result in a duplicate download.

OK, so the result of the first check is not kept, not available anymore after a new start of JD.

Could the links in the link grabber (when added to it) checked for dupe links ... marked (in another color as JD marks them) in the link grabber window to be able to remove them before starting the downloads / adding them to the download list?
Not possible using script. That is why the comment field is used to mark them as duplicate. Such a feature has to be implemented natively in JD GUI.

Or can they be marked in another way? For example as skipped or disabled like done in the download list window?

[mgpai]

OK, so the result of the first check is not kept, not available anymore after a new start of JD.

Result will be kept (dupe comment will still be present), but because the script will run only once, the link will not be checked again and skipped.

Or can they be marked in another way? For example as skipped or disabled like done in the download list window?

Links in linkgrabber tab cannot be skipped, but they can be disabled. In the script, below link.setComment(comment); add link.setEnabled(false);.

If you just want to find which links are marked as duplicate, use the search bar in the bottom toolbar. Select search by comment and add #duplicatelink in the search field.

Can be disabled. In the script, below link.setComment(comment); add link.setEnabled(false);.

Thank you. Looks like this now:

// Skip download link, if filename exists in the user-spcified list
// To download the file (prevent skipping), remove "#duplicatefile" from comment
// Trigger required: A Download Started

var fileNamesList = "I:\\jD-Downloads\\jD-Dummies.txt"; // < Set path to text file which contain the file names. Use "\\" or "/" as path separators.
var dupeFilecheck = link.getProperty("dupeFileCheck");
var linkName = link.getName();
var comment = link.getComment() || "";
var skipLink = function() {
    link.setSkipped(true); //To disable the link instead of skipping it, replace link.setSkipped(true); with link.setEnabled(false); in both the scripts.
    alert("Download Skipped: File \"" + linkName + "\" is present in files list.");
}

if (dupeFilecheck) {
    if (comment.indexOf("#duplicatefile") > -1) skipLink();
} else {
    var fileNames = readFile(getPath(fileNamesList)).trim().split("\r\n");

    fileNames.some(function(fileName) {
        if (linkName == fileName.trim()) {
            var text = "#duplicatefile";
            comment = comment ? text + " " + comment : text;
            link.setComment(comment);
            link.setEnabled(false);
            skipLink();
            return true;
        }
    })

    link.setProperty("dupeFileCheck", true);
}

If you just want to find which links are marked as duplicate, use the search bar in the bottom toolbar. Select search by comment and add #duplicatelink in the search field.

That is great, thank you!

[mgpai]

Thank you. Looks like this now

That is the file check script which runs on download start. You can also use disable the link there, but I thought you wanted to disable links in linkgrabber.

skipLink(); is a invalid command. To skip link you have to use link.setSkipped(true);. Also, if you choose to disable the link it is not necessary to also skip it.

You will also have to add link.setEnabled(false); in the link check script, if you want to disble the links in linkgrabber tab.

Oh sorry, so many scripts, the wrong one. So I comment it out and could use it if need be. So I can use link.setEnabled(false); twice (if need be)(?) - where the lines are commentet out:

skipLink(); is a invalid command. To skip link you have to use link.setSkipped(true);. Also, if you choose to disable the link it is not necessary to also skip it.

Oops, two times the invalid command, do not have any idea how these commands get there.
Very strange, the script (obviously) worked with an invalid command.

Also, if you choose to disable the link it is not necessary to also skip it.

Yes, isn't it disabled (commented out)? But it would work? Both ones at the same time, I mean? I just would exchange the commands, if need be, comment out the other one.

Hope, it is correct now:

// Skip download link, if filename exists in the user-spcified list
// To download the file (prevent skipping), remove "#duplicatefile" from comment
// Trigger required: A Download Started

var fileNamesList = "I:\\jD-Downloads\\jD-Dummies.txt"; // < Set path to text file which contain the file names. Use "\\" or "/" as path separators.
var dupeFilecheck = link.getProperty("dupeFileCheck");
var linkName = link.getName();
var comment = link.getComment() || "";
var skipLink = function() {
    link.setSkipped(true); //To disable the link instead of skipping it, replace link.setSkipped(true); with link.setEnabled(false); in both the scripts.
    alert("Download Skipped: File \"" + linkName + "\" is present in files list.");
}

if (dupeFilecheck) {
    if (comment.indexOf("#duplicatefile") > -1) link.setSkipped(true);
} else {
    var fileNames = readFile(getPath(fileNamesList)).trim().split("\r\n");

    fileNames.some(function(fileName) {
        if (linkName == fileName.trim()) {
            var text = "#duplicatefile";
            comment = comment ? text + " " + comment : text;
            link.setComment(comment);
            //            link.setEnabled(false); falls Duplikate im Download-Fenster enabled werden sollen
            link.setSkipped(true);
            return true;
        }
    })

    link.setProperty("dupeFileCheck", true);
}

You will also have to add link.setEnabled(false); in the link check script, if you want to disble the links in linkgrabber tab.

Hope, it is correct:

// If download link is present in history file, mark it as duplicae (add "#duplicatelink" to comment)
// Trigger required: Packagizer Hook

if (state == "AFTER") {
    var url = link.getURL();
    var historyFile = getPath(JD_HOME + "/cfg/history.txt");
    var history = historyFile.exists() ? readFile(historyFile) : "";

    if (history.indexOf(url) > -1) {
        var text = "#duplicatelink";
        var comment = link.getComment();
        comment = comment ? text + " " + comment : text;
        link.setComment(comment);
        link.setEnabled(false);
        
    }
}

[mgpai]

Oops, two times the invalid command, do not have any idea how these commands get there.
Very strange, the script (obviously) worked with an invalid command.

Sorry, my bad.

skipLink(); is a valid command. It is a custom command which I had created in the script to skip the link and show the message, which I totally forgot.

You can use skipLink(); (to skip and show the message) or link.setSkpped(true); (to only skip the link).

You can disable the link in both file check and link check scripts if you wish. I used skip insted of disable, because it is easier to unskp the link. Only one click in status bar or toolbar.

Modification appears to be OK.

Sorry, my bad.

No, no, no reason to say sorry, does not matter.

You can use skipLink(); (to skip and show the message) or link.setSkpped(true); (to only skip the link).

So if I want the message(s) to be shown I just would exchange these commands. But that does not sound like an invalid command, but just one with another function, I would think.

You can disable the link in both file check and link check scripts if you wish.

Disable the link check? Or what link? I would not turn any link check off.

I used skip insted of disable, because it is easier to unskp the link. Only one click in status bar or toolbar.

A good point. But unskipping a duplicate link would not be necessary here, just remove it.

Modification appears to be OK.

Thank you!

[mgpai]

So if I want the message(s) to be shown I just would exchange these commands. But that does not sound like an invalid command, but just one with another function, I would think.

Yes skipLink function has two commands link.setSkipped and alert (to show message). You can use only skipLink. To active/deactivate message display, simply comment/uncomment the alert command.

Disable the link check? Or what link? I would not turn any link check off.

Not link check. I was suggesting, if you wish, you can use link.setDisabled in both scripts, instead of skipLink.

A good point. But unskipping a duplicate link would not be necessary here, just remove it.

Deleting duplicate link directly using the script is also possible. But, there is no UNDO action in JD. So, it is better to give the user a chance to review the links before deleting it.

Yes skipLink function has two commands link.setSkipped and alert (to show message). You can use only skipLink. To active/deactivate message display, simply comment/uncomment the alert command.

OK. One can comment out / in entire lines only, not a part of a line? So to exchange two commands against each other the easiest way would to do it like this(?):

//skipLink();
link.setSkpped(true);

and

skipLink();
//link.setSkpped(true);

Not link check. I was suggesting, if you wish, you can use link.setDisabled in both scripts, instead of skipLink.

Sorry.

Deleting duplicate link directly using the script is also possible. But, there is no UNDO action in JD. So, it is better to give the user a chance to review the links before deleting it.

Yes, that's true. I will see now how it is using the great scripts in allday life and if need be may be I would try to aks for an instant removing option.

[mgpai]

One can comment out / in entire lines only, not a part of a line? So to exchange two commands against each other the easiest way would to do it like this(?):

Yes that is the correct way to comment out the entire line to prevent the command from being executed.

It is also possible to comment part of the line.

// Entire line is commented

This is part is not commented // This is part is commented

This is not commented /* This is commented */ This is not commented

/*
This is commented
This is commented
This is commented
*/

Ah yes, very understandable! Thank you very much!

[Dockel]

An unbelievably great script.

It seems with YouTube links / videos it does not work perfectly because somehow the YouTube links of (the same) videos are changing? And because one can choose different variants (of the same video) with different compression. Is there a way to treat all of the different variations of the same videos as duplicates? So once a video is downloaded all of the other smaller or bigger sizes of a video would count as duplicate by the script?

[mgpai]

Originally Posted by Dockel

... with YouTube links ... Is there a way to treat all of the different variations of the same videos as duplicates? ...

In the script, replace:

var url = link.getContentURL() || link.getPluginURL();

with:

var url = link.getHost() == "youtube.com" ? link.getContainerURL() : link.getContentURL() || link.getPluginURL();

You can manually strip the variant string (following the video ID) from the urls which already exist in your 'history' file. (Keep a backup of the file, just in case).

Example string to remove from url:

#variant=ew0KICAiaWQiIDogIk1QNF9IMjY0XzM2MFBfMzBGUFNfQUFDXzk2S0JJVCIsDQogICJkYXRhIiA6ICJ7XHJcbiAgXCJhQml0cmF0ZVwiIDogLTEsXHJcbiAgXCJoZWlnaHRcIiA6IDM1MixcclxuICBcIndpZHRoXCIgOiA2NDAsXHJcbiAgXCJmcHNcIiA6IDI5LFxyXG4gIFwicHJvamVjdGlvblwiIDogXCJOT1JNQUxcIlxyXG59Ig0KfQ%3D%3D

[Dockel]

Thank you very much!

I have changed one of the scripts like this:

// Skip link if it present in download history (has "#duplicatelink" in comment)
// To download the file (prevent skipping), remove "#duplicatelink" from comment
// Trigger required: A Download Started

var comment = link.getComment() || "";

if (comment.indexOf("#duplicatelink") > -1) {
    // var url = link.getContentURL() || link.getPluginURL(); // Nur tatsächlich runtergeladene Varianten von YouTube werden als Duplikate behandelt
    var url = link.getHost() == "youtube.com" ? link.getContainerURL() : link.getContentURL() || link.getPluginURL(); // alle Varianten auf YouTube werden als Duplikate behandelt
    link.setSkipped(true); //To disable the link instead of skipping it, replace link.setSkipped(true); with link.setEnabled(false); in both the scripts
    alert("Download Skipped: \"" + url + "\" is present in history file.");
}ff

You can manually strip the variant string (following the video ID) from the urls which already exist in your 'history' file. (Keep a backup of the file, just in case).

Sorry, I do not understand what this means. I can remove the code in the history.txt that causes the downloaded links / files be treaten as duplicates only (instead of all the variants of a video)?
What do I have to do? Search a part of the example string in the history file?

[Dockel]

Obviously I maed a mistake, the script does not seem to properly work anymore:

net.sourceforge.htmlunit.corejs.javascript.EcmaError: ReferenceError: "ff" is not defined. (#12)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3629)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3613)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.notFoundError(ScriptRuntime.java:3683)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.name(ScriptRuntime.java:1690)
	at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1622)
	at script(:12)
	at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
	at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
	at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411)
	at org.jdownloader.scripting.JSHtmlUnitPermissionRestricter$SandboxContextFactory.doTopCall(JSHtmlUnitPermissionRestricter.java:119)
	at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3057)
	at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
	at net.sourceforge.htmlunit.corejs.javascript.Context.evaluateString(Context.java:1212)
	at org.jdownloader.extensions.eventscripter.ScriptThread.evalUNtrusted(ScriptThread.java:288)
	at org.jdownloader.extensions.eventscripter.ScriptThread.executeScipt(ScriptThread.java:180)
	at org.jdownloader.extensions.eventscripter.ScriptThread.run(ScriptThread.java:160)

[tarkett]

Change the last line

}ff

to

}

and test it again.