[mgpai]

Originally Posted by Dockel

... the script obviously does not indicate them as dups (although they have already been added to history.txt. Could those links be marked / disabled as dups anyway?

If the urls were already present in the history file at the time of adding them to LG list, it will be (should be) marked as duplicate by the script. Only time they would not be marked as such is, if they were not an exact match. Check both urls and see if there is any difference. Also see if you can reproduce the issue with same or any other urls.

[Dockel]

Sorry, I made a mistake. The links in the link grabber are shown as dups by JD (red), they are in the download list already. When I then add the links in the download list to history.txt and press "Start all downloads" they are not marked as dups by the script. Is there a way the script could mark it es dups?

[Jiaz]

@Dockel
@mgpai
The script "move to history" needs to add the TAG to comment field, so the "Start Download" script can work as intended.

[Dockel]

I know there is a tag added to the comment, but I do not know which of the scripts it does.

[mgpai]

Originally Posted by Jiaz

@mgpai: The script "move to history" needs to add the TAG to comment field, so the "Start Download" script can work as intended.

They are not mapped to corresponding highlighted links in linkgrabber, so it will require another dupe check to match the selected links against each link present in linkgrabber and mark them as duplicate.

[Jiaz]

@mgpai: maybe another script to *recheck/match* against history.txt ?

[mgpai]

Originally Posted by Jiaz

maybe another script to *recheck/match* against history.txt ?

One more is one too many.

Implemented in existing script with slight changes. Also added availablestate check to the script.

/*
    Add finished download links from selection to history, and remove them from the list
    Tirgger required: Downloadlist Contextmenu Button Pressed
    Customize downloadlist context menu > Add new "Eventscripter Trigger" button > Rename it to "Move to history" (without quotes) > Change Icon (optional)
*/

if (name == "Move to history") {
    var file = getPath(JD_HOME + "/cfg/history.txt"),
        urls = file.exists() ? readFile(file) : "",
        newUrls = "",
        url, data, text, comment;

    dlSelection.links.forEach(function(link) {
        if (link.finished) {
            url = link.contentURL || link.pluginURL;

            if (urls.indexOf(url) == -1) {
                newUrls += url + "\r\n";
                link.remove();
            }
        }
    });

    if (newUrls.length) {
        data = "Added from the downloadlist list on " + Date().toString().substring(4, 24) + "\r\n";
        data += newUrls + "\r\n";
        writeFile(file, data, true);

        getAllCrawledLinks().forEach(function(link) {
            if (link.availableState == "ONLINE") {
                url = link.contentURL || link.pluginURL;
                text = "#duplicatelink";
                comment = link.comment || "";
                
                if (comment.indexOf(text) == -1 && (urls + newUrls).indexOf(url) > -1) {
                    link.comment = comment ? text + " " + comment : text;
                }
            }
        });
    }
}

[Dockel]

Great, thank you very much!

[mgpai]

@Dockel: You're welcome. Fee free to let me know if you need any new scripts, or make any changes to the existing ones.

[Dockel]

Yes, I will do so, very happy about that, very many thanks! And very, very needful the history script.

[Jiaz]

@mgpai: Thanks for your help with the scripts

[mgpai]

@Dockel: Please provide in a single post, the current versions of ALL your 'history' scripts. Use 'code' tag for scripts when composing the post.

[Dockel]

// Add selected linkgrabber links to history, and remove them from the list
// Tirgger required: Linkgrabber Contextmenu Button Pressed
// Customize linkgrabber context menu > Add new "Eventscripter Trigger" button > Rename it to "Move to history" (without quotes) > Change Icon (optional)

if (name == "Move to history") {
    var historyFile = getPath(JD_HOME + "/cfg/history.txt");
    if (!historyFile.exists()) writeFile(historyFile, "", true);
    var history = readFile(historyFile);

    var linkUrls = [];
    var linkIds = [];

    var removeLinks = function(linkIds) {
        callAPI("linkgrabberv2", "removeLinks", linkIds, []);
    }

    lgSelection.getLinks().forEach(function(link) {
        var url = link.getUrl();
        var id = link.getUUID();

        if (history.indexOf(url) == -1) linkUrls.push(url);
        linkIds.push(id);
    })

    if (linkUrls.length) {
        data = "Added from the lingkrabber list on " + Date().toString().substring(4, 24) + "\r\n";
        data += linkUrls.join("\r\n") + "\r\n\r\n";
        writeFile(historyFile, data, true);
    }

    if (linkIds.length) removeLinks(linkIds);
}

/*
    Add finished download links from selection to history, and remove them from the list
    Tirgger required: Downloadlist Contextmenu Button Pressed
    Customize downloadlist context menu > Add new "Eventscripter Trigger" button > Rename it to "Move to history" (without quotes) > Change Icon (optional)
*/

if (name == "Move to history") {
    var file = getPath(JD_HOME + "/cfg/history.txt"),
        urls = file.exists() ? readFile(file) : "",
        newUrls = "",
        url, data, text, comment;

    dlSelection.links.forEach(function(link) {
        if (link.finished) {
            url = link.contentURL || link.pluginURL;

            if (urls.indexOf(url) == -1) {
                newUrls += url + "\r\n";
                link.remove();
            }
        }
    });

    if (newUrls.length) {
        data = "Added from the downloadlist list on " + Date().toString().substring(4, 24) + "\r\n";
        data += newUrls + "\r\n";
        writeFile(file, data, true);

        getAllCrawledLinks().forEach(function(link) {
            if (link.availableState == "ONLINE") {
                url = link.contentURL || link.pluginURL;
                text = "#duplicatelink";
                comment = link.comment || "";
                
                if (comment.indexOf(text) == -1 && (urls + newUrls).indexOf(url) > -1) {
                    link.comment = comment ? text + " " + comment : text;
                }
            }
        });
    }
}

// Remove #duplicatelink and #dupliatefile from comment field for selected links
// Trigger: Downloadlist Contextmenu Button Pressed

if (name = "Clear dupe comment") {
    dlSelection.getLinks().forEach(function(link) {
        var myComments = ["#duplicatelink", "#duplicatefile"];
        myComments.forEach(function(myComment) {
            var comment = link.getComment() || "";
            if (comment.indexOf(myComment) > -1) link.setComment(comment.replace("#duplicatelink", "").replace("#duplicatefile", ""));
        })
    })
}

// Remove/Re-add filecrypt.cc links with wrong captcha
// Trigger: Interval (Recommended 60000 or more)
// IMPORTANT: Enable ''Synchronous execution of script' in top panel

var delay = 15; // Set delay (in seconds) between retries

links = getAllCrawledLinks().filter(function(crawledLink) {
    var link = crawledLink.contentURL || "";
    return link.indexOf("filecrypt.cc") > -1 && crawledLink.name.indexOf("Wrong Captcha!") > -1;
}).forEach(function(crawledLink) {
    sleep(delay * 1000);
    callAPI("linkgrabberv2", "addLinks", {
        "links": crawledLink.contentURL
    });
    crawledLink.remove();
})

/* 
    Disable duplicate link in Lingrabber
    Trigger required: A new link has been added
*/

var link = crawledLink;

if (link.availableState == "ONLINE") {
    var name = "history";
    var lock = getModifyLock(name);

    lock.writeLock();

    var urls = getProperty(name, true);

    if (urls == null) {
        try {
            var file = JD_HOME + "/cfg/history.txt";
            var urls = readFile(file);

            setProperty(name, urls, true);
        } catch (e) {}
    }

    lock.writeUnlock();

    if (urls) {
        var url = link.contentURL || link.pluginURL;

        if (link.host == "youtube.com") {
            url = link.containerURL;
        }

        if (urls.indexOf(url) > -1) {
            var text = "#duplicatelink";
            var comment = link.comment;

            if (comment) {
                text = text + " " + comment;
            }

            link.comment = text;
            link.enabled = false;
        }
    }
}

This one does not seem not to be needed anymore, it is deactivated, but who knows:

// Skip download link, if filename exists in the user-spcified list
// To download the file (prevent skipping), remove "#duplicatefile" from comment
// Trigger required: A Download Started

var fileNamesList = "I:\\jD-Downloads\\jD-Dummies von jD-Downloads - alle Dateien, Dummies aus dem Ordner - alle ca. 95.000 Dateien als Liste.txt"; // < Set path to text file which contain the file names. Use "\\" or "/" as path separators.
var dupeFilecheck = link.getProperty("dupeFileCheck");
var linkName = link.getName();
var comment = link.getComment() || "";
var skipLink = function() {
    link.setSkipped(true); //To disable the link instead of skipping it, replace link.setSkipped(true); with link.setEnabled(false); in both the scripts.
    alert("Download Skipped: File \"" + linkName + "\" is present in files list.");
}

if (dupeFilecheck) {
    if (comment.indexOf("#duplicatefile") > -1) link.setSkipped(true);
} else {
    var fileNames = readFile(getPath(fileNamesList)).trim().split("\r\n");

    fileNames.some(function(fileName) {
        if (linkName == fileName.trim()) {
            var text = "#duplicatefile";
            comment = comment ? text + " " + comment : text;
            link.setComment(comment);
            //            link.setEnabled(false); falls Duplikate im Download-Fenster enabled werden sollen
            link.setSkipped(true);
            return true;
        }
    })

    link.setProperty("dupeFileCheck", true);
}

And this also seems not be needed anymore, deactivated also:

// Skip download link, if filename exists in the user-spcified list
// To download the file (prevent skipping), remove "#duplicatefile" from comment
// Trigger required: A Download Started

var fileNamesList = "I:\\jD-Downloads\\jD-Dummies von jD-Downloads - alle Dateien, Dummies aus dem Ordner - alle ca. 95.000 Dateien als Liste.txt"; // < Set path to text file which contain the file names. Use "\\" or "/" as path separators.
var dupeFilecheck = link.getProperty("dupeFileCheck");
var linkName = link.getName();
var comment = link.getComment() || "";
var skipLink = function() {
    link.setSkipped(true); //To disable the link instead of skipping it, replace link.setSkipped(true); with link.setEnabled(false); in both the scripts.
    alert("Download Skipped: File \"" + linkName + "\" is present in files list.");
}

if (dupeFilecheck) {
    if (comment.indexOf("#duplicatefile") > -1) link.setSkipped(true);
} else {
    var fileNames = readFile(getPath(fileNamesList)).trim().split("\r\n");

    fileNames.some(function(fileName) {
        if (linkName == fileName.trim()) {
            var text = "#duplicatefile";
            comment = comment ? text + " " + comment : text;
            link.setComment(comment);
            //            link.setEnabled(false); falls Duplikate im Download-Fenster enabled werden sollen
            link.setSkipped(true);
            return true;
        }
    })

    link.setProperty("dupeFileCheck", true);
}

[mgpai]

I needed only 'history' scripts. I am less sure than you as to which scripts are needed. I'll try to figure out.

[Dockel]

Sorry, so I see I am not sure what the history scripts are, I thought the ones working together somehow, needed for the entire task. I have marked them as belonging to the history scripts.

[Dockel]

It seems the script(s) does not disable / mark the dups anymore. And there was a finished download I could not move to the history.txt (edit: when it already is in the history, it stays in the download window when trying to move it to history.txt). Two others I could move. May be I am missing anything. Is there anything I could do?

[Jiaz]

@Dockel: mgpai has to update all other *history* scripts as well to work with optimized inmemory history

[Dockel]

OK, thank you.

[mgpai]

Originally Posted by Jiaz

@Dockel: mgpai has to update all other *history* scripts as well to work with optimized inmemory history

The problem reported does not seem to be related to memory optimization or lack thereof.

I will optimize these scripts, but, I do not want to use global properties in new scripts, if it needs additional code to handle collision. It doesn't really make sense having to repeat the same code every time we need to read/write them. I think that should be the default behavior for global properties, or at least there should be a flag/switch to make it so.

[mgpai]

Originally Posted by Dockel

It seems the script(s) does not disable / mark the dups anymore.

When adding new links to linkgrabber? New script will only process links which are ONLINE, unlike the old script, which did not check for availability, which would result in wrong type of links (e.g. filecrypt container) being added to the history file.

Originally Posted by Dockel

And there was a finished download I could not move to the history.txt (edit: when it already is in the history, it stays in the download window when trying to move it to history.txt). Two others I could move. May be I am missing anything. Is there anything I could do?

I think is being caused by wrong placement of 'remove' method in current version of the script. Try with this script:

/*
    Add finished download links from selection to history, and remove them from the list
    Tirgger required: Downloadlist Contextmenu Button Pressed
    Customize downloadlist context menu > Add new "Eventscripter Trigger" button > Rename it to "Move to history" (without quotes) > Change Icon (optional)
*/

if (name == "Move to history") {
    var file = getPath(JD_HOME + "/cfg/history.txt"),
        urls = file.exists() ? readFile(file) : "",
        newUrls = "",
        url, data, text, comment;

    dlSelection.links.forEach(function(link) {
        if (link.finished) {
            url = link.contentURL || link.pluginURL;

            if (urls.indexOf(url) == -1) {
                newUrls += url + "\r\n";
            }
            
            link.remove();
        }
    });

    if (newUrls.length) {
        data = "Added from the downloadlist list on " + Date().toString().substring(4, 24) + "\r\n";
        data += newUrls + "\r\n";
        writeFile(file, data, true);

        getAllCrawledLinks().forEach(function(link) {
            if (link.availableState == "ONLINE") {
                url = link.contentURL || link.pluginURL;
                text = "#duplicatelink";
                comment = link.comment || "";
                
                if (comment.indexOf(text) == -1 && (urls + newUrls).indexOf(url) > -1) {
                    link.comment = comment ? text + " " + comment : text;
                }
            }
        });
    }
}