[georgegalily]

very large 500,000 links 45mb

[georgegalily]

It seems when the error occured the script is still runs but not well

[georgegalily]

How much it take to search this file?
JD2 detect duplicates in no time, but keeping the links in the list consume much memory, cpu and hard space

[georgegalily]

@Jiaz
Thank you very much
Removing the sleep make it work well but with low removing speed but acceptable.
This is the used script:

if (state == "AFTER") {
    var url = link.getURL();
    var historyFile = getPath(JD_HOME + "/cfg/history.txt");
    var lock = getModifyLock("history");
    var isDuplicate = false;
    lock.writeLock();
    try {
        var history = historyFile.exists() ? readFile(historyFile) : "";
        isDuplicate = history.indexOf(url) > -1;
        history = null;
    } finally {
        lock.writeUnlock();
    }
    if (isDuplicate) {
        var text = "#duplicatelink";
        var comment = link.getComment();
        comment = comment ? text + " " + comment : text;
        link.setComment(comment);
        link.getCrawledLink().remove();
    }
}

[georgegalily]

Hi, it seems this method is useless becuase the history file is large, checking every link in this history is slow. It takes forever for a bunch of links.
Not to mention that my pc becomes lag and I cannot tell if the even scripter finished checking the links or not, it does not show progress. I only know that it is running from the lag behaviour of the windows.

Putting the sleep in the script lead to failure.

If it is possible to implement an archieve in the download list that collect only the links in a package called (ArchievedLinks) without other informations, this may become handy. Beucase JD is checks the dupls in no time..
Thank you

[Jiaz]

Originally Posted by georgegalily

Hi, it seems this method is useless becuase the history file is large, checking every link in this history is slow. It takes forever for a bunch of links.

Yes, because the script (in current form) does read the whole 50mb file again and again for each link and thus require huge amount of data. It would be faster/better to read the file once into memory and then only read again from disk if content has changed.
It should be updated to read the file once to memory and then all scripts will use the history in memory and not read again and again from disk.
I will try to update the script to work that way, give me some time as I'm no master in scripting like mgpai is
Reference for me, https://board.jdownloader.org/showpo...4&postcount=18

[Jiaz]

Originally Posted by georgegalily

If it is possible to implement an archieve in the download list that collect only the links in a package called (ArchievedLinks) without other informations, this may become handy. Beucase JD is checks the dupls in no time..

We have an open ticket for such an idea but it makes no sense in doing this within JDownloader as this is a job of a *real* database with index/memory optimizations and other stuff. It would make most sense to use real database via http rest api in scripts to achieve this.the current scripting solution works good but of course depending on size of history it must be updated/optimized the larger the databse gets.

[georgegalily]

Originally Posted by Jiaz

Yes, because the script (in current form) does read the whole 50mb file again and again for each link and thus require huge amount of data. It would be faster/better to read the file once into memory and then only read again from disk if content has changed.
It should be updated to read the file once to memory and then all scripts will use the history in memory and not read again and again from disk.
I will try to update the script to work that way, give me some time as I'm no master in scripting like mgpai is
Reference for me, **External links are only visible to Support Staff**...

Thank you, please, take your time