Quote:
Originally Posted by Jiaz
The whole crawling process does not check against *already downloaded files* nor any *duplicate checks in Linkgrabber*.
|
Well, but obviously a dupe check is performed every time, after I click on
Continue in LinkGrabber.
And as I said, I have 150,000+ downloaded files.
So, for each and every link that I copy to LinkGrabber a dupe check will have to be performed against those 150,000+ files and also against those, which are still in the list in LinkGrabber pane.
How much time would that cost?
When I do that in Excel, ist is only a few seconds. But how fast is JD compared to that?
I can only tell, that
it is reproducible here, that a clean JD - which I just set up for my tests - is much, much faster, than my Work-JD with that large number of 'known' files/links.
And this also applies to the start-up time JD needs, which is understandable.
On the other hand, the zip-archives in cfg contain a lot(!) of data. Don't know how JD handles this and how much this can slow JD down.