[StefanM]

Originally Posted by Jiaz

The whole crawling process does not check against *already downloaded files* nor any *duplicate checks in Linkgrabber*.

Well, but obviously a dupe check is performed every time, after I click on Continue in LinkGrabber.

And as I said, I have 150,000+ downloaded files.

So, for each and every link that I copy to LinkGrabber a dupe check will have to be performed against those 150,000+ files and also against those, which are still in the list in LinkGrabber pane.

How much time would that cost?
When I do that in Excel, ist is only a few seconds. But how fast is JD compared to that?

I can only tell, that it is reproducible here, that a clean JD - which I just set up for my tests - is much, much faster, than my Work-JD with that large number of 'known' files/links.
And this also applies to the start-up time JD needs, which is understandable.

On the other hand, the zip-archives in cfg contain a lot(!) of data. Don't know how JD handles this and how much this can slow JD down.