I'm also having a similar issue. A site I'm trying to download links from happens to have some links to Twitter in its source. This leads to a endless rabbit hole which the Linkcrawler never finishes from. I've tried adding filters against twitter.com and twimg.com, but linkcrawler still churns away endlessly. Is there something I'm missing?
|