View Single Post
Old 10.11.2020, 00:44
serveral serveral is offline
Junior Loader
Join Date: Aug 2020
Posts: 10

Yes, the behavior is indeed odd as it only happens when I'm logged in through JD, otherwise the crawler functions just as intended. This wouldn't be so much of an issue if it wasn't for the fact that you need an account to crawl their other sister/mirror site

Also I had contacted their staff earlier when I was still figuring out what was triggering the bans, and they told me their system should return a "you've been banned" message of sorts (which should precede the site returning blank pages) explaining the reason of the ban, duration, etc. However since the ban is triggered by some sort of interaction between the crawler and the JD log-in system, (as opposed to it being triggered manually through user activity) that message gets kinda lost since it only shows up once, and I lack the technical knowledge as to how one would go around trying to catch that initial message (does the crawler log these sort of messages/interactions with the servers?). Just thought of adding this as well since the message might contain some extra technical data, I don't know...