[Solved] JD not capturing all links in Motherless.com

[globus999]

Hi,
If I select a Group in womanless.com and then either pictures or videos, for example:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

LG only crawls over a small percentage of the available material. For example, in the links above it only finds 500+ items when in reality, there are 9000+ items in each set. It is a tad strange because LG is not stopping at a fixed number, but at around 500 for many sets, but not always. For example, in this set:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

stops at 190 but there are 400+ items available.

If you could please take a look, it would be very much appreciated!

[pspzockerscene]

Originally Posted by globus999

LG only crawls over a small percentage of the available material. For example, in the links above it only finds 500+ items when in reality, there are 9000+ items in each set.

Groups often contain such huge amounts of items that our groups crawler is limited to 500 items by default in order to avoid spamming that website especially because a lot of users are adding items by mistake or through other websites and are not aware of the amount of crawl processes this can trigger.
You can change that limit under Settings -> Plugins

Originally Posted by globus999

It is a tad strange because LG is not stopping at a fixed number, but at around 500 for many sets, but not always.

That limit is checked only once after every crawled page.
The number of items per page may vary so even if the limit is set to 500 (=default), you may get a little bit more items. The number should be consistend for every attempt of the same link though.

Originally Posted by globus999

stops at 190 but there are 400+ items available.

Well at the first glance you're right as the website says "Videos(463)".
However, I can only see 4 pages (3 * 60 items + 11 items on the last page) with a total of 191 items.
Please double-check this and in case I'm right, you will need to contact the admins of that website regarding the potentially missing content / wrong number.

To sum this up:
Our plugin looks to be working as expected.

[globus999]

You are absolutely correct! Deepest apologies for wasting your time. Next time I'll check those parameters *before* posting. Again, terribly sorry.

[pspzockerscene]

No worries.

As said, bugs can happen on any side so you might want to report that to the website-owner.
I'll mark this thread as "Solved" now but it will remain open so you can add more information if you want.