Hi again Akasen,
I've investigated this.
It seems like twitter has a max. number of tweets you can see/go back.
Code:
641|twitter.com_jd.plugins.decrypter.TwitterCom 31.03.20 15:36:59 - INFO [ jd.plugins.decrypter.TwitterCom(crawlUserViaAPI) ] -> Numberof tweets on current page: 0 of expected max 20
641|twitter.com_jd.plugins.decrypter.TwitterCom 31.03.20 15:37:01 - INFO [ jd.plugins.decrypter.TwitterCom(crawlUserViaAPI) ] -> Numberof total tweets crawled: 829 of expected total 2748
By default, twitter will return 20 tweets per page --> Every tweet may contain a different amount of downloadable media.
Basically we noticed this in the past as well.
You can even check this via browser by going back as far as 41 "pages" which means 40x reloading by scrolling down.
The issue is also that a lot of websites would allow to e.g. "start at position 500".
Twitter however uses so called "cursors" which means to access the next page, you will have to get a token which is only available on the previous page so even if I wanted, I would not be able to give you any options to e.g. start at position 800 in this case.
For your other URL, it finds 200 objects which should be all as it only contains 162 tweets.
I could now e.g. experiment and display more objects per page but although thiy may return some more objects, I we would run into similar issues with URLs containing even more objects.
I recommend you to:
- Test via browser and see how far you can get and if you can e.g. get more than JD does
- Search the Internet for other Twitter downloader tools --> If you find one that does a better job than our crawler, let me know and I'll look into it again
-psp-