[michael88]

Tiktok profile crawler is not working, with inconsistent responses for different profiles.

1. Website mode is quiet inconsistent with it working some times and other times a blank response with not even an offline link added.
2. Api mode works sometimes and if not then consistently fails with 'Content Offline'.

eg
1.**External links are only visible to Support Staff****External links are only visible to Support Staff**
- works with api, not with website mode.
2.**External links are only visible to Support Staff****External links are only visible to Support Staff**
3.**External links are only visible to Support Staff****External links are only visible to Support Staff**
- both dont work with either website or api mode.

The individual links work in both website and api mode, it's just the crawler that's broken.

[pspzockerscene]

Hi,

1. Fixed bot-protection-check sometimes failing.

2. The API may also fail because: Even ion API mode, website needs to be used one time to find the internal user-ID.
In this case this failed due to their bot-protection.

Some time ago I made a post where I explained how to manually collect all video-links of a profile if our crawler fails for whatever reason:
https://board.jdownloader.org/showpo...4&postcount=13

[michael88]

Hi,
Thanks for the explanation for the inconsistency, i always use a VPN so probably trigger bot protection a lot.

The internal user-Id for eg :
**External links are only visible to Support Staff**...
is i think 107955.

for **External links are only visible to Support Staff**...
is 7081331206307398662.

I already have the internal id's for all the profiles that i want to scrape.

If the api is failing because of the one call to the website to get these id's, could there be some mechanism where i could supply the id's for the profile, and the plugin could get the id's from there instead of the call to the website.

maybe a way to user input these in the plugin?

[pspzockerscene]

Originally Posted by michael88

i always use a VPN so probably trigger bot protection a lot.

I also triggered it immediately and I'm not using a VPN.

Originally Posted by michael88

If the api is failing because of the one call to the website to get these id's, could there be some mechanism where i could supply the id's for the profile, and the plugin could get the id's from there instead of the call to the website.

If you want I could implement something like this so you could supply the IDs inside the URLs you're adding e.g.:
tiktok.com/@someUsername#?userid=123456

[pspzockerscene]

I just noticed that tiktok already has a user profile URL-format which contains the userID and our plugin also already supports this and will already avoid the website completely if you are adding such URLs while profile crawl mode is set to API.
Example:
tiktok.com/share/user/7031980866356495365