JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 23.09.2023, 15:54
michael88 michael88 is offline
Ultra Loader
 
Join Date: May 2017
Posts: 45
Default [Tiktok bot protection] Tiktok: profile grabber broken

Tiktok profile crawler is not working, with inconsistent responses for different profiles.

1. Website mode is quiet inconsistent with it working some times and other times a blank response with not even an offline link added.
2. Api mode works sometimes and if not then consistently fails with 'Content Offline'.

eg
1.**External links are only visible to Support Staff****External links are only visible to Support Staff**
- works with api, not with website mode.
2.**External links are only visible to Support Staff****External links are only visible to Support Staff**
3.**External links are only visible to Support Staff****External links are only visible to Support Staff**
- both dont work with either website or api mode.

The individual links work in both website and api mode, it's just the crawler that's broken.
Reply With Quote
  #2  
Old 25.09.2023, 12:05
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,683
Default

Hi,

1. Fixed bot-protection-check sometimes failing.

2. The API may also fail because: Even ion API mode, website needs to be used one time to find the internal user-ID.
In this case this failed due to their bot-protection.

Some time ago I made a post where I explained how to manually collect all video-links of a profile if our crawler fails for whatever reason:
https://board.jdownloader.org/showpo...4&postcount=13
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #3  
Old 25.09.2023, 13:37
michael88 michael88 is offline
Ultra Loader
 
Join Date: May 2017
Posts: 45
Default

Hi,
Thanks for the explanation for the inconsistency, i always use a VPN so probably trigger bot protection a lot.

The internal user-Id for eg :
**External links are only visible to Support Staff**...
is i think 107955.

for **External links are only visible to Support Staff**...
is 7081331206307398662.

I already have the internal id's for all the profiles that i want to scrape.

If the api is failing because of the one call to the website to get these id's, could there be some mechanism where i could supply the id's for the profile, and the plugin could get the id's from there instead of the call to the website.

maybe a way to user input these in the plugin?
Reply With Quote
  #4  
Old 25.09.2023, 13:52
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,683
Default

Quote:
Originally Posted by michael88 View Post
i always use a VPN so probably trigger bot protection a lot.
I also triggered it immediately and I'm not using a VPN.

Quote:
Originally Posted by michael88 View Post
If the api is failing because of the one call to the website to get these id's, could there be some mechanism where i could supply the id's for the profile, and the plugin could get the id's from there instead of the call to the website.
If you want I could implement something like this so you could supply the IDs inside the URLs you're adding e.g.:
tiktok.com/@someUsername#?userid=123456
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #5  
Old 25.09.2023, 14:04
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,683
Default

I just noticed that tiktok already has a user profile URL-format which contains the userID and our plugin also already supports this and will already avoid the website completely if you are adding such URLs while profile crawl mode is set to API.
Example:
tiktok.com/share/user/7031980866356495365
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 04:21.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.