JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 26.05.2022, 00:07
zylaxabi zylaxabi is offline
JD Beta
 
Join Date: Oct 2020
Posts: 57
Default [Can't reproduce] Tiktok Bot/Crawler Protection

Before this most recent update if tiktok's bot/crawler protection kicked in, we could tell that's what happened because we would get a link created titled something like:

"Wrong Captcha!Bot protection active, cannot crawl any items of user"

But with this most recent update nothing shows up at all. So would it be possible to fix that, just so we know why it's not grabbing a tiktok profile link.

Also another thing, while it's great that the profile crawler for tiktok grabs all the videos on a profile, does this not make more requests to tiktok and thus makes it more likely that their bot/crawler protection activates.

Would it be possible to have the option for the profile grabber to grab the thirty newest vids like it was before if thats all we wanted it to do?

Thanks again for everything you guys do.
Reply With Quote
  #2  
Old 26.05.2022, 19:51
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,343
Default

@zylaxabi: please provide a debug log
Enable Settings->Advanced Settings->Log.debugmodeenabled and restart JDownloader, reproduce the issue and then create a log, see https://support.jdownloader.org/Know...d-session-logs and post shown logID here

Quote:
Originally Posted by zylaxabi View Post
Would it be possible to have the option for the profile grabber to grab the thirty newest vids like it was before if thats all we wanted it to do?
Before there was no profile crawler and JDownloader did only support the first *page* and the results on it but could not look for more *pages/items*. See Settings->Plugins->tiktok.com

But I agree a setting to limit the number of results would be nice to have
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 26.05.2022 at 19:54.
Reply With Quote
  #3  
Old 31.05.2022, 01:31
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,143
Default

Hmm at this moment I don't see a need of that because:
  1. The website can only crawl the "first page" anyways which is usually max. 30 items.
  2. Our plugin is already supposed to recognize whenever tiktok.com bot-clock mechanism kicks in. My guess is that this simply fails...
  3. Using API mode (Profile crawling over the tiktok API), I wasn't able to ever trigger any anti bot measure.

Please provide a log so based on this we can decide whether fixing the "bot block recognization" is all that needs to be done or adding a limit as a feature is a good idea.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #4  
Old 31.05.2022, 17:24
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,143
Default

I've added said setting anways but either way I need a log to check/fix that "bot limit recognization".

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #5  
Old 01.06.2022, 21:13
zylaxabi zylaxabi is offline
JD Beta
 
Join Date: Oct 2020
Posts: 57
Default

i haven't been able to recreate it, but if I can i'll do a log.
Thanks for everything
Reply With Quote
  #6  
Old 01.06.2022, 21:23
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,143
Default

I'll mark this as solved for now then.

As explained, if you wish to you can now limit the number of results you get when adding a profile though I doubt tat this is of any help regarding bot-block prevention.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 13:49.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.