JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 04.08.2022, 20:29
goldensun87 goldensun87 is offline
Linkgrabbing Monster
 
Join Date: Feb 2012
Posts: 81
Default [API changes] TikTok.com Profile Crawler Not Working

Here is an example link.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Both my download and profile crawl modes are set to API, and I am logged into my account.
Reply With Quote
  #2  
Old 05.08.2022, 00:45
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,582
Default

Indeed I can reproduce that issue but I'm unsure if it's just an issue with their API or if we're using that API in a wrong way.
Fact is that it worked before and now it fails although we haven't pushed any updates for ~4 weeks.
Other open source projects using that API seem to fail too...

Here are your options for now:
1. Use website mode instead but then JD will only find the first ~30 items.
2. Use this method as a workaround.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3  
Old 05.08.2022, 15:24
lalaupoa lalaupoa is offline
Ultra Loader
 
Join Date: Oct 2015
Posts: 49
Default

same here
Reply With Quote
  #4  
Old 05.08.2022, 19:03
goldensun87 goldensun87 is offline
Linkgrabbing Monster
 
Join Date: Feb 2012
Posts: 81
Default

Quote:
Originally Posted by pspzockerscene View Post
Indeed I can reproduce that issue but I'm unsure if it's just an issue with their API or if we're using that API in a wrong way.
Fact is that it worked before and now it fails although we haven't pushed any updates for ~4 weeks.
Other open source projects using that API seem to fail too...

Here are your options for now:
1. Use website mode instead but then JD will only find the first ~30 items.
2. Use **External links are only visible to Support Staff**... as a workaround.
Thanks for checking. I hope you guys will be able to resolve this eventually.

Edit: I am aware that you guys have not pushed any updates for almost 4 weeks now. Perhaps that means TikTok has altered how their API functions, and as a result broke JD's TikTok profile crawler?

Last edited by goldensun87; 05.08.2022 at 19:32.
Reply With Quote
  #5  
Old 09.08.2022, 17:16
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,582
Default

Quote:
Originally Posted by goldensun87 View Post
Edit: I am aware that you guys have not pushed any updates for almost 4 weeks now.
We're working on it.

Quote:
Originally Posted by goldensun87 View Post
Perhaps that means TikTok has altered how their API functions, and as a result broke JD's TikTok profile crawler?
No.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #6  
Old 03.09.2022, 12:36
michael88 michael88 is offline
Vacuum Cleaner
 
Join Date: May 2017
Posts: 18
Default

Profile crawler not working, though single links still work fine. Any updates on this?
Reply With Quote
  #7  
Old 03.09.2022, 17:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,582
Default

The API still looks broken and other popular tools such as yt-dlp are failing too.

What you can do:
Settings -> Plugins -> tiktok.com -> Crawler mode -> Website
If it still doesn't find all items, use this way as a workaround.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #8  
Old 04.09.2022, 09:42
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 05.09.2022, 14:31
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,582
Default

I don't see a fix for this happening in the near future.
If you want to download all videos of profiles containing 30 or more items, use this method as a workaround in the meanwhile.

EDIT

One of the other tools for which it is also failing is the popular "yt-dlp" project. You can follow the discussion about this problem here:
github.com/yt-dlp/yt-dlp/issues/3776
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 06.09.2022 at 17:26. Reason: Highlighted important parts of this post and added the keyword "workaround"
Reply With Quote
  #10  
Old 05.09.2022, 17:51
goldensun87 goldensun87 is offline
Linkgrabbing Monster
 
Join Date: Feb 2012
Posts: 81
Default

*Sigh*...Oh well, it was fun while it lasted. Good thing LinkClump and Open Multiple URLs exists.
Reply With Quote
  #11  
Old 06.09.2022, 17:20
lalaupoa lalaupoa is offline
Ultra Loader
 
Join Date: Oct 2015
Posts: 49
Default

:(((((((((((((
I used this function all the time
Reply With Quote
  #12  
Old 06.09.2022, 17:21
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,582
Default

Quote:
Originally Posted by pspzockerscene View Post
I don't see a fix for this happening in the near future.
This doesn't mean that it will never be fixed

Please use the linked workarounds for now.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #13  
Old 10.10.2022, 15:45
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,582
Default

Quick update on this:

Status of JD:
- Profile crawling in API mode does not work anymore
- Profile crawling in website mode works for the first ~30 items

yt-dlp:
Looks like someone forked- and fixed the yt-dlp extractor plugin but that solution is Chromium based.
For now we're not planning to add such libraries to JDownloader which means that our tiktok profile crawler will probably remain broken for the most part for now.

You can find the mentioned fork of that yt-dlp extractor in the github ticket I l've linked in my post #9 in this thread.

How to crawl/download full tiktok profiles using JDownloader and some external helper tools even with the partially broken state of the JD profile crawler:
1. Use this method or any other external tools to find all single links to all videos of a profile.
2. Now add those links to JDownloader and let it download those videos.

Just a little heads up for those who e.g. always want to download all current/new videos of a particular profile:
You could first download all videos of a profile using the methods described at the beginning of this post.
Then set Settings -> Plugins -> tiktok.com -> Profile crawl mode -> Website
Now JDownloader should find the latest ~30 items whenever you're adding a profile URL.
Now you could add that profile URL every day to update your personal library of clips of that profile.
While JDownloader does not support that out of the box, you could also automate this process using an EventScripter script.

The above info is neither a fix nor an easy workaround but it should enable you guys to use some of the tiktok profile crawling capabilities of JD even without a real fix of the crawler issues.

Sidenote:
Videos without watermark are expected to be downloadable again after the release of our next set of CORE-updates, see this thread.

EDIT
That update has been released and watermark-free download is working again (set Download mode to "API" for this to work!).
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 17.11.2022 at 12:19.
Reply With Quote
  #14  
Old 19.10.2022, 06:31
goldensun87 goldensun87 is offline
Linkgrabbing Monster
 
Join Date: Feb 2012
Posts: 81
Default

Thanks for the update. I have devised a way to collect and batch download TikTok profiles using yt-dlp. However, yt-dlp's default filenaming scheme is not favorable. The default seems to be "*video title* [*video ID*].ext". I want my filename scheme for TikTok videos to be, "*uploader* *upload date* [*video ID*].ext".

If someone can show me how to do that, I can write out my method for batch downloading TikTok videos with yt-dlp, and share it here for you guys to add to your list of guides. My guide will not have images, so I will try to make my instructions as clear and as easy to understand as possible.
Reply With Quote
  #15  
Old 19.10.2022, 13:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

@goldensun87: maybe you can find a way to use yt-dlp or other tool to just collect the post URLs and then feed them to JDownloader. For your filename scheme, maybe check
mankier.com/1/yt-dlp#Output_Template
__________________
JD-Dev & Server-Admin
Reply With Quote
  #16  
Old 17.11.2022, 06:30
J.D J.D is offline
Modem User
 
Join Date: Nov 2022
Posts: 3
Default tiktok

hi
I tried to download an entire file, it didn't work
**External links are only visible to Support Staff****External links are only visible to Support Staff**

download mode does not work
website usually with watermark also private videos and
api usually without watermark only public videos
api hd usually without watermark only public videos

Works on website max first 30 items

Also has a problem
CANNOT_CRAWL_MORE_THAN_30_ITEMS_OF_PROFILE_vanesadeabreuu_IN_WEBSITE_PROFILE_CRAWL_MODE

screenshot
Spoiler:


17.11.22 06.14.47 <--> 17.11.22 06.18.02 jdlog://0021311370661/
Attached Images
File Type: png Untitled.png (183.2 KB, 7 views)

Last edited by pspzockerscene; 17.11.2022 at 12:20. Reason: Added SPOILER tag for image
Reply With Quote
  #17  
Old 17.11.2022, 07:37
J.D J.D is offline
Modem User
 
Join Date: Nov 2022
Posts: 3
Default

Here too
Content offline!
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
17.11.22 07.36.09 <--> 17.11.22 07.38.37 jdlog://3021311370661/
Spoiler:
Attached Images
File Type: png Untitled.png (81.3 KB, 9 views)

Last edited by pspzockerscene; 17.11.2022 at 12:21. Reason: Added SPOILER tag for image
Reply With Quote
  #18  
Old 17.11.2022, 12:20
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,582
Default

Merged tiktok profile crawler threads.
Please read the existing posts in this thread especially my update post here.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #19  
Old 18.11.2022, 05:07
J.D J.D is offline
Modem User
 
Join Date: Nov 2022
Posts: 3
Default

pspzockerscene
It does not download the video
It's one video
Look


**External links are only visible to Support Staff****External links are only visible to Support Staff**



Spoiler:
Attached Images
File Type: png Untitled.png (83.8 KB, 9 views)

Last edited by pspzockerscene; 18.11.2022 at 12:01. Reason: Added SPOILER for screenshot image
Reply With Quote
  #20  
Old 18.11.2022, 12:02
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,582
Default

@J.D
Looks like for some reason this particular item is not available via API.
Solution:
1. Delete that item in JD.
2. Go to Settings -> Plugins -> tiktok.com -> Download Mode -> Website
3. Re-add that item to JD.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 22:37.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.