JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #41  
Old 21.06.2023, 21:42
DukeM's Avatar
DukeM DukeM is offline
JD Adviser
 
Join Date: Sep 2019
Posts: 106
Default

Quote:
Originally Posted by pspzockerscene View Post
While I do believe you I'm unable to reproduce this.
You might want to reach out to the developers of said LinkGopher browser addon and/or open an issue on their github.
Link to their github project:
github.com/az0/linkgopher/
Thanks but I'm not too sure if it's a LinkGopher issue. Although, this happens on both Brave and Firefox for me, it only does affect TikTok it seems. And I feel like it's something with my browsers for some reason (even using Private/Incognito).

Like when you view a profile page and you hover your cursor over a video, you'd get like a link preview on the bottom left of the screen, right? But for me, it just shows **External links are only visible to Support Staff****External links are only visible to Support Staff**. That's why I initially thought perhaps TikTok changed how their website works but you disproved that on your reply earlier.

So I've been trying to figure it out intermittently the whole day because I can't get links again no matter how many refreshes I do. But very weirdly, whenever I try to get links from the example profile you used earlier, I don't encounter any problems.

But this isn't your issue to fix. Haha sorry, just wanted to let it out a bit.

Quote:
Originally Posted by pspzockerscene View Post
Depends on your settings.
In most cases you should at least get a dummy item as feedback if a crawler fails.
For tiktok, at this moment, it depends on your plugin settings:
- Profile crawl mode API: No feedback
- Profile crawl mode website: Crawler should find up to 30 items (first page of videos on tiktok website)
I see, thanks. I totally blanked out that there's a Bugtracker status on the thread but thanks for clarifying the crawl modes though.

Anyway, thank you so much again and again!
Reply With Quote
  #42  
Old 22.06.2023, 11:25
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 69,719
Default

Quote:
Originally Posted by DukeM View Post
Thanks but I'm not too sure if it's a LinkGopher issue. Although, this happens on both Brave and Firefox for me, it only does affect TikTok it seems. And I feel like it's something with my browsers for some reason (even using Private/Incognito).
As said I'm on Chrome and there it is working.
It might also be a "browser related issue" but either way I think it would be a good idea to report it to the developers of that addon as it is clearly not working as intended.
I highly recommend creating a gituhb.com account for that. There is so many more nice open source with public repositories for which you can then also report bugs / request new features or even contribute your own code

Quote:
Originally Posted by DukeM View Post
Like when you view a profile page and you hover your cursor over a video, you'd get like a link preview on the bottom left of the screen, right?
I can confirm that but looks like this is only applied for the first few items. Scroll down multiple times so the pagination will load more items and those will show the links, at least here (this time tested under Chrome and Firefox).

Also by looking deeper into this, I just found out that indeed LinkGopher is unable to find any items of the first page of a tiktok profile but it does find all items after that.
Indeed this would then not be a LinkGopher bug as it really can't see those links as it looks like they were intentionally "hidden" by tiktok.

So if I was to crawl a tiktok user profile now, this is how I would do it:
1. Add profile link to JD so JDcan grab all items of the first page.
2. Scroll all the way down in browser, grab all the remaining links via LinkGopher and add them to JD.

Good luck.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?


-----------------------------------
On Vacation / Im Urlaub
Start: 2023-12-09
End: TBA
Reply With Quote
  #43  
Old 24.06.2023, 00:30
An0nym0usAn0n An0nym0usAn0n is offline
Modem User
 
Join Date: Nov 2022
Posts: 2
Smile

Quote:
Originally Posted by goldensun87 View Post
I made my own guide detailing how I download TikTok profiles. It doesn't have visual steps like JD guides do, but I hope it will still be helpful.

Script: pastebin.com/ZbMUf5kS (copy code into notepad, save extension as .PS1 to convert to powershell)

If the link is not visible, it is a pastebin link, and the link ID is ZbMUf5kS.
This was very helpful, thank you!!! I've made my own home brew solution to automating steps 7+ in your guide.

TikTok Video Downloader
Credit: Made with the help of ChatGPT lol, it's not perfect, and has gone through a few iterations, let me know if you have any issues with it and I'll see if I can tweak it.

Script Location: pastebin.com/QG78NJ1C

This PowerShell script enables you to download TikTok videos from a specific user profile. It uses the yt-dlp command-line tool to download videos, and PowerShell to automate the process, and downloads all the videos into the username's folder.

The following are prerequisites for using the script:

Prerequisites

1: Operating System: Windows. The script is a PowerShell script, which is a Windows-based scripting language.
2: yt-dlp: This is a command-line tool that is used to download videos from YouTube.com and other video platforms. You can get it from the yt-dlp GitHub page. Download the Windows executable and place it in the C:\URLscrape directory on your machine.
3: Browser Extension: LinkClump ONLY (not open multiple URL's, logic built into script). This add-on is used to collect the URLs of the TikTok videos to download. Follow the steps in the provided JD instructions to configure this extension and collect the URLs. (Note: I use firefox).
4: Text Editor: Notepad++ or similar (I use Sublime with Powershell syntax). This is used to view and edit the PowerShell script if you need to.

How to Use

1: Follow the provided instructions in the original paste bin (steps 1-6, pastebin.com/ZbMUf5kS, credit: goldensun87) to use LinkClump to collect the URLs of the TikTok videos you want to download. Save the copied information in a text file named URLinput.txt and place it in the C:\URLscrape directory (copied information does not need filtered via Open Multiple URL's browser extension, script will handle).
2: Run the PowerShell script (copy code from green pastebin link above into a notepad file, and save file extension as ".PS1". If it's not already running with administrative privileges, it will relaunch itself with admin rights.
3: When prompted, enter the username of the TikTok profile you want to download videos from. The username should include the '@' symbol (e.g., @tiktokusername).
4: The script will filter the URLs in URLinput.txt to only include those that contain the specified username. It will save the filtered URLs in a new text file named output_{username}_{date}.txt in the C:\URLscrape directory, where {username} is the username you entered and {date} is the current date.
6: Each video file will be named after its title on TikTok in a folder named after the username provided. Default folder creation location is C:\URLscrape.

Notes

1: The script assumes that all required files (URLinput.txt, yt-dlp.exe) and directories are in the C:\URLscrape directory. If you want to use a different directory, you will need to edit the script to change the paths / file names. You can name the C:\ directory whatever you want, same with the input file, you just need to update the script to match.
2: The script does not provide any progress indication (other than window beep when done) while yt-dlp is downloading videos. It may appear to be doing nothing, but as long as it hasn't exited, it's likely that yt-dlp is still downloading videos.
3: The script will pause after downloading the videos to keep the PowerShell window open. Press any key to exit.
4: I've not tested trying to download multiple profiles at once via the script.

Last edited by An0nym0usAn0n; 24.06.2023 at 22:40. Reason: Included logic for ALL URL filtering (browser extension no longer necessary)
Reply With Quote
  #44  
Old 26.06.2023, 17:17
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 69,719
Default

@An0nym0usAn0n
Just a quick sidenote:
You're using yt-dlp which does already support downloading tiktok profiles out of the box but that feature is broken at this moment.
Related ticket:
github.com/yt-dlp/yt-dlp/issues/3776

If it was working, all you'd need to do is to run a single yt-dlp command
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?


-----------------------------------
On Vacation / Im Urlaub
Start: 2023-12-09
End: TBA
Reply With Quote
  #45  
Old 13.10.2023, 14:57
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 69,719
Default

Added support for crawling audio/photo content in profile crawler API mode and fixed this bug.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?


-----------------------------------
On Vacation / Im Urlaub
Start: 2023-12-09
End: TBA
Reply With Quote
  #46  
Old 25.10.2023, 16:07
wow843658435 wow843658435 is offline
Junior Loader
 
Join Date: Aug 2018
Posts: 12
Angry [Use plugin setting media crawl mode 'Website'] tiktok - ''content offline''

when i try to add links to some tiktok videos, this program says ''content offline'' and can't download anything. but when i paste the video link to browser, the video is working properly and of course it is not ''OFFLINE''!
wtf?
Reply With Quote
  #47  
Old 25.10.2023, 16:11
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 69,719
Default

Hi,
please provide example URLs, a debug- log and a screenshot of your tiktok.com plugin settings.

Please post your log-ID here
If your bugreport is about a specific website which JD supports via plugin, please also provide example URLs which can be used to reproduce the issue you are having.

Bitte poste deine Log-ID hier.
Falls dein gemeldetes Problem ein Problem mit einer Webseite ist, die per Plugin unterstützt wird, stelle bitte zusätzlich Testlinks zur Verfügung, mit denen sich dein Problem nachstellen lässt.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?


-----------------------------------
On Vacation / Im Urlaub
Start: 2023-12-09
End: TBA

Last edited by pspzockerscene; 25.10.2023 at 16:13. Reason: Fixed typo
Reply With Quote
  #48  
Old 25.10.2023, 23:56
wow843658435 wow843658435 is offline
Junior Loader
 
Join Date: Aug 2018
Posts: 12
Default

25.10.23 23.37.50 <--> 25.10.23 23.45.55 jdlog://0268311370661/

for example:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
this video can be downloaded
**External links are only visible to Support Staff****External links are only visible to Support Staff**
this video ''content offline'' but can be viewed in browser!
Reply With Quote
  #49  
Old 26.10.2023, 11:45
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 69,719
Default

Go to Settings -> Plugins -> tiktok.com -> Media crawl mode -> Website
-> Then delete all offline items and re-add them.

This is not a JDownloader bug. Some items are simply offline according to the tiktok API while they're working fine via website.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?


-----------------------------------
On Vacation / Im Urlaub
Start: 2023-12-09
End: TBA
Reply With Quote
  #50  
Old 28.10.2023, 13:19
wow843658435 wow843658435 is offline
Junior Loader
 
Join Date: Aug 2018
Posts: 12
Default

Quote:
Originally Posted by pspzockerscene View Post
Go to Settings -> Plugins -> tiktok.com -> Media crawl mode -> Website
with this settings everything is ''offline'' now.
Reply With Quote
  #51  
Old 30.10.2023, 13:09
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 69,719
Default

Quote:
Originally Posted by wow843658435 View Post
with this settings everything is ''offline'' now.
Most likely the website mode fails for you due to tiktoks anti bot protection. In this case there is nothing you can do.

Please provide a fresh debug log for me to check this.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?


-----------------------------------
On Vacation / Im Urlaub
Start: 2023-12-09
End: TBA
Reply With Quote
  #52  
Old 30.10.2023, 20:51
Snookie Snookie is offline
JD Adviser
 
Join Date: Jun 2015
Posts: 101
Default

plugin defect :/
Reply With Quote
  #53  
Old 31.10.2023, 10:42
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 69,719
Default

Same as I wrote before applies to you:

Please post your log-ID here
If your bugreport is about a specific website which JD supports via plugin, please also provide example URLs which can be used to reproduce the issue you are having.

Bitte poste deine Log-ID hier.
Falls dein gemeldetes Problem ein Problem mit einer Webseite ist, die per Plugin unterstützt wird, stelle bitte zusätzlich Testlinks zur Verfügung, mit denen sich dein Problem nachstellen lässt.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?


-----------------------------------
On Vacation / Im Urlaub
Start: 2023-12-09
End: TBA
Reply With Quote
  #54  
Old 31.10.2023, 13:27
Snookie Snookie is offline
JD Adviser
 
Join Date: Jun 2015
Posts: 101
Default

31.10.23 12.12.09 <--> 31.10.23 12.14.30 jdlog://7178311370661/

the issue is with profile crawler

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #55  
Old 31.10.2023, 13:49
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 69,719
Default

Merged tiktok profile crawler threads.

Looks like the tiktok profile API is either broken or the request we are using is outdated.
With a bit of luck this is only a temporary serverside issue!

In the meanwhile please use one of the following workarounds to add all videos of a profile:
As always we will not provide any ETA regarding a fix.

Ticket:
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?


-----------------------------------
On Vacation / Im Urlaub
Start: 2023-12-09
End: TBA
Reply With Quote
  #56  
Old 31.10.2023, 23:19
djones djones is offline
Baby Loader
 
Join Date: Nov 2022
Posts: 7
Default

Quote:
Originally Posted by pspzockerscene View Post
Looks like the tiktok profile API is either broken or the request we are using is outdated. With a bit of luck this is only a temporary serverside issue!
^ Thank you, had just come to post about this. Something did seem to change on the tiktok end a couple days ago.

Last edited by djones; 31.10.2023 at 23:22.
Reply With Quote
  #57  
Old 01.11.2023, 10:11
goldensun87 goldensun87 is offline
Linkgrabbing Monster
 
Join Date: Feb 2012
Posts: 88
Default

Quote:
Originally Posted by An0nym0usAn0n View Post
This was very helpful, thank you!!! I've made my own home brew solution to automating steps 7+ in your guide.
You're welcome, I'm glad my guide was helpful for you . I may or may not look into your new program/script, but I hope it will be very helpful for other users here.

On another note, it took me a while, but I realized that many, but not all, profile video links, have duplicates that get copied by linkclump. These duplicates do not get filtered out by the "username/video" find filter, and the reason the duplicates are there, is because the video link is embedded in both the video thumbnail as well as the title hyperlink.

This may or may not be relevant for future reference, I just wanted to share that info with everyone.
Reply With Quote
  #58  
Old 06.11.2023, 07:31
Sal Sal is offline
Modem User
 
Join Date: Nov 2023
Posts: 1
Default Tiktok Profile Crawl Doesn't Work

Adding any TikTok profile link into linkgrabber does not retrieve any videos/pictures. (I'm not IP banned or anything).
example link: **External links are only visible to Support Staff****External links are only visible to Support Staff**

None of the plugin options "enable fast linkcheck," "profile crawl mode: website/API," "profile crawler website mode," and "profile crawler: max items" fix this.

Issue started with JD version #48254

Adding individual post links works fine and JD is able to grab the vid/pic. example link: **External links are only visible to Support Staff****External links are only visible to Support Staff**

Last edited by Sal; 06.11.2023 at 07:38.
Reply With Quote
  #59  
Old 07.11.2023, 11:01
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 69,719
Default

Merged same topic threads.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?


-----------------------------------
On Vacation / Im Urlaub
Start: 2023-12-09
End: TBA
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 01:44.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.