JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 22.09.2018, 15:23
davis89 davis89 is offline
JD Alpha
 
Join Date: Sep 2018
Posts: 23
Lightbulb Youtube channel grabber order?

Hello

Is there any option to configure JDownloader so that I can tell him of either start to grab videos from given youtube channel from the beginning or the end?
Sometimes I just need few(e.g. +50) the newest videos and grabber seems to collect the videos from the oldest and when channel have a lot of videos it takes too much time.
Reply With Quote
  #2  
Old 23.09.2018, 03:21
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

There are no user options in respects to adding playlists or channels. It's been designed to add from the first video to the newest. Majority of users want parsing in order based on playlist entry (or upload date based on channels lists) typically want first video in the playlist vs last. Order is important also for filename tagging for instance on playlist (number), else the numbering would be incorrect.

If you are adding / monitoring channels for new videos, you can use rss feeds and add them automatically. Find the channel id in source code of said user. Then use
_**External links are only visible to Support Staff**
then you wont need to use channel/playlists and jd to parse.


raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 23.09.2018 at 03:38.
Reply With Quote
  #3  
Old 23.09.2018, 06:22
davis89 davis89 is offline
JD Alpha
 
Join Date: Sep 2018
Posts: 23
Default

Yes, I was just wondering how the script works, so that maybe it won't be hard to add just one check box - "add reversed" or sth.

Thanks for the reply.
Reply With Quote
  #4  
Old 24.09.2018, 14:56
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Can you please provide some example links? Then I can check if it would be possible to add such a plugin option to parse playlist either from end or beginning
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 27.09.2018, 07:14
davis89 davis89 is offline
JD Alpha
 
Join Date: Sep 2018
Posts: 23
Default

Sorry for the late reply @Jiaz and thanks for concern
Just take any youtube channel, for example:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

I also was thinking that adding to adres "?flow=grid&view=0&sort=dd" can do the trick, but nope.
Reply With Quote
  #6  
Old 27.09.2018, 11:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

I've fixed plugin. It should now crawl from newest to oldest. Wait for next plugin update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 28.09.2018, 19:04
user135711 user135711 is offline
DSL Light User
 
Join Date: Jul 2018
Posts: 34
Default

Quote:
Originally Posted by Jiaz View Post
I've fixed plugin. It should now crawl from newest to oldest. Wait for next plugin update
Someone should some day add the ability to parse channel playlists and download them as folders. I would pay $5 for a premium "feature" that did that. You could probably make a million bucks if you implemented it and everyone bought it.
Reply With Quote
  #8  
Old 28.09.2018, 19:06
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@user135711: I'm sorry but I don't understand? what feature are you missing?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 30.09.2018, 18:18
user135711 user135711 is offline
DSL Light User
 
Join Date: Jul 2018
Posts: 34
Default

@Jiaz, I badly want to be able to add a youtube channel to the link crawler, and then have the packagizer add all the playlists of the channel into a separate folder each and any videos in no playlist into another folder. It would work like the linkgrabber filter but in reverse. All links containing "&list=" with the same list would be packagized into a folder. I made a request on the feature request but never got a reply so I'm manually sorting channels very slowly into playlists.
Reply With Quote
  #10  
Old 01.10.2018, 17:26
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Let's continue in your other thread, thanks
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 13.12.2018, 10:14
InstallGentoo
Guest
 
Posts: n/a
Default

Sorry for bumping an old thread, wasn't sure if it would be better to make my own.
I don't understand why the crawling was changed so easily. I think what raztoki said is true. For me at least now the youtube plugin became less useful, the number tagging even works in reverse. I would appreciate if just an option is added, so we can choose if to do it from oldest to newest or newest to oldest.
Thank you.
Reply With Quote
  #12  
Old 14.12.2018, 08:10
heyfernand heyfernand is offline
JD Alpha
 
Join Date: Aug 2017
Posts: 22
Default

yeah. and I would like to parse the videos from a channel using the sorting order provided by youtube itself, they currently provide three methods so it's just a matter of adapting the code to theirs

Code:
newest to oldest (DEFAULT) - youtubecom/channel/CHANNELNAME/videos?flow=grid&view=0&sort=dd
oldest to newest - youtubecom/channel/CHANNELNAME/videos?flow=grid&view=0&sort=da
most popular (THE ONE I WANT TO USE) - youtubecom/channel/CHANNELNAME/videos?flow=grid&view=0&sort=p
notice the changing of letters at the very end of the link, the rest is the same. can this be done?
Reply With Quote
  #13  
Old 07.07.2019, 06:47
davis89 davis89 is offline
JD Alpha
 
Join Date: Sep 2018
Posts: 23
Default Youtube channel grabber order?

Quote:
Originally Posted by Jiaz View Post
I've fixed plugin. It should now crawl from newest to oldest. Wait for next plugin update
It's been already quite a while and I was wondering if this statement was fulfilled/is still working?
Reply With Quote
  #14  
Old 08.07.2019, 12:26
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@davis: what exactly do you mean? what doesn't work?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 09.07.2019, 04:23
davis89 davis89 is offline
JD Alpha
 
Join Date: Sep 2018
Posts: 23
Default

Quote:
Originally Posted by Jiaz View Post
@davis: what exactly do you mean? what doesn't work?
Thanks for concern.
TL:DR - Is there a way to solve this - to continue immediately link(yt playlist) grabbing from where it stopped last time?
I've added some youtube channel/playlist to grabber and I interrupted the grabbing in the middle.
If I want to complete grabbing the links the grabber would start to search for the links from the beginning of the list of videos, meaning it will take time when he comes to the middle and start adding the rest.
Changing the video list order, doesn't change anything.
Reply With Quote
  #16  
Old 09.07.2019, 19:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@davis89: That's not *that easy* because the linkcrawling process is heavily multithreaded. when you interrupt/abort the process, the yt plugin doesn't know if the found videos already are processed or also interrupted/aborted. because of that it's not easy to determine where to resume because the plugin can't know what got processed/added and what not.
the plugin doesn't support different sorting
for your use case I would suggest to use tool/create list of all videos from playlist and then add this list in chunks, so you'll know that one list is finished and then continue with next one
__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 10.07.2019, 04:26
davis89 davis89 is offline
JD Alpha
 
Join Date: Sep 2018
Posts: 23
Default

Yes, I get that point. I was just wondering if there could be a half fix to that, meaning that I could just start grabbing the links from the end of the playlist the second time and just again cancel it in the middle so that I would have complete playlist that way.
Reply With Quote
  #18  
Old 10.07.2019, 13:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Hmm, but that's not a real solution because when you crawl from the end and also abort, then you still might have some left in the middle. Maybe sort of plugin settings to tell plugin to check linkgrabber list for entry first and then skip it. so the crawler will first check linkgrabber if there is already the video in list and skip repeated processing of the video. that way the order of the list doesn't matter.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #19  
Old 11.07.2019, 03:11
davis89 davis89 is offline
JD Alpha
 
Join Date: Sep 2018
Posts: 23
Default

Quote:
Originally Posted by Jiaz View Post
Hmm, but that's not a real solution because when you crawl from the end and also abort, then you still might have some left in the middle.
Still it seems much better deal, while the normal way I have to wait sometimes even hour+ for the grabber to came to the point where I've ended and also I am losing a lot of internet data which I have limited here, so it's also money.
Reply With Quote
  #20  
Old 11.07.2019, 13:42
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

It's much better deal and uses way less data. just starting crawling from other end of the list doesn't reduce data amount as for each video, the whole parsing process has to run before JD detects that the video is already in list. with this optimization in mind this will speed up and reduce data. I would suggest to wait for feature to be available and just test it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #21  
Old 11.07.2019, 14:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,913
Default

He was talking about long waittimes as well.

Doesn't disabling the linkcheck/filesize check also speed up the process of adding YT URLs or is YT an exception?

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #22  
Old 11.07.2019, 15:09
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Quote:
Originally Posted by pspzockerscene View Post
Doesn't disabling the linkcheck/filesize check also speed up the process of adding YT URLs or is YT an exception?
-psp-
doesn't speedup the process that much because still several requests for each link + ressource intensive processing
__________________
JD-Dev & Server-Admin
Reply With Quote
  #23  
Old 11.07.2019, 15:36
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

wait for next update and enable Settings-Advanced Settings-Youtube.crawldupecheckenabled
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 16:21.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.