JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 05.10.2021, 06:00
Snookie Snookie is offline
JD Adviser
 
Join Date: Jun 2015
Posts: 101
Talking request for iwara.tv plugin improvement

when copying an iwara channel link is it possible to make iwara plugin capture all videos in that channel with correct videos filenames ?
Reply With Quote
  #2  
Old 05.10.2021, 11:50
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,084
Default

What exactly do you mean by *channel*? can you provide example links?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 05.10.2021, 16:14
Snookie Snookie is offline
JD Adviser
 
Join Date: Jun 2015
Posts: 101
Default

this is a channel **External links are only visible to Support Staff****External links are only visible to Support Staff**

on uploaded videos if you click on "see all" it will take you to that channel all videos
**External links are only visible to Support Staff****External links are only visible to Support Staff**

is it possible when copying one of those two urls jd2 grab all videos that channel has ?
Reply With Quote
  #4  
Old 05.10.2021, 16:20
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 69,719
Default

Added crawler for user profiles.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?


-----------------------------------
On Vacation / Im Urlaub
Start: 2023-12-09
End: TBA
Reply With Quote
  #5  
Old 05.10.2021, 17:27
Snookie Snookie is offline
JD Adviser
 
Join Date: Jun 2015
Posts: 101
Default

also when copying a video link the filename starts with the name of channel then the url of the video and at last the actual video title

is it possible to make it grab the actual video title only as it's provided below the video ?
Reply With Quote
  #6  
Old 05.10.2021, 17:31
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 69,719
Default

You can simply auto-alter filenames using a simple Packagizer rule.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?


-----------------------------------
On Vacation / Im Urlaub
Start: 2023-12-09
End: TBA
Reply With Quote
  #7  
Old 05.10.2021, 19:14
Snookie Snookie is offline
JD Adviser
 
Join Date: Jun 2015
Posts: 101
Default

it's hard to do.. how can i remove video uploader name and last part of url and only keep the actual filename, please help me do this

i tested with this link
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Last edited by Jiaz; 05.10.2021 at 19:25.
Reply With Quote
  #8  
Old 05.10.2021, 19:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,084
Default

Filename - contains - .+_(.+\..+)$
enable regex
Downloadurl - contains - iwara.tv
then set
Filename - <jd:orgfilename:1>
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 05.10.2021, 19:41
Snookie Snookie is offline
JD Adviser
 
Join Date: Jun 2015
Posts: 101
Default

that worked, thank you so much.
Reply With Quote
  #10  
Old 05.10.2021, 20:00
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,084
Default

You're welcome
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 08.10.2021, 02:47
Snookie Snookie is offline
JD Adviser
 
Join Date: Jun 2015
Posts: 101
Default

hey thank you so much for implementing the crawler in today's update

there are some minor things that needs fixing :

i tested with this link :
**External links are only visible to Support Staff****External links are only visible to Support Staff**

the folder that was captured by jd2 had the name in the url instead of the name inside the page, as you can see in the picture below the name in the url is different and not correct

also inside the folder grabbed by the crawler there is 2 files that aren't videos uploaded by uploader and are not supposed to be grabbed ( Plugin Defect!TSOF 402 & thumbnails.mp4 ) as you can see in the picture





there is another thing that i noticed with this particular profile

**External links are only visible to Support Staff****External links are only visible to Support Staff**

the videos titles captured aren't correct
fyi my jd2 font name is Dialog
Reply With Quote
  #12  
Old 08.10.2021, 11:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,084
Default

Wait for next plugin update
Can you provide screenshot/link for *wrong titles*? I can't reproduce here
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 08.10.2021, 18:00
Snookie Snookie is offline
JD Adviser
 
Join Date: Jun 2015
Posts: 101
Default

after today plugin update profile name was corrected but the captured videos names are short and incorrect

this above applies only for this particular user..

as other users and the one i reported yesterday are now showing correct username and correct videos titles

here is the example i reported yesterday and today it was corrected with the plugin update

Last edited by Jiaz; 08.10.2021 at 18:48.
Reply With Quote
  #14  
Old 08.10.2021, 18:48
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,084
Default

check again with next update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 09.10.2021, 18:26
Snookie Snookie is offline
JD Adviser
 
Join Date: Jun 2015
Posts: 101
Default

last issue wasn't fixed with yesterday 2 consequential updates, in fact the updates changed nothing.

and unfortunately i discovered a new issue :
the crawler doesn't always capture the uploaded videos if the link copied isn't the /videos of the user profile URL, some profiles just doesn't have a /videos URL section like the rest of the profiles, here is an example:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

so the crawler need to grab the videos without being in /videos section of the user profile URL

also there are some user profiles that can be viewed only if you are in Japan, but if you add /videos to their URL you can actually access the videos like this example :
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

the solution here is to make the crawler grab the videos in both ways with and without /videos in the URL thus grabbing the videos if the user have no /videos URL like first example and/or in 2nd example when the user profile is restricted only to japan we add /videos to the URL and copy that link to bypass the restriction.

Last edited by Snookie; 10.10.2021 at 01:08.
Reply With Quote
  #16  
Old 10.10.2021, 17:29
Snookie Snookie is offline
JD Adviser
 
Join Date: Jun 2015
Posts: 101
Default

please don't give up on this.
Reply With Quote
  #17  
Old 11.10.2021, 16:39
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 69,719
Default

Hi again,
I've updated the crawler to take the URLs as they come so if you add e.g.:
Code:
bla.iwara.tv/users/xyz
...exactly this will be used.
If you want it to use the one ending with "/videos", just add that.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?


-----------------------------------
On Vacation / Im Urlaub
Start: 2023-12-09
End: TBA
Reply With Quote
  #18  
Old 11.10.2021, 19:34
Snookie Snookie is offline
JD Adviser
 
Join Date: Jun 2015
Posts: 101
Default

thank you so much, i really appreciate you making our life easier.
Reply With Quote
  #19  
Old 11.10.2021, 19:38
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,084
Default

Quote:
Originally Posted by Snookie View Post
please don't give up on this.
Weekend

Quote:
Originally Posted by Snookie View Post
last issue wasn't fixed with yesterday 2 consequential updates, in fact the updates changed nothing.
The crawler can only fetch/crawl the names that are available on website. If the name is too long, the full name is not available at crawling but only after actual download.
To fetch full name while crawling, plugin would need to crawl/check each link individually and thus would be much much slower and can easier cause IP block because of too many requests.

EDIT: found a better way to grab name, I guess this solves the issue, wait for plugin update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 11.10.2021, 20:17
Snookie Snookie is offline
JD Adviser
 
Join Date: Jun 2015
Posts: 101
Default

Quote:
Originally Posted by Jiaz View Post
Weekend
sometimes i feel ashamed of how stupid i can be


Quote:
Originally Posted by Jiaz View Post
EDIT: found a better way to grab name, I guess this solves the issue, wait for plugin update
after plugin update :

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 23:41.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.