[Solved] request for iwara.tv plugin improvement

[pspzockerscene]

Website is back online.
They indeed have a space in their json of "title" so that one was not my fault

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[lafhej]

Thanks, I think everything is working now.

[pspzockerscene]

Thanks for your feedback.

[Luzashix]

Seems like the Crawler broke again. It is not getting any of the links in a profile crawl.

[Snookie]

Originally Posted by Luzashix

Seems like the Crawler broke again. It is not getting any of the links in a profile crawl.

it works fine for me.

[Snookie]

hi guys, is there a way to exclude external youtube links from being grabbed/added into the linkgrabber and make jd2 catch only the ecchi links that are hosted on iwara ?

[notice]

Originally Posted by Luzashix

Seems like the Crawler broke again. It is not getting any of the links in a profile crawl.

Example Links? Debug logs? see https://support.jdownloader.org/Know...d-session-logs

[notice]

Originally Posted by Snookie

hi guys, is there a way to exclude external youtube links from being grabbed/added into the linkgrabber and make jd2 catch only the ecchi links that are hosted on iwara ?

Can you please provide example links to reproduce/work on this?

[Snookie]

some MMD creators upload their non sexual videos to youtube then share those to their iwara profile so when jdownloader crowl, it captures those youtube videos as well



here is an example

**External links are only visible to Support Staff****External links are only visible to Support Staff**

if their is a way to exclude those youtube videos that would be great.


or maybe when it's crowling for an entire profile videos it's simply capturing the youtube link in the video discription


video example from creator above that has a youtube link in the description :

**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Settings -> Advanced Settings -> GeneralSettings.crawlercrawlerpluginblacklist ->

["youtube.com"]

[Snookie]

i still want to have the ability to download from youtube, this solution solve the iwara problem but deprive me from using jd2 to download YT videos.
i knew it was a long shot and its not a big bother anyways, ill just delete the yt links and keep only the iwara links before adding to download list..

and again thank you so much for creating this amazing plugin and for responding so quickly whenever we ask for your help.

[pspzockerscene]

I've added a plugin setting for you for the next set of CORE-updates.
Screenshot:

That setting is only for the profile crawler so let me know if you also want to have it for single added iwara.tv video links.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Snookie]

when checking for hundreds of profiles at once those youtube videos are an issue because they add up.. so skipping them is the best solution and there is no need for single added video links.
i’m grateful beyond words.

[Snookie]

hi guys, is it possible to make the plugin grab external mega links in videos description just like you made it ignore the youtube links ?

here is an example with external mega link

**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Added setting for that.

Keep in mind that this will override/disable fast linkcheck for profile crawling because the crawler needs to look into that videos' metadata to find- and parse such links.
That new setting will be disabled by default.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Snookie]

i'm really sorry there is 2 things that just now came to my mind that i should have mentioned in my last request..

first thing is sometime video creators write a lot of lines in video description describing their work and they put the external link after all those lines so to see the actual external link you have to click on Show More button, here is an example :

**External links are only visible to Support Staff****External links are only visible to Support Staff**

i don't know if that going to be a problem for the crawler so i just felt like mentioning it

second thing is sometimes creators separate the mega link into 2 halves to make it hard for copyrighted models owners finding their links via search engines and copyright reporting those links.

also not all creators use mega as an external links.. others use google drive and rule34video..

here is an example of a creator using external google drive link :

**External links are only visible to Support Staff****External links are only visible to Support Staff**

i came across creators using rule34video before, i just can't remember any to provide an example right now.. but basically when a creator create a video of a taboo genre that is not allowed on mega or google drive and they know that it will be immediately taken down and their account banned if reported they instead will post their higher resolution external link on rule34video instead of google drive or mega because rule34video has no forbidden genre and their video wont be taken down.

also when crawling for entire profile videos it's preferable to have all the internally hosted iwara videos and externally hosted videos/zipped files all be inside one package in linkgrabber.

i'm sorry if i'm asking for too much, i'm just trying to have the best possible outcome from this plugin.. because otherwise manually checking every video description for external links is not feasible due to the ridiculous amount of time it will take.

[pspzockerscene]

Originally Posted by Snookie

first thing is sometime video creators write a lot of lines in video description describing their work and they put the external link after all those lines so to see the actual external link you have to click on Show More button, here is an example :

The button is only cosmetic. All of the description gets loaded right away.
You can check this by looking into their API to find the metadata of each video.
Random example video:

iwara.tv/video/k2ayoueezfkx6gvq/defeat-of-irybelda

API link:

api.iwara.tv/video/k2ayoueezfkx6gvq

Originally Posted by Snookie

second thing is sometimes creators separate the mega link into 2 halves to make it hard for copyrighted models owners finding their links via search engines and copyright reporting those links.

I doubt that this is gonna help.
Ultimately, if they do this in a way that it is easy for us to automate to build the complete links, anyone with some technical knowledge will be able to do the same.
Also, the other way: If it is not possible to easily automatically build the original links, I won't be able to do it either and the only thing you can do is to download the description as .txt file and manually check for links later. Is there a demand to download the description text?

Originally Posted by Snookie

here is an example of a creator using external google drive link :

Works fine here in my dev version.
By the way you can get the dev version and test changes earlier, see:
https://support.jdownloader.org/Know...up-ide-eclipse

Originally Posted by Snookie

i came across creators using rule34video before

No further examples needed: As said, any http links in the description will be parsed
...except for ones that were crippled as explained in the beginning of my reply.

[shinji2009]

hi. is there a way now to bring back old folder naming style (same as filenames)(uploader_videoid_title) when downloading single video from iwara?

[pspzockerscene]

I haven't changed the defaults recently.

You can either use Packagizer rules or choose between some pre defined filename schemes here: Settings -> Plugins -> iwara.tv -> "Select preferred filename scheme"

[shinji2009]

i was thinking it is a bug and waiting for updates to fix it but it is not) folders are now title only. started between june 3 and june 4

[pspzockerscene]

How do you define "folders"?
The name of the package when crawling a user profile?

Either way I didn't purposely change any defaults so if I did, please let me know what has changed and how it was before.

[shinji2009]

yes, it is a name of package. and folders are named after name of package. june 2 it was uploader_videoid_title and june 4 become title. so probably package name was = filename scheme, but now it is hard title?

[pspzockerscene]

Yes and no: In the old version, the iwara.tv crawler could only always return one item: Either a youtube link or the iwara.tv selfhosted video.
Because of this, I did not set a package name at all so our auto-handling simply set the title of the video as package name.
Now the package name gets set hardcoded - yes!

So if you want to have the "old packagenames" back, here is what you can do:
Create a Packagizer rule which sets the file-title as packagename:
Screenshot:

[shinji2009]

thank you this is working, I would never have been able to deal with the packagizer myself.

[pspzockerscene]

No worries.

[Snookie]

Originally Posted by pspzockerscene

The button is only cosmetic. All of the description gets loaded right away.
You can check this by looking into their API to find the metadata of each video.
Random example video:

iwara.tv/video/k2ayoueezfkx6gvq/defeat-of-irybelda

API link:

api.iwara.tv/video/k2ayoueezfkx6gvq

I doubt that this is gonna help.
Ultimately, if they do this in a way that it is easy for us to automate to build the complete links, anyone with some technical knowledge will be able to do the same.
Also, the other way: If it is not possible to easily automatically build the original links, I won't be able to do it either and the only thing you can do is to download the description as .txt file and manually check for links later. Is there a demand to download the description text?


Works fine here in my dev version.
By the way you can get the dev version and test changes earlier, see:
**External links are only visible to Support Staff**...



No further examples needed: As said, any http links in the description will be parsed
...except for ones that were crippled as explained in the beginning of my reply.

i'm constantly checking for and waiting for this update to roll out, seeing those mega and google drive links in LinkGrabber is going to be so satisfying, this plugin can't get more perfect than this, you guys made what takes astonishing amount of time to browse and download manually, automatically doable in seconds without any efforts what's so ever, no words can convey my gratitude, i will forever be grateful for your help.

[pspzockerscene]

@Snookie
The update has been released yesterday.
You can also see that on our update FAQ page:
https://support.jdownloader.org/Know...pdate-behavior
--> You need to enable (opt-in!) the mentioned new plugin setting in the iwara.tv plugin settings.

[Snookie]

Originally Posted by pspzockerscene

@Snookie
The update has been released yesterday.
You can also see that on our update FAQ page:
**External links are only visible to Support Staff**...
--> You need to enable (opt-in!) the mentioned new plugin setting in the iwara.tv plugin settings.

i checked last night with a creator profile it catches only the iwara videos but after reading your comment now i checked again and it doesn't work still.. but then i checked with a single video link and it worked and catches the mega link.. its importance is when crawling for an entire creator profile not on single video link.. so is it a problem on my end only or does this work only on single video links ?

[pspzockerscene]

Should work fine.

Please _always_ include a log and example URLs for all reports otherwise I'm wasting one reply per request each time and you're wasting time too

I'm on vacation now so this issue, if there is one, will not be fixed until overnext week.
EDIT

I might have fixed it.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[ntljc16]

I want to make filename as below

[date] title (videoid).ext

Is there any way to solve this issue?
edit iwara plug-in's filename scheme? Or should I do something from package rule?

[pspzockerscene]

Use a Packagizer rule.
Available plugin properties for iwara.tv items:

• date
• user
• title
• videoid
• description

[ntljc16]

Originally Posted by pspzockerscene

Use a **External links are only visible to Support Staff**... rule.
Available plugin properties for iwara.tv items:

• date
• user
• title
• videoid
• description

I need examples or guide...

[pspzockerscene]

Guides and some examples:
https://support.jdownloader.org/Know...paketverwalter

You can find more examples in our support forums by searching for "Packagizer" or "Packagizer rule".

[Snookie]

Originally Posted by pspzockerscene

Should work fine.

Please _always_ include a log and example URLs for all reports otherwise I'm wasting one reply per request each time and you're wasting time too

I'm on vacation now so this issue, if there is one, will not be fixed until overnext week.
EDIT

I might have fixed it.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!

-psp-

after last core-update it's working as intended but look at the amount of links it's grabbing.. is it possible to limit this to just 3 hosts (mega,google drive, rule34video) all the rest are unnecessary as they are not files hosting websites.

[pspzockerscene]

Again / as always: Please provide example URLs.

I will add another plugin setting for iwara.tv which lets you whitelist URLs found in a video description via regular expressions.
Whitelisting multiple hosts would then work witha pattern like this:

.*(rapidgator\.net|mega\.nz).*

Good webtool for testing your regular expressions: regex101.com

[pspzockerscene]

iwara.tv plugin changes for the next CORE-update:
1. Fixed "Profile crawler: Skip externally hosted videos e.g. youtube.com URLs?" not working when other setting "Scan for URLs in content description?" was enabled.
2. Added whitelist RegEx setting for URLs found in content description.
If textfield is left empty [= default] or user enters an invalid regex, everything will be added/nothing will be ignored.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Snookie]

thank you so much for making this plugin as perfect as it can be.. its working flawlessly now, checks only for desired external hosts and takes only few seconds to do so.

[shinji2009]

hi. why do some videos show offline, although they are online and can be downloaded from the link that appears when you click on the "download" button?
examples:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Both of these links appear to be offline here.
All I can see in browser is:

Error
HTTP 404 - Not found

Maybe they're "shadow GEO-blocked"?
Are you maybe using a VPN in browser but not in JD?

[shinji2009]

no, sorry, they are indeed 404.