JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #161  
Old 28.03.2024, 15:12
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,977
Default

The "Avoid downloading 'too small' videos?" setting has been added with todays' update.
It is disabled by default.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #162  
Old 11.06.2024, 05:53
shinji2009 shinji2009 is offline
Mega Loader
 
Join Date: Aug 2021
Posts: 66
Default

hi. can you please show how regex whitelist for crawled urls should look like with at least two example links? i'm confused reading the description of this field

Last edited by shinji2009; 11.06.2024 at 06:21.
Reply With Quote
  #163  
Old 11.06.2024, 09:52
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,977
Default

You mean for at least two soures/websites?
Example:
Code:
https?://(website1\.tld|website2\.tld)/.+
Learn and test regular expressions with this free webtool: regex101.com
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #164  
Old 11.06.2024, 10:09
shinji2009 shinji2009 is offline
Mega Loader
 
Join Date: Aug 2021
Posts: 66
Default

i was just thinking that use real urls in the example will be less confusing
like
https?://(mega.nz\.tld|gofile.io\.tld)/.+
or
https?://(mega\.nz|gofile\.io)/.+
Reply With Quote
  #165  
Old 11.06.2024, 10:25
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,977
Default

I just noticed that the description of said setting already contains an example.
Just keep in mind that it's up to the user to learn how regular expressions work though for this sole purpose you only really need a very basic understanding.

I've updated the example inside the settings to use "real websites" for the next update.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #166  
Old 11.06.2024, 10:50
shinji2009 shinji2009 is offline
Mega Loader
 
Join Date: Aug 2021
Posts: 66
Default

thank you. best description is self-contained description
Reply With Quote
  #167  
Old 13.06.2024, 14:40
shinji2009 shinji2009 is offline
Mega Loader
 
Join Date: Aug 2021
Posts: 66
Default

by the way. haven't you thinking making blacklist for crawled urls more convinient and logical than whitelist? because when you crawl a lot of videos, you can fastly update what you don't want to crawl (mostly it is only twitter and youtube), but you never know what file host uploader will choose for his video in better quality. and there are hundreds of file hosts. after you whitelist something, you need to check every video page, because there can be some other hosts. is it possible maybe to add a blacklist?

Last edited by shinji2009; 13.06.2024 at 14:44.
Reply With Quote
  #168  
Old 13.06.2024, 15:30
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,977
Default

Quote:
Originally Posted by shinji2009 View Post
haven't you thinking making blacklist for crawled urls more convinient and logical than whitelist?
In the context of iwara.tv:
I didn't think about it at all since I'm not using the iwara.tv plugin myself.
Back then I simply decided to add a whitelist instead of just a checkbox "Crawl external URLs" as it is more versatile.

Quote:
Originally Posted by shinji2009 View Post
is it possible maybe to add a blacklist?
Sure I could add the functionality for this plugin but maybe you want to check the global blacklisting feature first:
Settings -> Advanced Settings:
Code:
GeneralSettings.crawlerhostpluginblacklist
and:
Code:
GeneralSettings.crawlercrawlerpluginblacklist

Quote:
Originally Posted by shinji2009 View Post
you need to check every video page
The iwara.tv crawler is logging URLs which were skipped according to your whitelist, you can find them in your logs by looking for the text string "Skipped URLs due du users' whitelist pattern".
There are a lot of possible ways to make it work and I'am just trying to make users stick to our global features vs adding plugin-specific features.
So additionally to the global blacklists I definitely recommend you to check Settings -> Linkgrabber filter
You should be able to e.g. only filter youtube links when they come from a sourceurl which contains "iwara.tv".
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #169  
Old 13.06.2024, 15:46
shinji2009 shinji2009 is offline
Mega Loader
 
Join Date: Aug 2021
Posts: 66
Default

i'm just saing from the convinience side. changing global settings is not an option for me beacuse i download from youtube too. i just like when you don't need to change settings depeding of where are you downloading from, and everythig was fine until recenly i find that some of iwara users are uploading videos in better qulity with external links. this make me look to iwara plugin settings to automate it. and today i realise that whitelist is not an option if i don't want to miss something. i guess i'll stick to linkgrabber options then, thank you
Reply With Quote
  #170  
Old 13.06.2024, 16:14
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,977
Default

Quote:
Originally Posted by shinji2009 View Post
changing global settings is not an option for me beacuse i download from youtube too
I understand that blocking YT globally is not an option.

Quote:
Originally Posted by shinji2009 View Post
i guess i'll stick to linkgrabber options then, thank you
Not only that but also the filters.
EDIT
The advantage when using the filters is that the items do not need to be processed in order to filter them.
This makes everything faster.

Here is an example rule - if you were to allow _everything_ in the iwara.tv whitelist, you can block stuff via rules as shown here:
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 13.06.2024 at 16:17. Reason: EDIT
Reply With Quote
  #171  
Old 13.06.2024, 16:20
shinji2009 shinji2009 is offline
Mega Loader
 
Join Date: Aug 2021
Posts: 66
Default

this is looks like just what i needed, thank you very much!
Reply With Quote
  #172  
Old 13.06.2024, 16:23
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,977
Default

There you go.
Keep in mind that my example is just a very really lazy rule.

I recommend using the full domain to block sites/sources and if you want to block more than one, enable the regular expression checkbox so that you can filter multiple domains with one rule.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #173  
Old 13.06.2024, 17:22
shinji2009 shinji2009 is offline
Mega Loader
 
Join Date: Aug 2021
Posts: 66
Default

there is this one user
**External links are only visible to Support Staff****External links are only visible to Support Staff**
he leaves in a description a link to a previous video god knows why.
if i filter iwara.tv/video for iwara.tv source, jd don't grab a thing
if i don't filter iwara.tv/video for iwara.tv, jd start to grab a whole iwara so it seems, because one link goes to another, and another and so on.
i personally will remember him. but can you please give a hint if this can be filtered?
Reply With Quote
  #174  
Old 14.06.2024, 12:23
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,977
Default

In this case, use the whitelist regex of the iwara.tv plugin to tell it "allow everything except iwara.tv links", using a 'negative lookahead' regular expression.
Regex:
Code:
https?://(?!(www\.)?iwara\.tv)[^/]+/.+
As already explained, learning regular expressions is they key to work with such filters.
It will also be very helpful for e.g. filename replaces and much more.
Use webtools like regex101.com to test regular expressions as that is easier than using JDownloader plugins to test them.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #175  
Old 15.06.2024, 04:54
shinji2009 shinji2009 is offline
Mega Loader
 
Join Date: Aug 2021
Posts: 66
Default

using whitelist as a blacklist sounds like a witchcraft)
for some reason i can't make grabber filter work for youtube. i make rules for both youtube and youtu.be, and youtu.be is working, but for youtube jd still ask me about downloading a playlist.
example links
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #176  
Old 15.06.2024, 12:23
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,384
Default

@shinji2009: just to be sure, you want to block youtube links getting crawled/processed from iwara, right?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #177  
Old 15.06.2024, 13:41
shinji2009 shinji2009 is offline
Mega Loader
 
Join Date: Aug 2021
Posts: 66
Default

yes. i make some rules to block different sites crawling from iwara and they all seem to work except "youtube"

Last edited by shinji2009; 15.06.2024 at 13:44.
Reply With Quote
  #178  
Old 16.06.2024, 21:07
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,384
Default

@shinji2009: currentlly not possible but I will work on a solution for this
__________________
JD-Dev & Server-Admin
Reply With Quote
  #179  
Old 17.06.2024, 11:09
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,977
Default

Quote:
Originally Posted by shinji2009 View Post
using whitelist as a blacklist sounds like a witchcraft)
That witchcraft continues here.

Jiaz is talking about the upper filter implementation but while it would be nice to have it working in this way, you can use the same magic inside the iwara.tv plugin to make this work right now.
Simply extend the regular epxression I've posted previously:
Code:
https?://(?!(www\.)?(iwara\.tv|youtube\.com|youtu\\.be))[^/]+/.+
Again:
You need to learn how regular expressions work if you want to continue working with such filters.

@Jiaz
I'll mark this as solved since the user will get it working this way.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #180  
Old 22.06.2024, 04:53
shinji2009 shinji2009 is offline
Mega Loader
 
Join Date: Aug 2021
Posts: 66
Default

code is not working for iwara.tv/users link
**External links are only visible to Support Staff****External links are only visible to Support Staff**
linkgrabber rule for this case also not working
sorry i don't understand why. for me if code blocks iwara.tv/video it must block iwara.tv/users, but is it not.

Last edited by shinji2009; 23.06.2024 at 10:39.
Reply With Quote
  #181  
Old 24.06.2024, 09:59
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,977
Default

@shinji2009
I expect you to put some work into this.
So far it looks like you haven't even bothered understanding the regular expression I made for you so one last time, I'll explain it.
Now here is the version which you are most likely using at this moment:
Code:
https?://(?!(www\.)?(iwara\.tv|youtube\.com|youtu\.be))[^/]+/.+
Here is how the iwara.tv link from said videos' description looks like:
ecchi.iwara.tv/users/sssmmddaisuki

How why doesn't irt work?
Because that link is starting with "ecchi." while the regular expression only works for links that start with "www.".

So what's the possible fix?
Code:
https?://(?!([a-z0-9]+\.)?(iwara\.tv|youtube\.com|youtu\.be))[^/]+/.+
Once again: use tools like regex101.com to test regular expressions.
Learn how to use them so you can use them to your advantage in the future.
I will not provide any further help with this one!
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 12:46.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.