JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #21  
Old 22.09.2022, 17:59
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 78,504
Default

@jDoX: Your rules are wrong.
DownloadURL -> this is the final URL that leads to the downloaded file/content
but the videos/content are NOT hosted on this domain!
That's why I use sourceURL in my example rule.
Example: you posted mediafire link here. DownloadURL is mediafire BUT sourceURL is board.jdownloader.org because that's the website where this link is posted.

Also please know that you specify filter/rules that tell JDownloader what to filter/block.
So for example your first rule blocks all Videos with name BLOGGER-video*.mp4 with size bewteen 1 and 980 mb and are hosted on the given domain. But those video are hosted on complete different domain!

Also please just specify the domain and leave alone the procotol else you will either have to use regex to support http and https or create two rules to support both. leave alone the procotol and it will work as well.

Just create a rule like explained
Quote:
file type - is not - drop down(video file)
and
sourceURL - contains - blogspot.com
That will block all files that have blogsport.com as sourceURL that are NOT videos
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 22.09.2022 at 18:03.
Reply With Quote
  #22  
Old 22.09.2022, 18:07
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 78,504
Default

Quote:
Originally Posted by jDoX View Post
T
BTW, if possible, I would like to download each video with its web page (each video in a folder containing the video and its web page), with all of them showing the video name (folder, web page and video). For instance, the video 1 corresponds to this link which corresponds to the 1 in the top matrix of the above web page
I'm sorry but that's not possible out of the box. You will have to
1.) add the html website as directhttp, so JDownloader downloads the html sourcecode. This is NO mirror. JDownloader does not support mirror/download of websites (including all files and rewriting the html)
It can only be used to download that actual html file.
2.) deep decrypt or linkcrawler rule to find the embedded videos on that website
__________________
JD-Dev & Server-Admin
Reply With Quote
  #23  
Old 22.09.2022, 18:11
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 78,504
Default

@jDoX: before we further trying to help you, maybe it would be better to make sure that it's possible to do what you want to achieve
__________________
JD-Dev & Server-Admin
Reply With Quote
  #24  
Old 22.09.2022, 22:27
jDoX jDoX is offline
JD Beta
 
Join Date: Jan 2012
Posts: 55
Default

Thanks. I have done as shown in the four screen captures at

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Please, note that I have merged the two previous Linkgrabber filters into a single one, removed some previous settings, including the ones that you said (sorry that I did not properly understood that before!) and also selected Condition - is true - Link is already in Downloadlist.

I do not know if all that is correct, but the fact is that it still does not work as I want (explained below). The first link is

**External links are only visible to Support Staff****External links are only visible to Support Staff**

and corresponds to the web page shown when clicking the number 1 in the matrix on top of such web pages. It is the first video out of about 400 (done just for testing). In such a case, the video shows in LinkGrabber, which is what I want.

But that can be also accomplished just pasting the link in jDownloader using the standard jDownloader factory-default settings. No need to create any Linkgrabber filter. I just did that as a testing example, as said above.

What do I want, then? I do not know if I explained it properly in my previous messages. What I want is to download all 400 videos at once, just using the index page containing their links as a number matrix on top:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

But when using such link, only four videos are shown. And that is because such web page contains them embedded below. So, the 400 videos corresponding to the top matrix of numbers are not shown. And those are the videos that I want to download, but not on a one-by-one basis, which would be utterly tedious and error-prone.

In other words, I do not want to click each one of the 400 links in the matrix, but just download them all at once using the index web page indicated above that contains such matrix. Is that possible? How?

Additionally, in the two examples of the screen captures indicated above, a lot a spurious unwanted files are shown. I only want the videos (and if possible the web pages that contain them). Is it possible to show only the desired files in LinkGrabber? How?

Last edited by jDoX; 23.09.2022 at 00:16.
Reply With Quote
  #25  
Old 24.09.2022, 01:52
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,016
Default

Hi again,
Quote:
Originally Posted by jDoX View Post
But when using such link, only four videos are shown. And that is because such web page contains them embedded below. So, the 400 videos corresponding to the top matrix of numbers are not shown. And those are the videos that I want to download, but not on a one-by-one basis, which would be utterly tedious and error-prone.
I recommend first getting that filter working then you can cope with the rest.
You will need to setup one or multiple LinkCrawler Rules to crawl the videos of all sub-pages and depending on how the website is made you might need to manually collect the URLs to all sub-pages in beforehand.
Some useful links:
https://support.jdownloader.org/Know...kcrawler-rules
and:
https://support.jdownloader.org/Know...orted-websites

Quote:
Originally Posted by jDoX View Post
Additionally, in the two examples of the screen captures indicated above, a lot a spurious unwanted files are shown. I only want the videos (and if possible the web pages that contain them). Is it possible to show only the desired files in LinkGrabber? How?
1. Your screenshots 3 and 4 are not soo useful because you haven't expanded the packages so we can't see what's inside but yeah probably non-video-files.
2. Your rule looks wrong as you've checked Condition -> is true -> Link is already in downloadlist
In the bottom toolbar in your screenshots 4 and 5 I can see that 39 links were filtered. These are probably items that are already in your downloadlist but all others got not filtered.
Change the rule accordingly and try again.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #26  
Old 24.09.2022, 11:53
jDoX jDoX is offline
JD Beta
 
Join Date: Jan 2012
Posts: 55
Default

Thanks. I am including expanded screen captures at

**External links are only visible to Support Staff****External links are only visible to Support Staff**

As before, they are sorted by size and only the top selected ones are videos (two and four, respectively). The others are spurious files that I do not want, yet they show in the LinkGrabber tab.

As said, link

**External links are only visible to Support Staff****External links are only visible to Support Staff**

was just the first video page out of the about 400 available. I can download it and all the rest of 400 videos just pasting their web pages in jDownloader. No special setting required. That works with the jDownloader factory-default settings, albeit they show in LinkGrabber tab with unwanted spurious files. Multiply about 400 web pages with those 37 unwanted spurious files for each and you get 14,800 unwanted spurious files. I just indicated such link for testing purposes.

The main problem is when I try to use jDownloader to download at once (automatically; not manually on a one-by-one basis) all about 400 videos without unwanted spurious files in the links of the top matrix of numbers shown at

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Is that possible? Could you accomplish that? I mean, could you get all such 400 video links to show in the jDownloader LinkGrabber tab without all the other spurious files that I do not want? How? I mean, which settings or procedure should be used for that?

Having said that, I want to download such about 400 videos, but it possible I would also like to download each of them with its web page as well (.html or similar like .webarchive, as Safari does). Ideally, each video in a subfolder, or at least all in the same folder with all videos and web pages named with the specific video names, for easier identification, if possible.

Thanks again for all, and sorry for my ignorance on this.

Last edited by jDoX; 24.09.2022 at 11:59.
Reply With Quote
  #27  
Old 26.09.2022, 16:00
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,016
Default

Okay let's start from the beginning.

This is a blog-style website.
It contains a mainpage which you can scrill down to view more post.
You want to have all videos of all posts.
It will be hard/impossible to auto find all posts but you can tech JD to only find videos when adding single post links.
It looks like all their videos are embedded via "blogger.com/video..." which is then hosting the videos on google servers.

Please disable the filter rule you've created before and continue reading:
Let's drop the filter approach for now and instead of filtering, just grab only what we want straight away using a LinkCrawler Rule:
Code:
[
  {
    "enabled": true,
    "logging": false,
    "maxDecryptDepth": 1,
    "name": "jdownloader.org example rule grab all video urls from 1000manerasdemorirx.blogspot.com",
    "pattern": "https?://1000manerasdemorirx\\.blogspot\\.com/[0-9]{4}/[0-9]{2}/[a-z0-9\\-_]+\\.html",
    "rule": "DEEPDECRYPT",
    "packageNamePattern": null,
    "deepPattern": "\"(https?://(www\\.)?blogger\\.com/video\\.g?[^\"]+)"
  }
]
Rule as plaintext for easier copy & paste:
pastebin.com/raw/fUaDP2QV

Once you got this working you can start thinking about how to collect all links to all individual posts...

Now I know you want to exclude stuff that is already in your downloadlist.
To simplify this, simply use the "Already in downloadlist" view in the linkgrabber and hide those items.

Quote:
Originally Posted by jDoX View Post
Having said that, I want to download such about 400 videos, but it possible I would also like to download each of them with its web page as well (.html or similar like .webarchive, as Safari does). Ideally, each video in a subfolder, or at least all in the same folder with all videos and web pages named with the specific video names, for easier identification, if possible..
Yes and no.
Automated no but you already need to collect all individual post links first so you can do it in 3 total steps:

1. Collect all links to individual posts.
2. Add them to JD to add- and download all video items.
3. Now modify each link and change the ending from ".html" to ".html.jdeatme" (basically just add ".jdeatme").
This will force JD to download that link/html without deep-analyzing it for non-html content.
You can easily modify all links by doing a "search & replace" action using the text-editor of your choice.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 27.09.2022 at 13:59. Reason: Updated LinkCrawler rule to allow underscores in blog post URLs
Reply With Quote
  #28  
Old 26.09.2022, 16:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 78,504
Default

Quote:
Originally Posted by jDoX View Post
Having said that, I want to download such about 400 videos, but it possible I would also like to download each of them with its web page as well (.html or similar like .webarchive, as Safari does). Ideally, each video in a subfolder, or at least all in the same folder with all videos and web pages named with the specific video names, for easier identification, if possible.
Please remember that this will be no website mirror but a pure/simple download of a static html file. No URLs will be updated/modified. You will not be able to correct open/view that file in browser because of missing references
__________________
JD-Dev & Server-Admin
Reply With Quote
  #29  
Old 26.09.2022, 16:20
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,016
Default

Quote:
Originally Posted by Jiaz View Post
Please remember that this will be no website mirror but a pure/simple download of a static html file. No URLs will be updated/modified. You will not be able to correct open/view that file in browser because of missing references
As addition to that:
There are tools specifically designed to download/copy all needed references.
I will link one old tool that I know from 2017 here though I'm pretty sure there are newer/better alternatives.
Old tool: httrack.com
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #30  
Old 26.09.2022, 20:29
jDoX jDoX is offline
JD Beta
 
Join Date: Jan 2012
Posts: 55
Default

Many thanks for the comprehensive reply! When I do it for instance for

**External links are only visible to Support Staff****External links are only visible to Support Staff**

I rename it as

**External links are only visible to Support Staff****External links are only visible to Support Staff**

and then paste it in jDownloader. Yet, then LinkGrabber only shows:

blog-post_448.html

No videos are shown. I must be doing something wrong, but I do not know what.
Reply With Quote
  #31  
Old 27.09.2022, 13:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,016
Default

Quote:
Originally Posted by jDoX View Post
No videos are shown.
Please re-read my last post.

I'll quote myself and add some more information:

Quote:
Originally Posted by pspzockerscene View Post
3. Now modify each link and change the ending from ".html" to ".html.jdeatme" (basically just add ".jdeatme").
This will force JD to download that link/html without deep-analyzing it for non-html content.
You can easily modify all links by doing a "search & replace" action using the text-editor of your choice.
You wanted to know how to download als/only the html code of such links and I've answered that question.

All other parts of my last reply cover the question "how to download only videos when adding such single blog-post URLs (without ".jdeatme" ending!!!).
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #32  
Old 27.09.2022, 13:59
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,016
Default

Addition:

I've just updated the LinkCrawler rule in my reply #27 as it didn't allow underscores in blog-post-URLs like the one you've used as an example in your last reply.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #33  
Old 27.09.2022, 17:41
jDoX jDoX is offline
JD Beta
 
Join Date: Jan 2012
Posts: 55
Default

Wow! Amazing! Many thanks!

Following your directions, jDownloader downloaded 302 vdeos after pasting

**External links are only visible to Support Staff****External links are only visible to Support Staff**

and 389 ".html" web pages after pasting the 389 URL which I previously extracted from the source of the web page above (replacing .html by .html.jdeatme).

The html web pages have the video names, but not the videos, so I will need to check them to place in the same subfolder each video file with its web page. What will take a while of manual work.

BTW, your post 27 above shows the code updated as you said, but the link
pastebin.com/raw/fUaDP2QV
shows the previous version.

Many thanks for all. If technically possible, it would be great if all the functionality required in this thread (or a similar one) could be built-in jDownloader for anyone to use.

Thanks again for all!
Reply With Quote
  #34  
Old 27.09.2022, 17:57
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,016
Default

Quote:
Originally Posted by jDoX View Post
Following your directions, jDownloader downloaded 302 vdeos after pasting...
Just in case wit this statement you want to indicate that some (video-)items are missing, here are possible reasons:
- Items are offline
- URLs contain characters which are not allowed by the rule I've created
- Items embed other types of videos which the rule does not pick up

You will need to analyze this on your own.

Quote:
Originally Posted by jDoX View Post
and 389 ".html" web pages after pasting the 389 URL which I previously extracted from the source of the web page above (replacing .html by .html.jdeatme).
That's the expected result

Quote:
Originally Posted by jDoX View Post
BTW, your post 27 above shows the code updated as you said, but the link
pastebin.com/raw/fUaDP2QV
shows the previous version.
Good catch!
I've corrected that in the existing paste so the link stays the same.

Quote:
Originally Posted by jDoX View Post
If technically possible, it would be great if all the functionality required in this thread (or a similar one) could be built-in jDownloader for anyone to use.
Of course it's technically possible but please consider that we're already supporting several thousands of websites and each one we add takes more time to add/maintain.
The generic tools JD comes with are there to be used by our users to automate downloading from simple websites like this one without the need to write any code.
I know it's not the most user friendly way but you probably won't find a software which provides user-friendlier ways to perform such tasks

Quote:
Originally Posted by jDoX View Post
Thanks again for all!
Thanks for your feedback.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #35  
Old 27.09.2022, 19:35
jDoX jDoX is offline
JD Beta
 
Join Date: Jan 2012
Posts: 55
Default

jDownloader is awesome. But I could not have accomplished the tasks of this thread without your help. That is why I said that it would be great if jDownloader could have it built-in.

Thanks again for making such awesome tool and for all your great help in this forum. You rock!
Reply With Quote
  #36  
Old 28.09.2022, 13:28
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,016
Default

Quote:
Originally Posted by jDoX View Post
jDownloader is awesome. But I could not have accomplished the tasks of this thread without your help. That is why I said that it would be great if jDownloader could have it built-in.
Yeah I get that point and probably it took us longer to help you than it would have taken me to write a crawler plugin but keep in mind that in the long term it is important that our users understand how to use such inbuilt tools because they will work for a lot of websites while we will never find enough time to add support for all websites our users would want to have plugins added for.
Just as a slight overview:
I'd estimate that at this moment we got at least 100+ open plugin request tickets which will probably never be done.
At the same time we have to maintain the existing plugins which takes several hours per day just check our forums daily updated threads and you will see what I mean.

Quote:
Originally Posted by jDoX View Post
Thanks again for making such awesome tool and for all your great help in this forum. You rock!
No worries.
Thanks for your awesome feedback!
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 07:06.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.