JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 11.06.2015, 20:55
djo_34 djo_34 is offline
DSL User
 
Join Date: Oct 2009
Posts: 39
Default Linkgrabber filtering

Three questions:

1) How do I disable the popup with the linkgrabber stats?

2) Is there a "clear list" button to delete all links in the linkgrabber list?

3) I tried creating a filter to all only "video files and custom types of ( rar, zip )" but if I copy anything in a browser, it adds virtually everything to the linkgrabber window. How can I filter out EVERYTHING except video, rar and zip?
Reply With Quote
  #2  
Old 11.06.2015, 21:32
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,338
Default

1.) at bottom right you can click on quick settings and disable overview/property window

2.) yes, at bottom on dowload-/linkgrabber-list, there is a trash/cleanup button

3.) A Filter filters out stuff. You need to create a Filter
Filetype - is not - and selection extension
Or
FileName - contains not - \.(rar|zip|avi|mpg|mp4) and enable the regex checkbox
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 11.06.2015, 22:45
djo_34 djo_34 is offline
DSL User
 
Join Date: Oct 2009
Posts: 39
Default

1) Thanks.
2) Thanks. I would suggest adding rollover text to all buttons. The icons are not intuitive to me.
3) This doesn't work. I checked FileType, "Is Not", checked Video File and Archive.
I pasted the link to this page into the test and got tons of GIF images accepted.
I can't turn uncheck the "File" option. So I set it to "Is Not", "Online".
Reply With Quote
  #4  
Old 12.06.2015, 07:28
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,659
Default

re2) there is tooltip with mouse hover for nearly every object in our interface. You might need to give it more time or you adjust the time value in advanced setting > GraphicalUserInterfaceSettings: Tooltip Delay

re3) rule is probably wrong? works fine here
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #5  
Old 01.07.2020, 13:08
Masteratarms Masteratarms is offline
JD Alpha
 
Join Date: Feb 2020
Posts: 20
Default

I have scraped a site but I did not pre apply my filter. I have so much stuff in the link grabber that even if I filter to display only what I don't want then clicking delete is lagging, not sure how long it will take to complete. As it is I am still able to do captures for previously queued items but can't add anything else till I clear this backlog. I don't want to use clearlinkgrabber list because it took a day to scrap this much (and I killed the process to stop it, 3900x 32GB ram). I did ask on how to filter before I started but I think I asked in the wrong forum not specific to jdownloader.:outch:

Last edited by Masteratarms; 01.07.2020 at 13:10.
Reply With Quote
  #6  
Old 01.07.2020, 13:10
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,140
Default

@Masteratarms
So how are we supposed to help you now?

You didn't even mention what you want to filter and there is no way we can speed-up your JD in that situation.
How much RAM does your system have?`
How much does JD use and what's the "max." value? See Help --> About JDownloader --> Memory

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #7  
Old 01.07.2020, 13:18
Masteratarms Masteratarms is offline
JD Alpha
 
Join Date: Feb 2020
Posts: 20
Default

Well I was just excluding the RAM as a possible cause of slowdown. I'm using 16GB/32 (JDownloader = 4GB, Firefox 1.4GB (5000+ tabs) + the rest).

I filtered to display archive, checksums, executable file, others, subtitle file. So I assume if I select all these will be removed from the list. Then I can check the filter to show audio, image & video files (I realise that zips could contain videos but I initially wanted images only). I have just one hoster which I excluded so it should remain after I delete all selected.

If it helps I have 356724 links from archive.org.

What it was it that I found some previews of some pictures on yandex search but the gallery was deleted. I edited the w & h dimentions in the url and I could get a iwebp file of 1065x1600 resolution. I used revese yandex reverse image lookup on that file and I got a jpg version (better fidelity) of one of the 2 pics. I guessed that there maybe more similar pictures but the gallery was deleted. So I deep scanned the url of the previews but without first filtering for iwebp, jpg, image file type.

I want to try filtering for image but program doesn't respond to a request to exit its using 10% cpu 303KB/s disk access (hard drive, I'd rather save my SSD writes).

Last edited by Masteratarms; 01.07.2020 at 13:34.
Reply With Quote
  #8  
Old 01.07.2020, 13:31
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,140
Default

How much RAM is JDs max. allowed and how much does it use?

Regarding your filters:
I would use Custom LinkGrabber Views instead of filters in this case as you can easily hide/show filtered items then without having to delete them.

See Settings -> Linkgrabber Filter --> Views

Your post still doesn't state what you want to filter and why a special filter is needed for that.
By default, JD has views for different file-types and hosts which seems to already cover what you want (?)

Sorry - without a more detailed description I cannot help you.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #9  
Old 01.07.2020, 13:38
Masteratarms Masteratarms is offline
JD Alpha
 
Join Date: Feb 2020
Posts: 20
Default

I assume its 4GB since its static at that number. How do I allow more? Maybe here: https://board.jdownloader.org/showth...allowed+memory

Can you be more specific in your line of questioning base on my previous post because I thought I was pretty specific in that one?

Last edited by Masteratarms; 01.07.2020 at 13:44.
Reply With Quote
  #10  
Old 01.07.2020, 13:47
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,140
Default

See JD install dir --> Edit file JDownloader2.vmoptions --> Change "Xmx" value.

Regaring filters:
Neither filters now views will just permanently remove items from your linkgrabber.
Also you're saying you've already added these items and probably do not want to re-add them to your linkgrabber (?)
In this case, the only thing that you can use "on the fly" if I remember correctly, are custom views.

Also, did you use filters or views here?
Please add screenshots of your filters.

If I understand, this is what you want to have:
Pictures

This is what you do not want to have:
Everything except pictures

If my assumption correct or did I mis-understood you?

I also thing you're doing it the wrong way - at least I would do it differently:
Filter (= hide) everyything that you do not want with a linkgrabber view --> Move everything you want from LinkGrabber to DownloadList --> Disable the filter again and check if you missed anything that you want to download --> If not, clear all items from your LinkGrabber

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 01.07.2020 at 14:00. Reason: What I need from you = BOLD
Reply With Quote
  #11  
Old 01.07.2020, 13:50
Masteratarms Masteratarms is offline
JD Alpha
 
Join Date: Feb 2020
Posts: 20
Default

JDownloader2.vmoptions

-Xms64m
-Dsun.java2d.d3d=false

edited to:
-Xms8g
-Dsun.java2d.d3d=false

It made no difference to the ram usage read from IObit perormance monitor.

Last edited by Masteratarms; 01.07.2020 at 13:56.
Reply With Quote
  #12  
Old 01.07.2020, 13:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,140
Default

It should be fine:
Again I've asked you how much it needs and you never answered that question.
See Help --> About JDownloader --> Memory: Usage: XX MB - Allocated XX MB- Max. XX MB
EDIT
I do NOT want to have the values you can see in your task-manager!

Just allowing more makes no sense if it doesn't require more

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 01.07.2020 at 14:00.
Reply With Quote
  #13  
Old 01.07.2020, 14:01
Masteratarms Masteratarms is offline
JD Alpha
 
Join Date: Feb 2020
Posts: 20
Default

I'm quoting my post above:
"Well I was just excluding the RAM as a possible cause of slowdown. I'm using 16GB/32 (JDownloader = 4GB, Firefox 1.4GB (5000+ tabs) + the rest)."

I have only seen 32GB come in handy when making an archive of extracted frames.

Just to state the obvious: Jdownloader is using 4GB ram and I want to allow more but that edit did not effect the displayed usage. I will try editing the other file.

Just FYI doing it "your way" and filtering for links unrelated to this crawl, I can set a default download directory but the "start downloads" option does not have any effect.

I think its best if I start again with clear link grabber and make my filters to the filetypes before hand. I'd need profiles for that though because I often use the scrape with a more specific url where pre filtering is unnecessary.

Last edited by Masteratarms; 01.07.2020 at 14:09.
Reply With Quote
  #14  
Old 01.07.2020, 14:06
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,140
Default

Quote:
Originally Posted by Masteratarms View Post
I'm quoting my post above:
Well I was just excluding the RAM as a possible cause of slowdown. I'm using 16GB/32 (JDownloader = 4GB, Firefox 1.4GB (5000+ tabs) + the rest).
I assumed you got that from your task manager that's why I was asking again.

Quote:
Originally Posted by Masteratarms View Post
I have only seen 32GB come in handy when making an archive of extracted frames.

Just to state the obvious: Jdownloader is using 4GB ram and I want to allow more but that edit did not effect the displayed usage. I will try editing the other file.
It will not increase the usage - only if JD really needs more.

I think we're on the wrong path here so let's continue with your filters:
Did my posts help you or is it still unclear for you, how to filter the items you have?

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #15  
Old 01.07.2020, 14:34
Masteratarms Masteratarms is offline
JD Alpha
 
Join Date: Feb 2020
Posts: 20
Default

It was sitting @ 4GB, I am now @ 2.15GB (probably because I have most of my past downloads still in the download list) after deleting all the link grabber list files manually:
https://board.jdownloader.org/showpo...88&postcount=2

Did you say that I could create a custom filter before adding a url to deep scan?

Also I tried editing that file JDownloader2.vmoptions & JDownloader2Update.vmoptions both to 8GB and I tested with the links back in the folder and I'm limited to 4GB according to IObit performance monitor. I need to use more RAM, I as I said I only have 2GB spare with my download list as it is.

I know how to filter for images, but can you filter for *.webp only?

https://imgur.com/a/azeBTx3
Is there anything else I should add?

I am not sure I'm using this feature in the way intended. I was looking to filter when on the add new links dialog (2nd screenshot).

Last edited by Masteratarms; 01.07.2020 at 14:55.
Reply With Quote
  #16  
Old 01.07.2020, 15:28
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,140
Default

Your 2GB should be fine - just try it out.

You cannot filter stuff "inside the add links dialog" - only afterwards, inside the linkgrabber.

Your rule should be fine as long as JD recognizes .webp as image.

Otherwise chose file type --> is --> custom --> "webp"

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #17  
Old 01.07.2020, 15:43
Masteratarms Masteratarms is offline
JD Alpha
 
Join Date: Feb 2020
Posts: 20
Default

Well its not fine, I can't seem to reproduce the scrape and I've got the linkgrabber files saved to a backup directory. I need to unlock more RAM space. I'm using 2GB out of 32GB that's ridiculous. Actually the ram usage has changed to 3.12GB so that is less than 1GB for these links. I've got 300,000 links I need to filter and the program just fails to respond when I try to make any configuration other than filter.

As a last resort I can backup in the same way my downloaded files and temporarily just work within 4GB limit (which is an 8th of my RAM capacity).
Reply With Quote
  #18  
Old 01.07.2020, 16:02
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,140
Default

Java Xms vs Xmx parameters - you probably did it wrong:
stackoverflow.com/questions/5374455/what-does-java-option-xmx-stand-for
stackoverflow.com/questions/1098488/jvm-heap-parameters

I was able to start JD with max ~1500 MB of max memory.
Trying to use 8GB, I got the following error and JD refused to start:


... so as long as your value hasn't changed, you probably only set "Xms" but not "Xmx".
Try e.g. "-Xmx1500m" and you should see a change of the max. value in the about dialog.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #19  
Old 01.07.2020, 16:26
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,533
Default

Quote:
Originally Posted by Masteratarms View Post
... I'm limited to 4GB according to IObit performance monitor.
If you are using 32-bit version, it is most likely due to JVM max heap limit, in which case. if you need more memory for JD, you will have to upgrade to 64-bit version.

Code:
javarevisited.blogspot.com/2013/04/what-is-maximum-heap-size-for-32-bit-64-JVM-Java-memory.html

Check "Help > About Jdownloader" for Java version and memory allocation/usage. It is dynamically allocated based on the current operation. As long as 'Allocated' value remains less than 'Max' value, you should be able to filter/prune the list without any issues.

The lag may also be caused due to frequently updating the link list file on disk, in which case you can try temporarily increasing the save/write delay in advanced settings:

Code:
LinkCollector.maximumsavedelay
LinkCollector.minimumsavedelay

As a last resort, you can also try removing the links using eventscripter script.
Reply With Quote
  #20  
Old 01.07.2020, 16:34
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,140
Default

@mgpai
Thanks for your added information!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 09:57.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.