JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 22.09.2016, 15:51
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default import csv/txt file with 100'000 links

how to import some csv/txt file with 100'000 links?

csv is not supported
I did convert csv to links only and txt file format...

but there is nothing happening when adding through same procedure like importing dlc...
Reply With Quote
  #2  
Old 22.09.2016, 15:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

can you please provide example csv/sample to support@jdownloader.org
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 22.09.2016, 17:00
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

JDownloader should be able to find the links just fine. Please provide a sample and I will tell you why JDownloader fails to load yours
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 22.09.2016, 17:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Settings-Advanced Settings-LinkCrawler.linkcrawlerrules

Code:
[ {  
  "pattern" : "file:/.*?\\.csv$",
  "rule" : "DEEPDECRYPT"
} ]
Then JDownloader will react / autoparse csv files.

Don't forget to increase LinkCrawler.deepdecryptloadlimit
in case your file is larger
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 26.09.2016, 13:21
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

must not be csv file format...
I also converted to txt file with link list


**External links are only visible to Support Staff****External links are only visible to Support Staff**
...
...
**External links are only visible to Support Staff****External links are only visible to Support Staff**


but there is nothing happening :(
Reply With Quote
  #6  
Old 26.09.2016, 13:48
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by Jiaz View Post
Settings-Advanced Settings-LinkCrawler.linkcrawlerrules

Code:
[ {  
  "pattern" : "file:/.*?\\.csv$",
  "rule" : "DEEPDECRYPT"
} ]
Then JDownloader will react / autoparse csv files.

Don't forget to increase LinkCrawler.deepdecryptloadlimit
in case your file is larger
You have to create rules for txt and or csv files. By default JDownloader will not parse them
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 28.10.2016, 01:57
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
You have to create rules for txt and or csv files. By default JDownloader will not parse them
how to create a rule for that kind of usage?

(I am using csv file with multiple columns, mostly 4th or 5th columns is about url links to images...)
Reply With Quote
  #8  
Old 28.10.2016, 09:47
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

See the quote
Code:
[ {  
  "pattern" : "file:/.*?\\.csv$",
  "rule" : "DEEPDECRYPT"
} ]
in Settings-Advanced Settings-LinkCrawler.linkcrawlerrules
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 16.03.2017, 22:13
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
See the quote
Code:
[ {  
  "pattern" : "file:/.*?\\.csv$",
  "rule" : "DEEPDECRYPT"
} ]
in Settings-Advanced Settings-LinkCrawler.linkcrawlerrules

that works! great!

but how to make jdownloader work with csv files with 100'000 and more links?
- okay, it needs time to let linkcrawler process such a large number... but it always gives no response or is freezing...

well, it is running on win7 64bit 16gb ram...
Reply With Quote
  #10  
Old 16.03.2017, 22:28
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,533
Default

Quote:
Originally Posted by Jiaz View Post
Don't forget to increase LinkCrawler.deepdecryptloadlimit
in case your file is larger
Have you tried increasing 'LinkCrawler.deepdecryptloadlimit' limit in 'Advanced Settings'?
Reply With Quote
  #11  
Old 16.03.2017, 23:47
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by mgpai View Post
Have you tried increasing 'LinkCrawler.deepdecryptloadlimit' limit in 'Advanced Settings'?
yes of course, it is 5090000, max is given with 5242880 (5mb)


what is about "deepdecryptloadlimit" ? csv files have 300-500kb...
Reply With Quote
  #12  
Old 17.03.2017, 16:11
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

can anyone change from "solved" to "open" ?
Reply With Quote
  #13  
Old 18.03.2017, 15:04
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by verheiratet1952 View Post
that works! great!

but how to make jdownloader work with csv files with 100'000 and more links?
- okay, it needs time to let linkcrawler process such a large number... but it always gives no response or is freezing...

well, it is running on win7 64bit 16gb ram...

is there anything I can help with more details?

"Developer Feedback required"
Reply With Quote
  #14  
Old 18.03.2017, 17:18
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,507
Default

There is no "open" prefix, so I put "Developer Feedback required", so please wait for feedback (reply) from developer on Monday. If we need more info from you, the prefix is "User feedback required".
__________________
FAQ: How to upload a Log
Reply With Quote
  #15  
Old 20.03.2017, 18:22
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Please provide a screenshot of the About Dialog of JDownloader.
Also what filesize is the csv? And what is the value of Settings-Advanced Settings-DeepDecryptLoadLimit ?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #16  
Old 20.03.2017, 19:51
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
Please provide a screenshot of the About Dialog of JDownloader.
Also what filesize is the csv? And what is the value of Settings-Advanced Settings-DeepDecryptLoadLimit ?

please see screenshots...

csv file is always at about 300-1800kb...
I am using nearly max value from given available range...
Attached Thumbnails
jd-about.JPG   jd-deepdecryptloadlimit.JPG  
Reply With Quote
  #17  
Old 20.03.2017, 19:55
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Please start JDownloader with more heap memory. I can see that JDownloader is working near heap limit!
In your JDownloader folder there should be a .vmoptions file. edit it and add/replace
-Xmx8g
to increase max allowed heap to 8 gb.
In case there is no such file, you can start JDownloader manually with more memory
java -Xmx8g -jar JDownloader.jar

How many links/packages do you have in both list? What kind of links are you adding? What host?
Memory usage per link highly depends on the link type!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #18  
Old 20.03.2017, 19:58
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

I can also use csv splitter (to split into 500-10'000 links each) and add csv files one by one, but I think deepdecrypt can also handle that with one single csv file... it is more easy to add one single file and be out of office or away from keyword...
Reply With Quote
  #19  
Old 20.03.2017, 20:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

It's not the size of CSV, the number of links/type if links. To me it looks like you have LOTS of links/packages in JDownloader or the links you are trying to add require much memory.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 20.03.2017, 20:09
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
Please start JDownloader with more heap memory. I can see that JDownloader is working near heap limit!
In your JDownloader folder there should be a .vmoptions file. edit it and add/replace
-Xmx8g
to increase max allowed heap to 8 gb.
In case there is no such file, you can start JDownloader manually with more memory
java -Xmx8g -jar JDownloader.jar

How many links/packages do you have in both list? What kind of links are you adding? What host?
Memory usage per link highly depends on the link type!

.vmoptions currently has -Xms64m ?!
Reply With Quote
  #21  
Old 20.03.2017, 20:11
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

add new line with
-Xmx8g
__________________
JD-Dev & Server-Admin
Reply With Quote
  #22  
Old 20.03.2017, 20:12
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by Jiaz View Post
How many links/packages do you have in both list? What kind of links are you adding? What host?
Memory usage per link highly depends on the link type!
Any feedback about those questions?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #23  
Old 21.03.2017, 10:15
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Any Feedback ?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #24  
Old 21.03.2017, 13:25
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
It's not the size of CSV, the number of links/type if links. To me it looks like you have LOTS of links/packages in JDownloader or the links you are trying to add require much memory.

"LOTS of links" mean how many for you?

"links require much memory" - it also depends on host? it is normal websites... for that special case it is ranked round about 9000th with alexa.com
Reply With Quote
  #25  
Old 21.03.2017, 13:34
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Is it a direct http link or a link that requires plugin support in JDownloader?
LOTS of links = 200k-500k, 1 mio or more
Did the increased heap solve the issue?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #26  
Old 21.03.2017, 15:34
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
Is it a direct http link or a link that requires plugin support in JDownloader?
LOTS of links = 200k-500k, 1 mio or more
Did the increased heap solve the issue?
sorry for delay, but I was away from keyboard...

it is direct http/https links... no plugin in between...

changing to 8gb is a great performance push... taskmanager is now showing at about 8.800.000 K...

what about adding another or 2nd TMP/TEMP folder (for JD only) ?
I could use a 128gb USB 3.0 drive...




I was a little bit surprised that moving/copying a csv file from D to C drive was linkcrawler to start crawling for all urls... better saying - it was CTRL+C that started jd with linkcrawler...
Reply With Quote
  #27  
Old 21.03.2017, 15:43
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

would you suggest to close and open JD again before adding 2nd 3rd csv file?

I was using csv splitter to have 10-12'000 links only with one single csv file...
for me it seems to have better performance after close-reopen JD...
Reply With Quote
  #28  
Old 21.03.2017, 16:28
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

I guess the csv got auto added because of a bug in auto extension learning (will be fixed with next core update). JDownloader learned to autoparse .csv files in your case, that's why it also works when you copy the file to clipboard.

There should be no need to close-reopen JDownloader. 8GB heap should easily handle 1 million links. With enough free heap available there is no need for it. You can check about Dialog. In case it becomes slow again, please provide a new screenshot of the about dialog.

Of course increasing max heap will result in better performance
__________________
JD-Dev & Server-Admin
Reply With Quote
  #29  
Old 21.03.2017, 16:29
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by verheiratet1952 View Post
what about adding another or 2nd TMP/TEMP folder (for JD only) ?
I could use a 128gb USB 3.0 drive...
I'm sorry but I don't understand. JDownloader does not use any tmp folder.
What do you hope to achieve from this?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #30  
Old 21.03.2017, 18:33
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
I guess the csv got auto added because of a bug in auto extension learning (will be fixed with next core update). JDownloader learned to autoparse .csv files in your case, that's why it also works when you copy the file to clipboard.

There should be no need to close-reopen JDownloader. 8GB heap should easily handle 1 million links. With enough free heap available there is no need for it. You can check about Dialog. In case it becomes slow again, please provide a new screenshot of the about dialog.

Of course increasing max heap will result in better performance
I have 16gb ram, I can also change to 12gb instead of using 8gb

replacing -Xmx8g with -Xmx12g ?
Reply With Quote
  #31  
Old 21.03.2017, 18:42
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
I'm sorry but I don't understand. JDownloader does not use any tmp folder.
What do you hope to achieve from this?
I am not into coding, I'm just saying...

it is only about trying to find some way to result in better performance...
Reply With Quote
  #32  
Old 21.03.2017, 18:51
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
I guess the csv got auto added because of a bug in auto extension learning (will be fixed with next core update). JDownloader learned to autoparse .csv files in your case, that's why it also works when you copy the file to clipboard.

There should be no need to close-reopen JDownloader. 8GB heap should easily handle 1 million links. With enough free heap available there is no need for it. You can check about Dialog. In case it becomes slow again, please provide a new screenshot of the about dialog.

Of course increasing max heap will result in better performance
it is slow again... system is now using all 8gb... (8.889.362 K taskmanager)
Attached Thumbnails
jd-about-8gb.JPG  
Reply With Quote
  #33  
Old 21.03.2017, 18:55
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by verheiratet1952 View Post
it is slow again... system is now using all 8gb... (8.889.362 K taskmanager)
how to optimize CPU usage?
Attached Thumbnails
jd-cpu-usage.JPG  
Reply With Quote
  #34  
Old 21.03.2017, 18:56
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

How many Links do you have in List? That is important to know! Please provide any numbers. Screenshot shows that JDownloader is taking all the memory so either you have hundrets of thousands of links in JDownloader or something you do causes memleak.
8GB heap should easily be enough for 1mio and more links
__________________
JD-Dev & Server-Admin
Reply With Quote
  #35  
Old 21.03.2017, 18:57
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

No need to optimize CPU. Java is working near max memory limit and therefore spends lots of time in memory management.

How many links do you have in list? Can you provide example csv to support@jdownloader.org ?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #36  
Old 21.03.2017, 19:20
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
How many Links do you have in List? That is important to know! Please provide any numbers. Screenshot shows that JDownloader is taking all the memory so either you have hundrets of thousands of links in JDownloader or something you do causes memleak.
8GB heap should easily be enough for 1mio and more links
you talk about number of linkcrawler section only?
or download section is also important?
Reply With Quote
  #37  
Old 21.03.2017, 19:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Overall in JDownloader. You can enable overview panel that will show number of links/packages. I just need number of downloadlist/linkgrabber to be able to say that is has to to with number of links or must be some kind of memleak.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #38  
Old 21.03.2017, 19:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

PS: you can also write in german
__________________
JD-Dev & Server-Admin
Reply With Quote
  #39  
Old 21.03.2017, 19:39
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
Overall in JDownloader. You can enable overview panel that will show number of links/packages. I just need number of downloadlist/linkgrabber to be able to say that is has to to with number of links or must be some kind of memleak.

download has at about 3'500 folders with 1,2 million image links

linkgrabber has 30'000 folders with 1,6 million image links...
Reply With Quote
  #40  
Old 21.03.2017, 19:41
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

does it makes sense to save as DLC and re-import when ready to download?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 22:31.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.