JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 22.09.2016, 14:51
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default import csv/txt file with 100'000 links

how to import some csv/txt file with 100'000 links?

csv is not supported
I did convert csv to links only and txt file format...

but there is nothing happening when adding through same procedure like importing dlc...
Reply With Quote
  #2  
Old 22.09.2016, 14:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 64,029
Default

can you please provide example csv/sample to support@jdownloader.org
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 22.09.2016, 16:00
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 64,029
Default

JDownloader should be able to find the links just fine. Please provide a sample and I will tell you why JDownloader fails to load yours
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 22.09.2016, 16:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 64,029
Default

Settings-Advanced Settings-LinkCrawler.linkcrawlerrules

Code:
[ {  
  "pattern" : "file:/.*?\\.csv$",
  "rule" : "DEEPDECRYPT"
} ]
Then JDownloader will react / autoparse csv files.

Don't forget to increase LinkCrawler.deepdecryptloadlimit
in case your file is larger
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 26.09.2016, 12:21
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

must not be csv file format...
I also converted to txt file with link list


**External links are only visible to Support Staff****External links are only visible to Support Staff**
...
...
**External links are only visible to Support Staff****External links are only visible to Support Staff**


but there is nothing happening :(
Reply With Quote
  #6  
Old 26.09.2016, 12:48
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 64,029
Default

Quote:
Originally Posted by Jiaz View Post
Settings-Advanced Settings-LinkCrawler.linkcrawlerrules

Code:
[ {  
  "pattern" : "file:/.*?\\.csv$",
  "rule" : "DEEPDECRYPT"
} ]
Then JDownloader will react / autoparse csv files.

Don't forget to increase LinkCrawler.deepdecryptloadlimit
in case your file is larger
You have to create rules for txt and or csv files. By default JDownloader will not parse them
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 28.10.2016, 00:57
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

Quote:
Originally Posted by Jiaz View Post
You have to create rules for txt and or csv files. By default JDownloader will not parse them
how to create a rule for that kind of usage?

(I am using csv file with multiple columns, mostly 4th or 5th columns is about url links to images...)
Reply With Quote
  #8  
Old 28.10.2016, 08:47
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 64,029
Default

See the quote
Code:
[ {  
  "pattern" : "file:/.*?\\.csv$",
  "rule" : "DEEPDECRYPT"
} ]
in Settings-Advanced Settings-LinkCrawler.linkcrawlerrules
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 16.03.2017, 21:13
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

Quote:
Originally Posted by Jiaz View Post
See the quote
Code:
[ {  
  "pattern" : "file:/.*?\\.csv$",
  "rule" : "DEEPDECRYPT"
} ]
in Settings-Advanced Settings-LinkCrawler.linkcrawlerrules

that works! great!

but how to make jdownloader work with csv files with 100'000 and more links?
- okay, it needs time to let linkcrawler process such a large number... but it always gives no response or is freezing...

well, it is running on win7 64bit 16gb ram...
Reply With Quote
  #10  
Old 16.03.2017, 21:28
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 555
Default

Quote:
Originally Posted by Jiaz View Post
Don't forget to increase LinkCrawler.deepdecryptloadlimit
in case your file is larger
Have you tried increasing 'LinkCrawler.deepdecryptloadlimit' limit in 'Advanced Settings'?
Reply With Quote
  #11  
Old 16.03.2017, 22:47
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

Quote:
Originally Posted by mgpai View Post
Have you tried increasing 'LinkCrawler.deepdecryptloadlimit' limit in 'Advanced Settings'?
yes of course, it is 5090000, max is given with 5242880 (5mb)


what is about "deepdecryptloadlimit" ? csv files have 300-500kb...
Reply With Quote
  #12  
Old 17.03.2017, 15:11
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

can anyone change from "solved" to "open" ?
Reply With Quote
  #13  
Old 18.03.2017, 14:04
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

Quote:
Originally Posted by verheiratet1952 View Post
that works! great!

but how to make jdownloader work with csv files with 100'000 and more links?
- okay, it needs time to let linkcrawler process such a large number... but it always gives no response or is freezing...

well, it is running on win7 64bit 16gb ram...

is there anything I can help with more details?

"Developer Feedback required"
Reply With Quote
  #14  
Old 18.03.2017, 16:18
tony2long's Avatar
tony2long tony2long is online now
English Supporter
 
Join Date: Jun 2009
Posts: 6,142
Default

There is no "open" prefix, so I put "Developer Feedback required", so please wait for feedback (reply) from developer on Monday. If we need more info from you, the prefix is "User feedback required".
__________________
FAQ: How to upload a Log
Reply With Quote
  #15  
Old 20.03.2017, 17:22
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 64,029
Default

Please provide a screenshot of the About Dialog of JDownloader.
Also what filesize is the csv? And what is the value of Settings-Advanced Settings-DeepDecryptLoadLimit ?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #16  
Old 20.03.2017, 18:51
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

Quote:
Originally Posted by Jiaz View Post
Please provide a screenshot of the About Dialog of JDownloader.
Also what filesize is the csv? And what is the value of Settings-Advanced Settings-DeepDecryptLoadLimit ?

please see screenshots...

csv file is always at about 300-1800kb...
I am using nearly max value from given available range...
Attached Thumbnails
jd-about.JPG   jd-deepdecryptloadlimit.JPG  
Reply With Quote
  #17  
Old 20.03.2017, 18:55
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 64,029
Default

Please start JDownloader with more heap memory. I can see that JDownloader is working near heap limit!
In your JDownloader folder there should be a .vmoptions file. edit it and add/replace
-Xmx8g
to increase max allowed heap to 8 gb.
In case there is no such file, you can start JDownloader manually with more memory
java -Xmx8g -jar JDownloader.jar

How many links/packages do you have in both list? What kind of links are you adding? What host?
Memory usage per link highly depends on the link type!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #18  
Old 20.03.2017, 18:58
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

I can also use csv splitter (to split into 500-10'000 links each) and add csv files one by one, but I think deepdecrypt can also handle that with one single csv file... it is more easy to add one single file and be out of office or away from keyword...
Reply With Quote
  #19  
Old 20.03.2017, 19:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 64,029
Default

It's not the size of CSV, the number of links/type if links. To me it looks like you have LOTS of links/packages in JDownloader or the links you are trying to add require much memory.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 20.03.2017, 19:09
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

Quote:
Originally Posted by Jiaz View Post
Please start JDownloader with more heap memory. I can see that JDownloader is working near heap limit!
In your JDownloader folder there should be a .vmoptions file. edit it and add/replace
-Xmx8g
to increase max allowed heap to 8 gb.
In case there is no such file, you can start JDownloader manually with more memory
java -Xmx8g -jar JDownloader.jar

How many links/packages do you have in both list? What kind of links are you adding? What host?
Memory usage per link highly depends on the link type!

.vmoptions currently has -Xms64m ?!
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 08:58.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.