#1
|
|||
|
|||
import csv/txt file with 100'000 links
how to import some csv/txt file with 100'000 links?
csv is not supported I did convert csv to links only and txt file format... but there is nothing happening when adding through same procedure like importing dlc... |
#2
|
||||
|
||||
can you please provide example csv/sample to support@jdownloader.org
__________________
JD-Dev & Server-Admin |
#3
|
||||
|
||||
JDownloader should be able to find the links just fine. Please provide a sample and I will tell you why JDownloader fails to load yours
__________________
JD-Dev & Server-Admin |
#4
|
||||
|
||||
Settings-Advanced Settings-LinkCrawler.linkcrawlerrules
Code:
[ { "pattern" : "file:/.*?\\.csv$", "rule" : "DEEPDECRYPT" } ] Don't forget to increase LinkCrawler.deepdecryptloadlimit in case your file is larger
__________________
JD-Dev & Server-Admin |
#5
|
|||
|
|||
must not be csv file format...
I also converted to txt file with link list **External links are only visible to Support Staff****External links are only visible to Support Staff** ... ... **External links are only visible to Support Staff****External links are only visible to Support Staff** but there is nothing happening :( |
#6
|
||||
|
||||
You have to create rules for txt and or csv files. By default JDownloader will not parse them
__________________
JD-Dev & Server-Admin |
#7
|
|||
|
|||
Quote:
(I am using csv file with multiple columns, mostly 4th or 5th columns is about url links to images...) |
#8
|
||||
|
||||
See the quote
Code:
[ { "pattern" : "file:/.*?\\.csv$", "rule" : "DEEPDECRYPT" } ]
__________________
JD-Dev & Server-Admin |
#9
|
|||
|
|||
Quote:
that works! great! but how to make jdownloader work with csv files with 100'000 and more links? - okay, it needs time to let linkcrawler process such a large number... but it always gives no response or is freezing... well, it is running on win7 64bit 16gb ram... |
#10
|
|||
|
|||
Have you tried increasing 'LinkCrawler.deepdecryptloadlimit' limit in 'Advanced Settings'?
|
#11
|
|||
|
|||
Quote:
what is about "deepdecryptloadlimit" ? csv files have 300-500kb... |
#12
|
|||
|
|||
can anyone change from "solved" to "open" ?
|
#13
|
|||
|
|||
Quote:
is there anything I can help with more details? "Developer Feedback required" |
#14
|
||||
|
||||
There is no "open" prefix, so I put "Developer Feedback required", so please wait for feedback (reply) from developer on Monday. If we need more info from you, the prefix is "User feedback required".
__________________
FAQ: How to upload a Log |
#15
|
||||
|
||||
Please provide a screenshot of the About Dialog of JDownloader.
Also what filesize is the csv? And what is the value of Settings-Advanced Settings-DeepDecryptLoadLimit ?
__________________
JD-Dev & Server-Admin |
#16
|
|||
|
|||
Quote:
please see screenshots... csv file is always at about 300-1800kb... I am using nearly max value from given available range... |
#17
|
||||
|
||||
Please start JDownloader with more heap memory. I can see that JDownloader is working near heap limit!
In your JDownloader folder there should be a .vmoptions file. edit it and add/replace -Xmx8g to increase max allowed heap to 8 gb. In case there is no such file, you can start JDownloader manually with more memory java -Xmx8g -jar JDownloader.jar How many links/packages do you have in both list? What kind of links are you adding? What host? Memory usage per link highly depends on the link type!
__________________
JD-Dev & Server-Admin |
#18
|
|||
|
|||
I can also use csv splitter (to split into 500-10'000 links each) and add csv files one by one, but I think deepdecrypt can also handle that with one single csv file... it is more easy to add one single file and be out of office or away from keyword...
|
#19
|
||||
|
||||
It's not the size of CSV, the number of links/type if links. To me it looks like you have LOTS of links/packages in JDownloader or the links you are trying to add require much memory.
__________________
JD-Dev & Server-Admin |
#20
|
|||
|
|||
Quote:
.vmoptions currently has -Xms64m ?! |
#21
|
||||
|
||||
add new line with
-Xmx8g
__________________
JD-Dev & Server-Admin |
#22
|
||||
|
||||
Any feedback about those questions?
__________________
JD-Dev & Server-Admin |
#23
|
||||
|
||||
Any Feedback ?
__________________
JD-Dev & Server-Admin |
#24
|
|||
|
|||
Quote:
"LOTS of links" mean how many for you? "links require much memory" - it also depends on host? it is normal websites... for that special case it is ranked round about 9000th with alexa.com |
#25
|
||||
|
||||
Is it a direct http link or a link that requires plugin support in JDownloader?
LOTS of links = 200k-500k, 1 mio or more Did the increased heap solve the issue?
__________________
JD-Dev & Server-Admin |
#26
|
|||
|
|||
Quote:
it is direct http/https links... no plugin in between... changing to 8gb is a great performance push... taskmanager is now showing at about 8.800.000 K... what about adding another or 2nd TMP/TEMP folder (for JD only) ? I could use a 128gb USB 3.0 drive... I was a little bit surprised that moving/copying a csv file from D to C drive was linkcrawler to start crawling for all urls... better saying - it was CTRL+C that started jd with linkcrawler... |
#27
|
|||
|
|||
would you suggest to close and open JD again before adding 2nd 3rd csv file?
I was using csv splitter to have 10-12'000 links only with one single csv file... for me it seems to have better performance after close-reopen JD... |
#28
|
||||
|
||||
I guess the csv got auto added because of a bug in auto extension learning (will be fixed with next core update). JDownloader learned to autoparse .csv files in your case, that's why it also works when you copy the file to clipboard.
There should be no need to close-reopen JDownloader. 8GB heap should easily handle 1 million links. With enough free heap available there is no need for it. You can check about Dialog. In case it becomes slow again, please provide a new screenshot of the about dialog. Of course increasing max heap will result in better performance
__________________
JD-Dev & Server-Admin |
#29
|
||||
|
||||
Quote:
What do you hope to achieve from this?
__________________
JD-Dev & Server-Admin |
#30
|
|||
|
|||
Quote:
replacing -Xmx8g with -Xmx12g ? |
#31
|
|||
|
|||
Quote:
it is only about trying to find some way to result in better performance... |
#32
|
|||
|
|||
Quote:
|
#33
|
|||
|
|||
how to optimize CPU usage?
|
#34
|
||||
|
||||
How many Links do you have in List? That is important to know! Please provide any numbers. Screenshot shows that JDownloader is taking all the memory so either you have hundrets of thousands of links in JDownloader or something you do causes memleak.
8GB heap should easily be enough for 1mio and more links
__________________
JD-Dev & Server-Admin |
#35
|
||||
|
||||
No need to optimize CPU. Java is working near max memory limit and therefore spends lots of time in memory management.
How many links do you have in list? Can you provide example csv to support@jdownloader.org ?
__________________
JD-Dev & Server-Admin |
#36
|
|||
|
|||
Quote:
or download section is also important? |
#37
|
||||
|
||||
Overall in JDownloader. You can enable overview panel that will show number of links/packages. I just need number of downloadlist/linkgrabber to be able to say that is has to to with number of links or must be some kind of memleak.
__________________
JD-Dev & Server-Admin |
#38
|
||||
|
||||
PS: you can also write in german
__________________
JD-Dev & Server-Admin |
#39
|
|||
|
|||
Quote:
download has at about 3'500 folders with 1,2 million image links linkgrabber has 30'000 folders with 1,6 million image links... |
#40
|
|||
|
|||
does it makes sense to save as DLC and re-import when ready to download?
|
Thread Tools | |
Display Modes | |
|
|