[Solved] import csv/txt file with 100'000 links - Page 2 - JDownloader Community

Jiaz · #41 21.03.2017, 18:51

It makes absolute NO sense to export/import

You have about 2,8 mio links!?! Then I suggest to provide more memory

-Xmx12g

raztoki · #42 22.03.2017, 01:05

it comes to a point when you can't just keep adding more memory
you need to remove/cleanup links that have been downloaded to free up memory.
each link requires specific amount of memory.

The other solution is to only add links when you have (near on) finished the previous group.

my 2 cents

raztoki

verheiratet1952 · #43 22.03.2017, 02:45

Quote:

Originally Posted by raztoki

it comes to a point when you can't just keep adding more memory
you need to remove/cleanup links that have been downloaded to free up memory.
each link requires specific amount of memory.

The other solution is to only add links when you have (near on) finished the previous group.

my 2 cents

raztoki

well, that makes me believe I can create DLC from added links, remove them and add them again with DLC...

by the way: -Xmx12g seems to work with almost 2.9 million links

raztoki · #44 22.03.2017, 02:57

dlc is just another container like csv with more downside than upside, its crypted and would require more resources to add links.

As stated earlier I would remove old links when downloaded, I would add new batch of csv when required. This is the best and really the only solution when talking about sheer volume of links you are dealing with.

Also I would recommend that you disable linkchecking in linkgrabber, to speed up adding of links, settings > advanced > LinkCollector.dolinkcheck
This will reduce a lot of overheads with the required browser requests just to check filesize (if your links are directlink you should already know the filename) and online availability. linkchecking will then take place within download routine.

raztoki

Jiaz · #45 22.03.2017, 09:11

You can copy/backup the linkcollector or downloadList zips from cfg folder. They contain the complete list. For example copy the file, then empty the list and later you can import the file as normal container (similiar to dlc). But it keeps all settings and is MUCH MUCH faster as JDownloader doesn't has to process all links again

verheiratet1952 · #46 22.03.2017, 14:32

Quote:

Originally Posted by raztoki

dlc is just another container like csv with more downside than upside, its crypted and would require more resources to add links.

As stated earlier I would remove old links when downloaded, I would add new batch of csv when required. This is the best and really the only solution when talking about sheer volume of links you are dealing with.

Also I would recommend that you disable linkchecking in linkgrabber, to speed up adding of links, settings > advanced > LinkCollector.dolinkcheck
This will reduce a lot of overheads with the required browser requests just to check filesize (if your links are directlink you should already know the filename) and online availability. linkchecking will then take place within download routine.

raztoki

disable linkchecking is a good idea, definitely when doing those 1 million and more links jobs...

-Xmx12g or -Xmx8g I think was best for some massive performance push...

but it's really wicked to see 7,33gb 11,86gb 11,86gb
(screenshot)

what does that mean exactly? why three different specs?

I am just trying to get the most out of it, so I can handle most issues...

verheiratet1952 · #47 22.03.2017, 14:40

Quote:

Originally Posted by Jiaz

You can copy/backup the linkcollector or downloadList zips from cfg folder. They contain the complete list. For example copy the file, then empty the list and later you can import the file as normal container (similiar to dlc). But it keeps all settings and is MUCH MUCH faster as JDownloader doesn't has to process all links again

can you show me more precisely please?
cfg folder?
copy/backup the linkcollector or downloadList zips?

(I was looking for another backup over the years to export/import links than DLC, but I was sure there is no other...)

Jiaz · #48 22.03.2017, 14:53

Quote:

Originally Posted by verheiratet1952

but it's really wicked to see 7,33gb 11,86gb 11,86gb
(screenshot)
what does that mean exactly? why three different specs?

(a)Current heap memory in use - (b)Current allocated heap memory - (c)Max possible heap memory

b should be about the same as taskmanager shows

in case b is near c and a near b, that means that heap memory is nearly full and java will spend more time/cpu on memory management

Jiaz · #49 22.03.2017, 14:55

Quote:

Originally Posted by verheiratet1952

can you show me more precisely please?
cfg folder?
copy/backup the linkcollector or downloadList zips?

(I was looking for another backup over the years to export/import links than DLC, but I was sure there is no other...)

linkcollector zips contain the current complete linkgrabber list
downloadlist zips contain the current complete download list

you can easy copy the files somewhere else and later import them back via drag/drop or load container.

verheiratet1952 · #50 22.03.2017, 15:08

Quote:

Originally Posted by Jiaz

(a)Current heap memory in use - (b)Current allocated heap memory - (c)Max possible heap memory

b should be about the same as taskmanager shows

in case b is near c and a near b, that means that heap memory is nearly full and java will spend more time/cpu on memory management

upgrading to 32gb could be an option

Jiaz · #51 22.03.2017, 15:19

More memory is always an option but no solution

verheiratet1952 · #52 22.03.2017, 15:39

Quote:

Originally Posted by Jiaz

linkcollector zips contain the current complete linkgrabber list
downloadlist zips contain the current complete download list

you can easy copy the files somewhere else and later import them back via drag/drop or load container.

now I got it

...\JDownloader v2.0\cfg

better to close JD first and then copy those ZIPs?

there is more than just one file each? how do I know what files exactly?

there are also .backup files

changing cfg folder to another fast drive does not bring better performance?

Jiaz · #53 22.03.2017, 15:52

You can copy the zip without closing JDownloader. JDownloader uses CopyOnWrite.
The highest number is the current one.
.backup files are auto created in case something went wrong when trying to load the zip
putting jd on faster drive/ssd has no real influence on performance

verheiratet1952 · #54 23.03.2017, 12:02

Quote:

Originally Posted by Jiaz

You can copy the zip without closing JDownloader. JDownloader uses CopyOnWrite.
The highest number is the current one.
.backup files are auto created in case something went wrong when trying to load the zip
putting jd on faster drive/ssd has no real influence on performance

if I collect links with my older desktop computer having 8gb RAM, I can use -Xmx7g ?

Jiaz · #55 23.03.2017, 12:10

Try it and you will see if it works

verheiratet1952 · #56 24.03.2017, 19:58

Quote:

Originally Posted by Jiaz

Try it and you will see if it works

it is telling me "java heap size might be too large"
java dev kit 64bit must be installed?

I am using 64-bit Java for Windows recommended Version 8 Update 121.

(I was trying with -Xmx7g, -Xmx6g, -Xmx4g)

Jiaz · #57 24.03.2017, 20:15

Of course your system must have enough free memory available!
Check about dialog for used java version. JDownloader uses its bundled java version by default.
32Bit JVM only supports up to 2G heap.

verheiratet1952 · #58 25.03.2017, 13:56

Quote:

Originally Posted by Jiaz

It makes absolute NO sense to export/import

You have about 2,8 mio links!?! Then I suggest to provide more memory

-Xmx12g

now it is working fine... with linkcrawler...

something strange is about my csv file that has at about 45'000 lines...
(I cleaned with Excel, removing duplicates, ...)

but JD only detects 37'000 (+3'000 offline), what csv file coding is best?

another issue: 90% from JD shown as offline are not offline !?

by the way: I was re-doing linkcrawler... but there was no fill-up...

raztoki · #59 26.03.2017, 00:44

Quote:

Originally Posted by raztoki

Also I would recommend that you disable linkchecking in linkgrabber, to speed up adding of links, settings > advanced > LinkCollector.dolinkcheck
This will reduce a lot of overheads with the required browser requests just to check filesize (if your links are directlink you should already know the filename) and online availability. linkchecking will then take place within download routine.

raztoki

seems you haven't done this?
you could be getting blocked accessing too often/fast. linkchecking basically does a full download just to test that its downloadable and get the filesize from response header.

If you are also downloading at the time, this could be making it even worse, or lagging out your internet connection which might create socket issues.

raztoki

Jiaz · #60 27.03.2017, 16:20

Quote:

Originally Posted by verheiratet1952

but JD only detects 37'000 (+3'000 offline), what csv file coding is best?

Hard to tell without having access to such a csv file.
JDownloader does dupe checks. So adding the same url will result in only 1 link in list.

verheiratet1952 · #61 28.03.2017, 13:45

Quote:

Originally Posted by Jiaz

Hard to tell without having access to such a csv file.
JDownloader does dupe checks. So adding the same url will result in only 1 link in list.

well, I was removing duplicates with Excel from csv file list...

how to be sure that file coding is correct? it also has special characters from German, Spanish, Turkish, ... language...

Jiaz · #62 28.03.2017, 14:13

Use UTF-8 for Encoding!

verheiratet1952 · #63 29.03.2017, 22:22

Quote:

Originally Posted by Jiaz

Use UTF-8 for Encoding!

how to save all offline links/files at linkcrawler as a simple txt file?
(I can save as DLC, but that does have encryption)

I need offline links/files (that are not really offline) to process further...

Jiaz · #64 30.03.2017, 08:56

At the moment the best solution would be to use the zips from cfg folder. they contain the complete list. in future I will add support to export only selected ones

mgpai · #65 30.03.2017, 10:25

Quote:

Originally Posted by verheiratet1952

how to save all offline links/files at linkcrawler as a simple txt file?

You can use this script.

Code:

// Save 'contentURLs' of selected links to text file.
// Trigger required: "Linkgrabber Contextmenu Button Pressed"
// Forum Topic: https://board.jdownloader.org/showthread.php?t=70907

var saveTo = "c:/myFolder"; // <- Specify an existing folder to to save the text file.

if (name == "Save URLs to text file") /* <-- Name of the button to be used in context menu (case-sensitive) */ {
    var links = lgSelection.getLinks();
    var urls = [];
    var date = new Date().toISOString().replace(/([^T]+).+/, "$1");
    var saveAs = "/" + date + ".txt";

    for (i = 0; i < links.length; i++) {
        var link = links[i];
        var url = link.getContentURL();
        if (url) urls.push(url);
    }
    if (urls.length) writeFile(getPath(saveTo + saveAs), urls.join("\r\n"), true);
}

Instructions:

Add script to event scripter
Go to Linkgrabber Tab > Context Menu > Open Menu Manager
Add Action > EventScripter Trigger
In the right-side panel change the name from "EventScripter Trigger" to the exact name of the button used in the script.
Save

verheiratet1952 · #66 07.04.2017, 18:33

Quote:

Originally Posted by Jiaz

At the moment the best solution would be to use the zips from cfg folder. they contain the complete list. in future I will add support to export only selected ones

if I try to save a DLC from all offline files only there is some "container encryption failed" error !?

Jiaz · #67 07.04.2017, 18:41

See here https://board.jdownloader.org/showthread.php?t=73231

verheiratet1952 · #68 12.04.2017, 12:25

script is starting and is asking me to allow or deny... I press allow... but then I get that error...

line 18 is exactly same like you did send

and I did create folder C:\myFolder on my local drive...

############################

net.sourceforge.htmlunit.corejs.javascript.WrappedException: Wrapped org.jdownloader.extensions.eventscripter.EnvironmentException: Line 18
java.io.IOException: Das System kann den angegebenen Pfad nicht finden
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(Unknown Source)
at org.appwork.utils.IO.writeStringToFile(IO.java:636)
at org.appwork.utils.IO.writeStringToFile(IO.java:625)
at org.jdownloader.extensions.eventscripter.sandboxobjects.ScriptEnvironment.writeFile(ScriptEnvironmen t.java:956)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at net.sourceforge.htmlunit.corejs.javascript.MemberBox.invoke(MemberBox.java:120)
at net.sourceforge.htmlunit.corejs.javascript.NativeJavaMethod.call(NativeJavaMethod.java:213)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1531)
at script(:18)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411)
at org.jdownloader.scripting.JSHtmlUnitPermissionRestricter$SandboxContextFactory.doTopCall(JSHtmlUnitP ermissionRestricter.java:119)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3057)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
at net.sourceforge.htmlunit.corejs.javascript.Context.evaluateString(Context.java:1212)
at org.jdownloader.extensions.eventscripter.ScriptThread.evalUNtrusted(ScriptThread.java:232)
at org.jdownloader.extensions.eventscripter.ScriptThread.executeScipt(ScriptThread.java:126)
at org.jdownloader.extensions.eventscripter.ScriptThread.run(ScriptThread.java:106)
(#18)
at net.sourceforge.htmlunit.corejs.javascript.Context.throwAsScriptRuntimeEx(Context.java:1907)
at net.sourceforge.htmlunit.corejs.javascript.MemberBox.invoke(MemberBox.java:146)
at net.sourceforge.htmlunit.corejs.javascript.NativeJavaMethod.call(NativeJavaMethod.java:213)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1531)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411)
at org.jdownloader.scripting.JSHtmlUnitPermissionRestricter$SandboxContextFactory.doTopCall(JSHtmlUnitP ermissionRestricter.java:119)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3057)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
at net.sourceforge.htmlunit.corejs.javascript.Context.evaluateString(Context.java:1212)
at org.jdownloader.extensions.eventscripter.ScriptThread.evalUNtrusted(ScriptThread.java:232)
at org.jdownloader.extensions.eventscripter.ScriptThread.executeScipt(ScriptThread.java:126)
at org.jdownloader.extensions.eventscripter.ScriptThread.run(ScriptThread.java:106)
Caused by: org.jdownloader.extensions.eventscripter.EnvironmentException: Line 18
java.io.IOException: Das System kann den angegebenen Pfad nicht finden
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(Unknown Source)
at org.appwork.utils.IO.writeStringToFile(IO.java:636)
at org.appwork.utils.IO.writeStringToFile(IO.java:625)
at org.jdownloader.extensions.eventscripter.sandboxobjects.ScriptEnvironment.writeFile(ScriptEnvironmen t.java:956)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at net.sourceforge.htmlunit.corejs.javascript.MemberBox.invoke(MemberBox.java:120)
at net.sourceforge.htmlunit.corejs.javascript.NativeJavaMethod.call(NativeJavaMethod.java:213)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1531)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411)
at org.jdownloader.scripting.JSHtmlUnitPermissionRestricter$SandboxContextFactory.doTopCall(JSHtmlUnitP ermissionRestricter.java:119)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3057)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
at net.sourceforge.htmlunit.corejs.javascript.Context.evaluateString(Context.java:1212)
at org.jdownloader.extensions.eventscripter.ScriptThread.evalUNtrusted(ScriptThread.java:232)
at org.jdownloader.extensions.eventscripter.ScriptThread.executeScipt(ScriptThread.java:126)
at org.jdownloader.extensions.eventscripter.ScriptThread.run(ScriptThread.java:106)

at org.jdownloader.extensions.eventscripter.sandboxobjects.ScriptEnvironment.writeFile(ScriptEnvironmen t.java:962)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at net.sourceforge.htmlunit.corejs.javascript.MemberBox.invoke(MemberBox.java:120)
... 12 more
Caused by: java.io.IOException: Das System kann den angegebenen Pfad nicht finden
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(Unknown Source)
at org.appwork.utils.IO.writeStringToFile(IO.java:636)
at org.appwork.utils.IO.writeStringToFile(IO.java:625)
at org.jdownloader.extensions.eventscripter.sandboxobjects.ScriptEnvironment.writeFile(ScriptEnvironmen t.java:956)
... 17 more

Quote:

Originally Posted by mgpai

You can use this script.

Code:

// Save 'contentURLs' of selected links to text file.
// Trigger required: "Linkgrabber Contextmenu Button Pressed"
// Forum Topic: https://board.jdownloader.org/showthread.php?t=70907

var saveTo = "c:/myFolder"; // <- Specify an existing folder to to save the text file.

if (name == "Save URLs to text file") /* <-- Name of the button to be used in context menu (case-sensitive) */ {
    var links = lgSelection.getLinks();
    var urls = [];
    var date = new Date().toISOString().replace(/([^T]+).+/, "$1");
    var saveAs = "/" + date + ".txt";

    for (i = 0; i < links.length; i++) {
        var link = links[i];
        var url = link.getContentURL();
        if (url) urls.push(url);
    }
    if (urls.length) writeFile(getPath(saveTo + saveAs), urls.join("\r\n"), true);
}

Instructions:

Add script to event scripter
Go to Linkgrabber Tab > Context Menu > Open Menu Manager
Add Action > EventScripter Trigger
In the right-side panel change the name from "EventScripter Trigger" to the exact name of the button used in the script.
Save

mgpai · #69 12.04.2017, 12:57

If you use backslash in path you need to escape it like this.

Code:

var saveTo = "c:\\myFolder"; // <- Specify an existing folder to to save the text file.

Jiaz · #70 12.04.2017, 13:56

Thanks @mgpai for helping with this one

verheiratet1952 · #71 12.04.2017, 16:34

Quote:

Originally Posted by mgpai

If you use backslash in path you need to escape it like this.

Code:

var saveTo = "c:\\myFolder"; // <- Specify an existing folder to to save the text file.

thanks a lot

how to add hour, minute and second to that line?

#######
var saveAs = "/" + date + ".txt";
#######

Jiaz · #72 12.04.2017, 17:05

I suggest to contact/ask mgpai for help

mgpai · #73 12.04.2017, 17:44

UTC Date/Time (Sample Format: 2017-04-13 04.43.59)

Code:

var date = new Date().toISOString().replace(/(.+)T(.+)\..+/, "$1 $2").replace(/:/g,".");

Local Date/Time (Sample Format: Apr 13 2017 08.15.36)

Code:

var date = new Date().toString().replace(/.{4}(.+) G.+/,"$1").replace(/:/g,".");

verheiratet1952 · #74 13.04.2017, 04:27

Quote:

Originally Posted by mgpai

UTC Date/Time

Code:

var date = new Date().toISOString().replace(/([^.]+).+/, "$1").replace(/T/," ").replace(/:/g,".");

Local Date/Time

Code:

var date = new Date().toString().replace(/.{4}(.+) G.+/,"$1").replace(/:/g,".");

adding that line only or replacing old line completely?

mgpai · #75 13.04.2017, 05:25

Replace old line completely, with one of the two new lines, depending on which type of Date/Time you prefer.

verheiratet1952 · #76 21.04.2017, 01:28

what about csv files larger than 5mb? all links will be crawled by linkgrabber?

pspzockerscene · #77 21.04.2017, 02:01

Examples please

GreeZ psp

Jiaz · #78 21.04.2017, 09:45

I will update Linkcrawler to support larger files
After next core update, Settings-Advanced Settings, search for DeepDecryptFileSizeLimit
set max 50 mbyte or -1 for unlimited

verheiratet1952 · #79 21.04.2017, 14:27

Quote:

Originally Posted by Jiaz

I will update Linkcrawler to support larger files
After next core update, Settings-Advanced Settings, search for DeepDecryptFileSizeLimit
set max 50 mbyte or -1 for unlimited

next core update is already scheduled? in a few days, weeks or months?

I am just asking because sometimes it's working with 7-12mb csv files, and sometimes there is nothing happening...

raztoki · #80 21.04.2017, 14:40

its already live,
the change that jiaz uploaded to implement those features was
https://svn.jdownloader.org/projects...evisions/36645
and last build full was on https://svn.jdownloader.org/projects...evisions/36649
you can usually see/monitor here, but there are projects used which do not have redmine front end https://svn.jdownloader.org/projects/jd/repository

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

	JDownloader Community Board - Archive - Top
Provided By AppWork GmbH \| Privacy \| Imprint