#41
|
||||
|
||||
It makes absolute NO sense to export/import
You have about 2,8 mio links!?! Then I suggest to provide more memory -Xmx12g
__________________
JD-Dev & Server-Admin |
#42
|
||||
|
||||
it comes to a point when you can't just keep adding more memory
you need to remove/cleanup links that have been downloaded to free up memory. each link requires specific amount of memory. The other solution is to only add links when you have (near on) finished the previous group. my 2 cents raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#43
|
|||
|
|||
Quote:
well, that makes me believe I can create DLC from added links, remove them and add them again with DLC... by the way: -Xmx12g seems to work with almost 2.9 million links |
#44
|
||||
|
||||
dlc is just another container like csv with more downside than upside, its crypted and would require more resources to add links.
As stated earlier I would remove old links when downloaded, I would add new batch of csv when required. This is the best and really the only solution when talking about sheer volume of links you are dealing with. Also I would recommend that you disable linkchecking in linkgrabber, to speed up adding of links, settings > advanced > LinkCollector.dolinkcheck This will reduce a lot of overheads with the required browser requests just to check filesize (if your links are directlink you should already know the filename) and online availability. linkchecking will then take place within download routine. raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] Last edited by raztoki; 22.03.2017 at 03:11. Reason: downloadside/downside |
#45
|
||||
|
||||
You can copy/backup the linkcollector or downloadList zips from cfg folder. They contain the complete list. For example copy the file, then empty the list and later you can import the file as normal container (similiar to dlc). But it keeps all settings and is MUCH MUCH faster as JDownloader doesn't has to process all links again
__________________
JD-Dev & Server-Admin |
#46
|
|||
|
|||
Quote:
disable linkchecking is a good idea, definitely when doing those 1 million and more links jobs... -Xmx12g or -Xmx8g I think was best for some massive performance push... but it's really wicked to see 7,33gb 11,86gb 11,86gb (screenshot) what does that mean exactly? why three different specs? I am just trying to get the most out of it, so I can handle most issues... |
#47
|
|||
|
|||
Quote:
cfg folder? copy/backup the linkcollector or downloadList zips? (I was looking for another backup over the years to export/import links than DLC, but I was sure there is no other...) |
#48
|
||||
|
||||
Quote:
b should be about the same as taskmanager shows in case b is near c and a near b, that means that heap memory is nearly full and java will spend more time/cpu on memory management
__________________
JD-Dev & Server-Admin |
#49
|
||||
|
||||
Quote:
downloadlist zips contain the current complete download list you can easy copy the files somewhere else and later import them back via drag/drop or load container.
__________________
JD-Dev & Server-Admin |
#50
|
|||
|
|||
Quote:
upgrading to 32gb could be an option |
#51
|
||||
|
||||
More memory is always an option but no solution
__________________
JD-Dev & Server-Admin |
#52
|
|||
|
|||
Quote:
...\JDownloader v2.0\cfg better to close JD first and then copy those ZIPs? there is more than just one file each? how do I know what files exactly? there are also .backup files changing cfg folder to another fast drive does not bring better performance? |
#53
|
||||
|
||||
You can copy the zip without closing JDownloader. JDownloader uses CopyOnWrite.
The highest number is the current one. .backup files are auto created in case something went wrong when trying to load the zip putting jd on faster drive/ssd has no real influence on performance
__________________
JD-Dev & Server-Admin |
#54
|
|||
|
|||
Quote:
if I collect links with my older desktop computer having 8gb RAM, I can use -Xmx7g ? |
#55
|
||||
|
||||
Try it and you will see if it works
__________________
JD-Dev & Server-Admin |
#56
|
|||
|
|||
it is telling me "java heap size might be too large"
java dev kit 64bit must be installed? I am using 64-bit Java for Windows recommended Version 8 Update 121. (I was trying with -Xmx7g, -Xmx6g, -Xmx4g) |
#57
|
||||
|
||||
Of course your system must have enough free memory available!
Check about dialog for used java version. JDownloader uses its bundled java version by default. 32Bit JVM only supports up to 2G heap.
__________________
JD-Dev & Server-Admin |
#58
|
|||
|
|||
Quote:
something strange is about my csv file that has at about 45'000 lines... (I cleaned with Excel, removing duplicates, ...) but JD only detects 37'000 (+3'000 offline), what csv file coding is best? another issue: 90% from JD shown as offline are not offline !? by the way: I was re-doing linkcrawler... but there was no fill-up... |
#59
|
||||
|
||||
Quote:
you could be getting blocked accessing too often/fast. linkchecking basically does a full download just to test that its downloadable and get the filesize from response header. If you are also downloading at the time, this could be making it even worse, or lagging out your internet connection which might create socket issues. raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#60
|
||||
|
||||
Quote:
JDownloader does dupe checks. So adding the same url will result in only 1 link in list.
__________________
JD-Dev & Server-Admin |
#61
|
|||
|
|||
Quote:
well, I was removing duplicates with Excel from csv file list... how to be sure that file coding is correct? it also has special characters from German, Spanish, Turkish, ... language... |
#62
|
||||
|
||||
Use UTF-8 for Encoding!
__________________
JD-Dev & Server-Admin |
#63
|
|||
|
|||
how to save all offline links/files at linkcrawler as a simple txt file?
(I can save as DLC, but that does have encryption) I need offline links/files (that are not really offline) to process further... |
#64
|
||||
|
||||
At the moment the best solution would be to use the zips from cfg folder. they contain the complete list. in future I will add support to export only selected ones
__________________
JD-Dev & Server-Admin |
#65
|
|||
|
|||
Quote:
Code:
// Save 'contentURLs' of selected links to text file. // Trigger required: "Linkgrabber Contextmenu Button Pressed" // Forum Topic: https://board.jdownloader.org/showthread.php?t=70907 var saveTo = "c:/myFolder"; // <- Specify an existing folder to to save the text file. if (name == "Save URLs to text file") /* <-- Name of the button to be used in context menu (case-sensitive) */ { var links = lgSelection.getLinks(); var urls = []; var date = new Date().toISOString().replace(/([^T]+).+/, "$1"); var saveAs = "/" + date + ".txt"; for (i = 0; i < links.length; i++) { var link = links[i]; var url = link.getContentURL(); if (url) urls.push(url); } if (urls.length) writeFile(getPath(saveTo + saveAs), urls.join("\r\n"), true); } Instructions:
|
#66
|
|||
|
|||
if I try to save a DLC from all offline files only there is some "container encryption failed" error !?
|
#67
|
||||
|
||||
__________________
JD-Dev & Server-Admin |
#68
|
|||
|
|||
script is starting and is asking me to allow or deny... I press allow... but then I get that error...
line 18 is exactly same like you did send and I did create folder C:\myFolder on my local drive... ############################ net.sourceforge.htmlunit.corejs.javascript.WrappedException: Wrapped org.jdownloader.extensions.eventscripter.EnvironmentException: Line 18 java.io.IOException: Das System kann den angegebenen Pfad nicht finden at java.io.WinNTFileSystem.createFileExclusively(Native Method) at java.io.File.createNewFile(Unknown Source) at org.appwork.utils.IO.writeStringToFile(IO.java:636) at org.appwork.utils.IO.writeStringToFile(IO.java:625) at org.jdownloader.extensions.eventscripter.sandboxobjects.ScriptEnvironment.writeFile(ScriptEnvironmen t.java:956) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at net.sourceforge.htmlunit.corejs.javascript.MemberBox.invoke(MemberBox.java:120) at net.sourceforge.htmlunit.corejs.javascript.NativeJavaMethod.call(NativeJavaMethod.java:213) at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1531) at script(:18) at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798) at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105) at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411) at org.jdownloader.scripting.JSHtmlUnitPermissionRestricter$SandboxContextFactory.doTopCall(JSHtmlUnitP ermissionRestricter.java:119) at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3057) at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115) at net.sourceforge.htmlunit.corejs.javascript.Context.evaluateString(Context.java:1212) at org.jdownloader.extensions.eventscripter.ScriptThread.evalUNtrusted(ScriptThread.java:232) at org.jdownloader.extensions.eventscripter.ScriptThread.executeScipt(ScriptThread.java:126) at org.jdownloader.extensions.eventscripter.ScriptThread.run(ScriptThread.java:106) (#18) at net.sourceforge.htmlunit.corejs.javascript.Context.throwAsScriptRuntimeEx(Context.java:1907) at net.sourceforge.htmlunit.corejs.javascript.MemberBox.invoke(MemberBox.java:146) at net.sourceforge.htmlunit.corejs.javascript.NativeJavaMethod.call(NativeJavaMethod.java:213) at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1531) at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798) at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105) at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411) at org.jdownloader.scripting.JSHtmlUnitPermissionRestricter$SandboxContextFactory.doTopCall(JSHtmlUnitP ermissionRestricter.java:119) at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3057) at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115) at net.sourceforge.htmlunit.corejs.javascript.Context.evaluateString(Context.java:1212) at org.jdownloader.extensions.eventscripter.ScriptThread.evalUNtrusted(ScriptThread.java:232) at org.jdownloader.extensions.eventscripter.ScriptThread.executeScipt(ScriptThread.java:126) at org.jdownloader.extensions.eventscripter.ScriptThread.run(ScriptThread.java:106) Caused by: org.jdownloader.extensions.eventscripter.EnvironmentException: Line 18 java.io.IOException: Das System kann den angegebenen Pfad nicht finden at java.io.WinNTFileSystem.createFileExclusively(Native Method) at java.io.File.createNewFile(Unknown Source) at org.appwork.utils.IO.writeStringToFile(IO.java:636) at org.appwork.utils.IO.writeStringToFile(IO.java:625) at org.jdownloader.extensions.eventscripter.sandboxobjects.ScriptEnvironment.writeFile(ScriptEnvironmen t.java:956) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at net.sourceforge.htmlunit.corejs.javascript.MemberBox.invoke(MemberBox.java:120) at net.sourceforge.htmlunit.corejs.javascript.NativeJavaMethod.call(NativeJavaMethod.java:213) at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1531) at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798) at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105) at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411) at org.jdownloader.scripting.JSHtmlUnitPermissionRestricter$SandboxContextFactory.doTopCall(JSHtmlUnitP ermissionRestricter.java:119) at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3057) at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115) at net.sourceforge.htmlunit.corejs.javascript.Context.evaluateString(Context.java:1212) at org.jdownloader.extensions.eventscripter.ScriptThread.evalUNtrusted(ScriptThread.java:232) at org.jdownloader.extensions.eventscripter.ScriptThread.executeScipt(ScriptThread.java:126) at org.jdownloader.extensions.eventscripter.ScriptThread.run(ScriptThread.java:106) at org.jdownloader.extensions.eventscripter.sandboxobjects.ScriptEnvironment.writeFile(ScriptEnvironmen t.java:962) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at net.sourceforge.htmlunit.corejs.javascript.MemberBox.invoke(MemberBox.java:120) ... 12 more Caused by: java.io.IOException: Das System kann den angegebenen Pfad nicht finden at java.io.WinNTFileSystem.createFileExclusively(Native Method) at java.io.File.createNewFile(Unknown Source) at org.appwork.utils.IO.writeStringToFile(IO.java:636) at org.appwork.utils.IO.writeStringToFile(IO.java:625) at org.jdownloader.extensions.eventscripter.sandboxobjects.ScriptEnvironment.writeFile(ScriptEnvironmen t.java:956) ... 17 more Quote:
|
#69
|
|||
|
|||
If you use backslash in path you need to escape it like this.
Code:
var saveTo = "c:\\myFolder"; // <- Specify an existing folder to to save the text file. |
#70
|
||||
|
||||
Thanks @mgpai for helping with this one
__________________
JD-Dev & Server-Admin |
#71
|
|||
|
|||
Quote:
thanks a lot how to add hour, minute and second to that line? ####### var saveAs = "/" + date + ".txt"; ####### |
#72
|
||||
|
||||
I suggest to contact/ask mgpai for help
__________________
JD-Dev & Server-Admin |
#73
|
|||
|
|||
UTC Date/Time (Sample Format: 2017-04-13 04.43.59)
Code:
var date = new Date().toISOString().replace(/(.+)T(.+)\..+/, "$1 $2").replace(/:/g,"."); Local Date/Time (Sample Format: Apr 13 2017 08.15.36) Code:
var date = new Date().toString().replace(/.{4}(.+) G.+/,"$1").replace(/:/g,"."); Last edited by mgpai; 13.04.2017 at 06:49. Reason: Optimized regex and added sample formats. |
#74
|
|||
|
|||
adding that line only or replacing old line completely?
|
#75
|
|||
|
|||
Replace old line completely, with one of the two new lines, depending on which type of Date/Time you prefer.
|
#76
|
|||
|
|||
what about csv files larger than 5mb? all links will be crawled by linkgrabber?
|
#77
|
||||
|
||||
Examples please
GreeZ psp
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#78
|
||||
|
||||
I will update Linkcrawler to support larger files
After next core update, Settings-Advanced Settings, search for DeepDecryptFileSizeLimit set max 50 mbyte or -1 for unlimited
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 21.04.2017 at 09:47. |
#79
|
|||
|
|||
Quote:
I am just asking because sometimes it's working with 7-12mb csv files, and sometimes there is nothing happening... |
#80
|
||||
|
||||
its already live,
the change that jiaz uploaded to implement those features was https://svn.jdownloader.org/projects...evisions/36645 and last build full was on https://svn.jdownloader.org/projects...evisions/36649 you can usually see/monitor here, but there are projects used which do not have redmine front end https://svn.jdownloader.org/projects/jd/repository
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
|
|