JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 31.10.2016, 09:30
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default Possible improve the speed of the addition of a single or more packages to download?

Possible improve the speed of the addition of a single or more packages to download?
Ps. The current jdownloader installation on a fast SSD
500K-1M links | Current setting -Xmx7GB
The duration of the addition of one or more package ~1 minutes ~1,5 minutes
Reply With Quote
  #2  
Old 31.10.2016, 11:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default


Speed highly depends on number of links in package.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 31.10.2016, 11:46
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Quote:
Speed highly depends on number of links in package.
Not always:

TEST

1000 links (1 package) = FAST ADD
7000+ links (1 package) = SLOW ADD
Links (6 + 11 + 18 + 14 + 12) (5 package) = SLOW ADD
Reply With Quote
  #4  
Old 31.10.2016, 12:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

You talked about adding package and NOT crawling!
Crawling highly depends on the plugin and limits. If this is about VK plugin then we can stop right here!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 31.10.2016, 12:15
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Add packages from [TAB Linkgrabber] to [TAB Download], very clearly I expressed itself.
Reply With Quote
  #6  
Old 31.10.2016, 12:19
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

And how many Links/Packages are within your Linkgrabber/DownloadList?
If there are many then I already linked the correct ticket
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 31.10.2016, 12:24
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Above 200k - 1000k (LinkGrabber list)
The problem only affect "Linkgrabber list" (does not apply to "Download list")
Reply With Quote
  #8  
Old 31.10.2016, 12:30
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

Linkgrabber List has to maintain a duplicate map. so very long list = more time required.
That's exactly what the linked ticket is about
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 31.10.2016, 12:54
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

This problem is always present, even after the new installation, and even if it contains only one link and one package (in Linkgrabber)
VERRRRYYYY SMALL LIST!!!

It occurs only in the case of a domain Vk!

Last edited by Jiaz; 31.10.2016 at 13:30.
Reply With Quote
  #10  
Old 31.10.2016, 13:30
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

The stacktrace shows that the zip file is empty and not correct written. Either filesystem error or you killed JDownloader during shutdown. And again this has NOTHING to do with the thread topic!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 31.10.2016, 13:58
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Quote:
1. Either filesystem error
2. killed JDownloader during shutdown
Not true !
Every Run or restart JD2 is the same error.

I checked it on various system configurations
1. Acer (Laptop) XP/Vista
2. and PC Desktop Win 7

My friend is also checked and said that for him the same error.

The error occurs only if the domain VK and a ONLY new version/build of JD2


In all the old versions (test many different installation), for example. 2013, 2014 (No error!!!)
Reply With Quote
  #12  
Old 31.10.2016, 14:36
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

1. Either filesystem error
2. killed JDownloader during shutdown

I tested domain
Code:
zippyshare.com, and various other domain
a) I killed the process = No error !!!
b) cutting off Power Supply (!) = No error !!!

I tested domain
Code:
vk.com
(only one 1 link)
a)Correct Exit or Restart = ALWAYS Error
b)Without power off = ALWAYS Error


So what's going on here? Where there is an Internal Error in the new version JD2?
Reply With Quote
  #13  
Old 31.10.2016, 16:11
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

Please provide a single VK link for testing.
And PLEASE post in the correct thread! This is the wrong thread!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 31.10.2016, 16:15
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

This error
Empty/Invalid Zip:C:\JDownloader2\cfg\linkcollector1416.zip|Size:0
Signals that the Zip file is empty.
When there are issues during saving the file will be deleted and there are traces in logfile.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 31.10.2016, 16:21
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

I found an old topic of August 2016, so you could move:
https://board.jdownloader.org/showpo...3&postcount=38

Restart or Run
repeating a 1000 times restart or run
generate 1000 errors in the log

1 Run or Restart = 1 Error Found
1000 Run or Restart = 1000 Error Found
only the number of erroneous archive change (each time a different number)
Reply With Quote
  #16  
Old 31.10.2016, 16:24
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

Works perfectly fine for me.
I can offer you a teamviewer session about this. Just send me ID and PW to support@jdownloader.org
__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 31.10.2016, 16:59
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Jiaz - Now I know what causes this error.
The startup parameters causing this error: Empty/Invalid Zip
Code:
"C:\Program Files\Java\jre1.8.0_112\bin\javaw.exe" -Xdiag -Xms2G -Xmx7G -jar C:\Jdownloader2\JDownloader.jar
Please correct the startup parameters that do not cause this problem because without parameters, not enough memory allocated (Especially for a large list of links and stability GUI)
Reply With Quote
  #18  
Old 31.10.2016, 17:06
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

The zip file is created during saving and not during loading
Nothing to fix in JDownloader. It simply fails to save the list because it is out of memory!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #19  
Old 31.10.2016, 17:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

I will reduce memory usage during saving the list. Wait for next core update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 31.10.2016, 17:29
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

Reduced memory usage during linklist saving, please check again with next core update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #21  
Old 31.10.2016, 17:49
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

True, during the closing JD2 (process javaw.exe uses much more RAM)
The application (process javaw.exe) for a long time before it closes probably save)
Reply With Quote
  #22  
Old 31.10.2016, 18:28
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Quote:
It simply fails to save the list because it is out of memory!
How much need to reserve physical RAM to run jdownloader with
Code:
[jd.controlling.linkcollector.LinkCollector$16(run)
CollectorList found: 10564/1157667

java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.<init>(Unknown Source)
at jd.controlling.linkcollector.LinkCollector$23.run(LinkCollector.java:2315)
at jd.controlling.linkcollector.LinkCollector$23.run(LinkCollector.java:2249)
at org.appwork.utils.event.queue.QueueAction.start(QueueAction.java:202)
at org.appwork.utils.event.queue.Queue.startItem(Queue.java:491)
at org.appwork.utils.event.queue.Queue.runQueue(Queue.java:425)
at org.appwork.utils.event.queue.QueueThread.run(QueueThread.java:64)
Reply With Quote
  #23  
Old 31.10.2016, 18:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

Thanks for the logfile. Now I know what you mean, Please update next core update in a few minutes
__________________
JD-Dev & Server-Admin
Reply With Quote
  #24  
Old 01.11.2016, 13:46
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default BUG: New update (31-10-16) causes the loss of 1000,000+ links

Quote:
Jiaz: Reduced memory usage during linklist saving, please check again with next core update
New update - causes the loss of links (!!!)
1 mln - 1,4 mln links
Update causes damage to the archives!

Previous Size "LinkCollector": 657 MB

Now
1,9 MB !!! zip.backup
9 MB linkcollector.zip

Previously in the previous version JD2 (example date 30-10.16) NEVER damage archive even if forcibly cut off power supply (so better!)
Old Archive rename larger number, but in the future again and again and again cause damage. A very serious bug!

Reply With Quote
  #25  
Old 01.11.2016, 22:04
editestowy's Avatar
editestowy editestowy is offline
Polish Supporter
 
Join Date: May 2009
Location: PL
Posts: 3,026
Default

Not confirmed - your screenshot doesn't prove anything.
How do we know that you didn't delete them by yourself?

Here are mine list - before (21:47) and after (21:51) latest core update:


Nothing was lost after updating, filesize was updated after I've moved 2 packages into Downloads.

Last edited by raztoki; 01.11.2016 at 23:41.
Reply With Quote
  #26  
Old 02.11.2016, 05:36
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

editestowy -

1. Misread my previous post!!!
Linkcollector!!!
(NOT(!) DownloadList
2. You tested with 1,000,000 - 1,400,000 Link??? 600 - 700 MB ! Certainly not! So I do not understand your answer!

Jiaz had to update the core (was supposed to reduce the use of creating a backup that caused damage to the archive eg. After a long terminate process javaw.exe (more than 60 seconds TimeOut! Socket Exception Error)

I do not like the new update
https://board.jdownloader.org/showpo...5&postcount=19
https://board.jdownloader.org/showpo...6&postcount=20
because it bring catastrophic consequences.
I'm going back to the old version. I'm sorry! I'm telling the truth.

Reproduce
a) 1+ mln Links (600-700 MB)
b) Restart JD2
WARNING: Unexpected end of archive 0000 (broken) (1 MB)
Reply With Quote
  #27  
Old 02.11.2016, 18:47
editestowy's Avatar
editestowy editestowy is offline
Polish Supporter
 
Join Date: May 2009
Location: PL
Posts: 3,026
Default

Quote:
Originally Posted by djmakinera View Post
editestowy -

1. Misread my previous post!!!
Linkcollector!!!
(NOT(!) DownloadList
2. You tested with 1,000,000 - 1,400,000 Link??? 600 - 700 MB ! Certainly not! So I do not understand your answer!
Did you read and understand my post??
"Nothing was lost after updating, filesize was updated after I've moved 2 packages into Downloads."
Obviously if packaged were moved into Downloads they were in Linkcollector list! Nor Download neither Linkcollector list was broken/lost.
As Jiaz wrote in the previous post - nothing wrong in his test. So the problem is only on your instance...
Reply With Quote
  #28  
Old 02.11.2016, 10:51
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

02.11.16 10.50.48 <--> 02.11.16 10.51.03 jdlog://3681881887641/
Reply With Quote
  #29  
Old 02.11.2016, 14:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

Here is the answer
https://board.jdownloader.org/showpo...2&postcount=11
__________________
JD-Dev & Server-Admin
Reply With Quote
  #30  
Old 02.11.2016, 14:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

Please answer my question in the link
__________________
JD-Dev & Server-Admin
Reply With Quote
  #31  
Old 02.11.2016, 19:03
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

editestowy - I tested with a smaller archive file is fine, but with a very large archive file is an unexpected end.
TimeOut: Limit exceeded 60 seconds.
It requires an increase.
Example: 180 second or 5 minut
Quote:
java.net.SocketException: socket closed
Reply With Quote
  #32  
Old 02.11.2016, 21:13
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Edit by Jiaz: how about providing the zip file?!

Last edited by Jiaz; 03.11.2016 at 10:28.
Reply With Quote
Old 03.11.2016, 10:27
Jiaz
Message deleted by Jiaz.
  #34  
Old 01.11.2016, 13:57
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default Serious Bug JD2 (31.10.2016 - 1.11.2016)

Previous Version: (...) (...) - 30.10.2016 work
In previous logs is not has performed this entry, now find all the time

New:
This causes damage to the removal and proper archives (even old archives !!!)
All the time the loss of links

Code:
------------------------Thread: 95:Log.L.log-----------------------
--ID:95TS:1478004548042-01.11.16 13:49:08 -  [org.appwork.shutdown.ShutdownController$3(run)] -> Exit Now: Code: 0
------------------------Thread: 21:Log.L.log-----------------------
--ID:21TS:1478004548421-01.11.16 13:49:08 -  [] -> java.net.SocketException: socket closed
	at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
	at java.net.PlainSocketImpl.accept(Unknown Source)
	at java.net.ServerSocket.implAccept(Unknown Source)
	at java.net.ServerSocket.accept(Unknown Source)
	at org.appwork.utils.singleapp.SingleAppInstance$1.run(SingleAppInstance.java:364)
	at java.lang.Thread.run(Unknown Source)
Reply With Quote
  #35  
Old 02.11.2016, 10:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

Please provide complete logfile. Nothing changed and that you have links loss is the same reason for your other bugreport about those empty/zero size files. It is the exact same cause.
The Stracktrace you've provided has nothing to do with this
__________________
JD-Dev & Server-Admin
Reply With Quote
  #36  
Old 02.11.2016, 20:32
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Quote:
Originally Posted by Jiaz View Post
Please provide complete logfile.
NON-STOP damaged list!
1.15 M -> now only 139K
Probably now handle only 139K links

The full debug-log.
02.11.16 20.28.10 <--> 02.11.16 20.28.33 jdlog://9222881887641/

1. I close an application JD2 with very large archives of large packages after thousands of VK links
2. Again Run JD2 and damaged archive /corrupted LinkCollector List WTF?!
Reply With Quote
  #37  
Old 02.11.2016, 10:19
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

Quote:
Originally Posted by djmakinera View Post
This causes damage to the removal and proper archives (even old archives !!!)
Please stop telling lies. Only difference is that instead of writing everything to memory first and then one big write to disk (which caused the high memory usage for you) it is now written with smaller buffer. So in fact it is the same issue you suffer all the time that the shutdown has not enough time to write list and instead of having a zero size file (before) you now have a half finished file.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #38  
Old 02.11.2016, 10:38
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Quote:
Originally Posted by Jiaz View Post
Please stop telling lies.
I'm not a liar. It's too big a word. :-(

Reply With Quote
  #39  
Old 02.11.2016, 10:28
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

f I'm lying, please withdraw this update. I repeat once again, please remove this update. I do not need an update, which causes damage to (the loss of links) Thanks.
Previously, no damage archive LinkCollector (only bakup) 0byte.
That's all from my side!
I do not have the strength to prove everything every time (this is my personal opinions / testing)
Sorry for wszytsko. I'm just experiencing problems so he writes.
You do not believe me. Excuse me :(
Reply With Quote
  #40  
Old 02.11.2016, 13:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,415
Default

Quote:
Originally Posted by djmakinera View Post
Previously, no damage archive LinkCollector (only bakup) 0byte.
Are you kidding me. I will not change it back. Because before it also did not work for you. a 0byte file is the same (broken) as a half written fie (broken).

The question is, why is the file broken/not finished. So do you close JDownloader normally or kill it (eg on computer shutdown)?
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 15:10.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.