#121
|
|||
|
|||
BUFFER = 4,13 GB C:\pagefile.sys
Quote:
Total Memory 10231 MB Used Memory 9000 MB (90%) Available Memory (RAM) 1000 MB (10%) Total Pagefile 14464 MB Used Pagefile 120017 MB (83%) Available Pagefile 24443 MB (17%) Used Virtual Memory 143 MB / 4096 MB (3%) Last edited by djmakinera; 22.05.2015 at 16:59. |
#122
|
||||
|
||||
You should find out what eats the pageFile! I suggest to use process explorer to see very detailed infos. 12GB Swap is EXTREM! JD Only uses 3,3GB in your Screenshots
__________________
JD-Dev & Server-Admin |
#123
|
|||
|
|||
Jiaz - You are wrong!
javaw.exe = MEMORY LEAK Last edited by djmakinera; 22.05.2015 at 17:47. |
#124
|
||||
|
||||
5,84 GB + Java -> 6,4GB..no memleak here. We can do teamviewer if you like. Send ID and PW via email
__________________
JD-Dev & Server-Admin |
#125
|
|||
|
|||
More memory, because the files are cached in memory
RamMap: File summary DownloadList 200MB + 200M + 200MB + 200MB + log (MB) + tmp + OTHER = 1GB - 2GB RAM(Cashe) Quote:
What is a Metafile (Screen 2)? To solve the problem of high memory usage and I used this script helps :D Code:
# Filename: setfc.ps1 $version = 1.1 ######################### # Settings ######################### # The percentage of physical ram that will be used for SetSystemFileCache Maximum $MaxPercent = 12.5 ######################### # Init multipliers ######################### $OSBits = ([System.IntPtr]::Size) * 8 switch ( $OSBits) { 32 { $KiB = [int]1024 } 64 { $KiB = [long]1024 } default { # not 32 or 64 bit OS. what are you doing?? $KiB = 1024 # and hope it works anyway write-output "You have a weird OS which is $OSBits bit. Having a go anyway." } } # These values "inherit" the data type from $KiB $MiB = 1024 * $KiB $GiB = 1024 * $MiB $TiB = 1024 * $GiB $PiB = 1024 * $TiB $EiB = 1024 * $PiB ######################### # Calculated Settings ######################### # Note that because we are using signed integers instead of unsigned # these values are "limited" to 2 GiB or 8 EiB for 32/64 bit OSes respectively $PhysicalRam = 0 $PhysicalRam = [long](invoke-expression (((get-wmiobject -class "win32_physicalmemory").Capacity) -join '+')) if ( -not $? ) { write-output "Trying another method of detecting amount of installed RAM." } if ($PhysicalRam -eq 0) { $PhysicalRam = [long]((Get-WmiObject -Class Win32_ComputerSystem).TotalPhysicalMemory) # gives value a bit less than actual } if ($PhysicalRam -eq 0) { write-error "Cannot Detect Physical Ram Installed. Assuming 4 GiB." $PhysicalRam = 4 * $GiB } $NewMax = [long]($PhysicalRam * 0.01 * $MaxPercent) # The default value # $NewMax = 1 * $TiB ######################### # constants ######################### # Flags bits $FILE_CACHE_MAX_HARD_ENABLE = 1 $FILE_CACHE_MAX_HARD_DISABLE = 2 $FILE_CACHE_MIN_HARD_ENABLE = 4 $FILE_CACHE_MIN_HARD_DISABLE = 8 ################################ # C# code # for interface to kernel32.dll ################################ $source = @" using System; using System.Runtime.InteropServices; namespace MyTools { public static class cache { [DllImport("kernel32", SetLastError = true, CharSet = CharSet.Unicode)] public static extern bool GetSystemFileCacheSize( ref IntPtr lpMinimumFileCacheSize, ref IntPtr lpMaximumFileCacheSize, ref IntPtr lpFlags ); [DllImport("kernel32", SetLastError = true, CharSet = CharSet.Unicode)] public static extern bool SetSystemFileCacheSize( IntPtr MinimumFileCacheSize, IntPtr MaximumFileCacheSize, Int32 Flags ); [DllImport("kernel32", CharSet = CharSet.Unicode)] public static extern int GetLastError(); public static bool Get( ref IntPtr a, ref IntPtr c, ref IntPtr d ) { IntPtr lpMinimumFileCacheSize = IntPtr.Zero; IntPtr lpMaximumFileCacheSize = IntPtr.Zero; IntPtr lpFlags = IntPtr.Zero; bool b = GetSystemFileCacheSize(ref lpMinimumFileCacheSize, ref lpMaximumFileCacheSize, ref lpFlags); a = lpMinimumFileCacheSize; c = lpMaximumFileCacheSize; d = lpFlags; return b; } public static bool Set( IntPtr MinimumFileCacheSize, IntPtr MaximumFileCacheSize, Int32 Flags ) { bool b = SetSystemFileCacheSize( MinimumFileCacheSize, MaximumFileCacheSize, Flags ); if ( !b ) { Console.Write("SetSystemFileCacheSize returned Error with GetLastError = "); Console.WriteLine( GetLastError() ); } return b; } } public class AdjPriv { [DllImport("advapi32.dll", ExactSpelling = true, SetLastError = true)] internal static extern bool AdjustTokenPrivileges(IntPtr htok, bool disall, ref TokPriv1Luid newst, int len, IntPtr prev, IntPtr relen); [DllImport("advapi32.dll", ExactSpelling = true, SetLastError = true)] internal static extern bool OpenProcessToken(IntPtr h, int acc, ref IntPtr phtok); [DllImport("advapi32.dll", SetLastError = true)] internal static extern bool LookupPrivilegeValue(string host, string name, ref long pluid); [StructLayout(LayoutKind.Sequential, Pack = 1)] internal struct TokPriv1Luid { public int Count; public long Luid; public int Attr; } internal const int SE_PRIVILEGE_ENABLED = 0x00000002; internal const int SE_PRIVILEGE_DISABLED = 0x00000000; internal const int TOKEN_QUERY = 0x00000008; internal const int TOKEN_ADJUST_PRIVILEGES = 0x00000020; public static bool EnablePrivilege(long processHandle, string privilege, bool disable) { bool retVal; TokPriv1Luid tp; IntPtr hproc = new IntPtr(processHandle); IntPtr htok = IntPtr.Zero; retVal = OpenProcessToken(hproc, TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, ref htok); tp.Count = 1; tp.Luid = 0; if(disable) { tp.Attr = SE_PRIVILEGE_DISABLED; } else { tp.Attr = SE_PRIVILEGE_ENABLED; } retVal = LookupPrivilegeValue(null, privilege, ref tp.Luid); retVal = AdjustTokenPrivileges(htok, false, ref tp, 0, IntPtr.Zero, IntPtr.Zero); return retVal; } } } "@ # Add the c# code to the powershell type definitions Add-Type -TypeDefinition $source -Language CSharp ######################### # Powershell Functions ######################### function output-flags ($flags) { Write-output ("FILE_CACHE_MAX_HARD_ENABLE : " + (($flags -band $FILE_CACHE_MAX_HARD_ENABLE) -gt 0) ) Write-output ("FILE_CACHE_MAX_HARD_DISABLE : " + (($flags -band $FILE_CACHE_MAX_HARD_DISABLE) -gt 0) ) Write-output ("FILE_CACHE_MIN_HARD_ENABLE : " + (($flags -band $FILE_CACHE_MIN_HARD_ENABLE) -gt 0) ) Write-output ("FILE_CACHE_MIN_HARD_DISABLE : " + (($flags -band $FILE_CACHE_MIN_HARD_DISABLE) -gt 0) ) write-output "" } ######################### # Main program ######################### write-output "" ######################### # Get and set privilege info $ProcessId = $pid $processHandle = (Get-Process -id $ProcessId).Handle $Privilege = "SeIncreaseQuotaPrivilege" $Disable = $false Write-output ("Enabling SE_INCREASE_QUOTA_NAME status: " + [MyTools.AdjPriv]::EnablePrivilege($processHandle, $Privilege, $Disable) ) write-output ("Program has elevated privledges: " + ([Security.Principal.WindowsPrincipal] [Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] "Administrator") ) write-output "" whoami /PRIV | findstr /I "SeIncreaseQuotaPrivilege" | findstr /I "Enabled" if ( -not $? ) { write-error "user Security Token SE_INCREASE_QUOTA_NAME: Disabled`r`n" } write-output "`r`n" ######################### # Get Current Settings # Init variables $SFCMin = 0 $SFCMax = 0 $SFCFlags = 0 #Get Current values from kernel $status = [MyTools.cache]::Get( [ref]$SFCMin, [ref]$SFCMax, [ref]$SFCFlags ) #typecast values so we can do some math with them $SFCMin = [long]$SFCMin $SFCMax = [long]$SFCMax $SFCFlags = [long]$SFCFlags write-output "Return values from GetSystemFileCacheSize are: " write-output "Function Result : $status" write-output " Min : $SFCMin" write-output (" Max : $SFCMax ( " + $SFCMax / 1024 / 1024 / 1024 + " GiB )") write-output " Flags : $SFCFlags" output-flags $SFCFlags ######################### # Output our intentions write-output ("Physical Memory Detected : $PhysicalRam ( " + $PhysicalRam / $GiB + " GiB )") write-output ("Setting Max to " + $MaxPercent + "% : $NewMax ( " + $NewMax / $MiB + " MiB )`r`n") ######################### # Set new settings $SFCFlags = $SFCFlags -bor $FILE_CACHE_MAX_HARD_ENABLE # set max enabled $SFCFlags = $SFCFlags -band (-bnot $FILE_CACHE_MAX_HARD_DISABLE) # unset max dissabled if set # or if you want to override this calculated value # $SFCFlags = 0 $status = [MyTools.cache]::Set( $SFCMin, $NewMax, $SFCFlags ) # calls the c# routine that makes the kernel API call write-output "Set function returned: $status`r`n" # if it was successfull the new SystemFileCache maximum will be NewMax if ( $status ) { $SFCMax = $NewMax } ######################### # After setting the new values, get them back from the system to confirm # Re-Init variables $SFCMin = 0 $SFCMax = 0 $SFCFlags = 0 #Get Current values from kernel $status = [MyTools.cache]::Get( [ref]$SFCMin, [ref]$SFCMax, [ref]$SFCFlags ) #typecast values so we can do some math with them $SFCMin = [long]$SFCMin $SFCMax = [long]$SFCMax $SFCFlags = [long]$SFCFlags write-output "Return values from GetSystemFileCacheSize are: " write-output "Function Result : $status" write-output " Min : $SFCMin" write-output (" Max : $SFCMax ( " + $SFCMax / 1024 / 1024 / 1024 + " GiB )") write-output " Flags : $SFCFlags" output-flags $SFCFlags Last edited by Lram32; 20.09.2015 at 03:22. Reason: merged multiple posts |
#126
|
||||
|
||||
Metafile is part of the system cache and consists of NTFS metadata. NTFS metadata includes the MFT as well as the other various NTFS metadata files… In the MFT each file attribute record takes 1KB and each file has at least one attribute record. Add to this the other NTFS metadata files and you can see why the Metafile category can grow quite large on servers with lots of files
-> Many files on disk -> LARGE Metafile This has nothing to do with JDownloader!! The more Files you actively have in your Filesystem, the more memory Windows will need! Also the number of links <-> memory usage has nothing to say. This depends on your memory tuning jvm params.
__________________
JD-Dev & Server-Admin |
#127
|
|||
|
|||
JD2 200,000 downloads and save many files disk and using large Cache... Dont worry I found on the internet solution, solves problem of memory high usage. Now it is all OK... now not uses all available RAM :D
RAM 9GB reduced to 5GB Metafile 2,5GB reduced to 440 MB Last edited by djmakinera; 26.05.2015 at 11:07. |
#128
|
||||
|
||||
I guess you just limited the cache sizes?
__________________
JD-Dev & Server-Admin |
#129
|
|||
|
|||
@Jiaz - Yeah.
|
#130
|
||||
|
||||
Then I only need logfile about non removal of links when finished.
__________________
JD-Dev & Server-Admin |
#131
|
|||
|
|||
I am not sure what you mean.
|
#132
|
||||
|
||||
Bug issue 2:
Remove finished downloads: immediately (NOT ALWAYS WORK) Need a logfile from this. When a download finishes but is not removed
__________________
JD-Dev & Server-Admin |
#133
|
|||
|
|||
This time, there is no problem.
If this error occur again - I'll post the log. It seems to me that it was caused by manipulation in the "Download Management -> Remove finished downloads: Immediately & at startup & when package is ready". Please note that in the case of a large list of JD2 restarts or shuts down for a long time. |
#134
|
||||
|
||||
shutdown and startup can also download/install/apply updates also, which can take time.
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#135
|
||||
|
||||
I will still optimize for large lists, wait for next commit(s)
__________________
JD-Dev & Server-Admin |
#136
|
|||
|
|||
JD2 reminded me that JD2 "kept in the cache" all archives "downloadListXXXX", which, depending on the number of links can have of several Megabyte-Gigabyte. Now I set limit and is not used to this RAM, just kept on my hard drive
|
#137
|
||||
|
||||
;)
__________________
JD-Dev & Server-Admin |
#138
|
|||
|
|||
problem there again - I have no idea why Metafile is so big:crying:
Disable Security ESET SMART SECURITY 8 Last edited by djmakinera; 27.05.2015 at 19:52. |
#139
|
||||
|
||||
MetaFile is NOT cache of files. Its the cache of filesystem. The more files you have on disk, the more memory is needed!
A Filesystem with many files need of course more memory. Any idea how many files your have on your disks?
__________________
JD-Dev & Server-Admin |
#140
|
|||
|
|||
For Windows XP / Vista and Windows Server 2008 (Many millions of files) - can be reduced effectively use by
Quote:
Used script should be used only 12.5% of RAM, but probably not always work. Why on Windows Server - can have millions, millions and millions of files - and DynCashe - working and Windows 7 - it will not work? Last edited by djmakinera; 27.05.2015 at 20:41. |
#141
|
||||
|
||||
Maybe Windows SERVER -> server for many files
Normal Desktop PCs normally dont have many(millions) of files.
__________________
JD-Dev & Server-Admin |
#142
|
|||
|
|||
you're wrong.
|
#143
|
||||
|
||||
Then this page should also show how to "fix" it?
__________________
JD-Dev & Server-Admin |
#144
|
|||
|
|||
Jiaz - On the Microsoft website or blog where this issue is discussed - there is no solution for Windows 7 - because these are unique situations.
**External links are only visible to Support Staff****External links are only visible to Support Staff** Last edited by djmakinera; 27.05.2015 at 20:55. |
#145
|
||||
|
||||
Maybe try to move your files into more folders instead of directories with many files in it. This could help
__________________
JD-Dev & Server-Admin |
#146
|
|||
|
|||
Quote:
It seems to me that Everything 64 bit (application) or or other tool that reads the MFT, even if closed application, cache is not flush fully. |
#147
|
||||
|
||||
Cache can never be fully flushed(emptied) because at least every open file/dll/app needs its data in cache. The more files are scanned/read/opened the bigger the cache will be. Thats why I asked you to spread your many files into more folders, so less "many files" folders -> less cache needed
__________________
JD-Dev & Server-Admin |
#148
|
|||
|
|||
500.000 files
11.000 folders 66GB Last edited by djmakinera; 28.05.2015 at 11:12. |
#149
|
||||
|
||||
maybe less files in more folder can help?
__________________
JD-Dev & Server-Admin |
#150
|
|||
|
|||
Jiaz - you're very funny
Windows XP/Vista is better than Windows 7 ? Windows XP/Vista/Server 2008/Server 2012 - 50 million files, DynCashe - the service and the parameters allow for maximum reduction Metafile (use only 30-400 MB) Windows 7 - no? (use 2-4 GB) Maybe it's better to install Vista ^.^ Last edited by djmakinera; 28.05.2015 at 13:04. |
#151
|
||||
|
||||
Or Try Windows 8.1
__________________
JD-Dev & Server-Admin |
#152
|
|||
|
|||
Iddle: 2 GB RAM
Run JD2 93 K links = 8GB RAM Close JD2 - still use high RAM (Not force flushing cashe?) JD2 not clear the cache when close the application javaw.exe MemLeak only Vk download Quote:
"C:\Program Files\Java\jre1.8.0_45\bin\javaw.exe" -Xmx4G -XX:MaxHeapFreeRatio=X -XX:MinHeapFreeRatio=X -jar V:\Jdownloader2\JDownloader.jar ------------------------- JVM is greedy, sometimes you need to run major GC twice to get the memory released? only GC that gives the memory back system is the G1GC ? "C:\Program Files\Java\jre1.8.0_45\bin\javaw.exe" -Xmx4G -XX:+UseG1GC -jar V:\Jdownloader2\JDownloader.jar Last edited by Lram32; 20.09.2015 at 03:20. Reason: merged multiple posts |
#153
|
||||
|
||||
All GC release memory, but they work complete different. So of course on GC needs more memory to work than another
__________________
JD-Dev & Server-Admin |
#154
|
|||
|
|||
Run Java VisualVM
I clicked on "Perform GC" Memory consumption has increased from 4GB to 9+ GB Why? |
#155
|
||||
|
||||
Parallel GC -> will double memory, work on new memory, then cleanup old one and then shrink.
Each GC has different advantages/disadvantages depending on your needs and stop the world timeouts
__________________
JD-Dev & Server-Admin |
#156
|
|||
|
|||
Not work
Quote:
Java Virtual Machine Launcher --------------------------- Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. --------------------------- Last edited by djmakinera; 28.05.2015 at 19:23. |
#157
|
||||
|
||||
how should max 20 but min 40 work
40(wished minimum) is bigger than 20(wished maximum)
__________________
JD-Dev & Server-Admin |
#158
|
|||
|
|||
Gui black
Gui - black
sometimes... does not always Nothing works. After killing the process and JD2 re-start to normal. Package: 31 Links: 42K logfile: 28.05.15 20.55.16to28.05.15 20.58.42 jdlog://9679119980341/ Last edited by djmakinera; 28.05.2015 at 21:33. |
#159
|
||||
|
||||
of course you cannot simply set two values and expect to work fine. with too aggressive params you force java to free/allocate memory and no more time to work on jdownloader.
Just set -Xmx and you are fine.
__________________
JD-Dev & Server-Admin |
#160
|
||||
|
||||
-XX:MaxHeapFreeRatio=100, -XX:MinHeapFreeRatio=0
Really..what do you think you are doing? min = 0...no wonder java is spending all time with memory handling. please ignore those values unless you know what they are doing!
__________________
JD-Dev & Server-Admin |
|
|