Java Xms vs Xmx parameters - you probably did it wrong:
stackoverflow.com/questions/5374455/what-does-java-option-xmx-stand-for
stackoverflow.com/questions/1098488/jvm-heap-parameters
I was able to start JD with max ~1500 MB of max memory.
Trying to use 8GB, I got the following error and JD refused to start:
... so as long as your value hasn't changed, you probably only set "Xms" but not "Xmx".
Try e.g. "-Xmx1500m" and you should see a change of the max. value in the about dialog.
-psp-