[pspzockerscene]

I have the owner- and audio-ID but I do not know the request to get fresh links...
Via API, permissions are needed and we don't have them!

[djmakinera]

pspzockerscene -

API permissions - auth API with "audio" permission

[pspzockerscene]

I know.
Please stop posting parts of the puzzle.
If your "friend" knows something, either ask him to post it here or post it for him but please post something that actually helps us...

[djmakinera]

Spoiler:

I'm sorry :\ I write your post to him and my friend to answer your questions. I hope you understand?
Write the message (it would be better if you will converse with him on the Social Network.
If possible, write to him to chat:
Send a question here:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

No - you can do this as you're the one who needs the feature.

[djmakinera]

Wrote only this:

"scope" param
vk.com/dev/auth_mobile

Nothing more to help you with this. I do not have more information on it. Sorry!

[djmakinera]

Forum problem write "Russian name"
I added the name here
http://jdownloader.org/pastebin/133300


**External links are only visible to Support Staff****External links are only visible to Support Staff**

JD2 save file:

Spoiler:

[pspzockerscene]

What is the problem here?

[djmakinera]

Account is missing or "try restarting, plugin defect?"
Some link - not work (audio)

05.03.15 14.08.20to05.03.15 14.10.51 jdlog://3095196763241/
**External links are only visible to Support Staff****External links are only visible to Support Staff**


Edit: 6.3.2015

Hi. pspzockerscne -

decrypt + add links <4800 Mega RAM ----> add1 link **External links are only visible to Support Staff****External links are only visible to Support Staff**
"C:\Program Files\Java\jre1.8.0_31\bin\javaw.exe" -Xmx4800M -jar Y:\JDownloader3\JDownloader.jar


Idle - 700MB - 1700 MB

high memory usage is only for decrypt + add links to a larger package
Avoid freezing applications and javaw crash
Please understand well that this problem only occurs when parse/crawling walls of the large amount of posts (100k post)
GUI crashed but still works in the background to decrypt and probably go to perform fully

Decrypted offset 120600 / 120602
05.03.15 18.46.47to05.03.15 23.25.47 jdlog://5277196763241

(Number of of links does not matter * Screenshot number 2)
Maybe to reduce memory usage CHANGE CODE?

Reached 5000 links per run limit
}
this.logger.info("Decrypted offset " + currentOffset + " / " + total_numberof_entries);
this.logger.info("Found " + this.decryptedLinks.size() + " links so far");
if (this.decryptedLinks.size() >= 5000)
{
this.logger.info("Reached 5000 links per run limit -> Returning link to continue");
DownloadLink loopBack = createDownloadlink(this.CRYPTEDLINK_FUNCTIONAL + "-maxoffset=" + total_numberof_entries + "-currentoffset=" + currentOffset);
fp.add(loopBack);
this.decryptedLinks.add(loopBack);
break;
}
currentOffset += 100;

Hmm? Reduced to:

Adding 50000+ links to one package

Up to 10,000 pictures per package
10000 + 10000 + 10000 + 10000 + 10000 ^.^ SPLIT

[pspzockerscene]

Hm I only tested it up to 25 000 links.
Memory usage increased from 190 to 550 MB and CPU stayed under 15%.
I fixed the "defect" bug for audio links.

[djmakinera]

When add a package to a single 30K-50K - high memory usage - 4.8 G and Out Of Memory...
Java not responding

Example:


400K link but small multi-packets = no problems
50K - 80K One or more very large package = the problem (but only if crawler added links to the package)

[pspzockerscene]

Please provide logs and wait for a response of jiaz.

[djmakinera]

I do not have logs. I deleted the logs.
I would have to analyze the walls again ... and create a NEW log

[pspzockerscene]

Well then please do this.

@jiaz
Just in case you check this: Simply use my vk account - you ll find it somewhere in closed vk related tickets.

[djmakinera]

Real memory usage (percentage or number)

always on top

If the load GUI - it is not possible to operate and can not be selected
"About JDownloader" and check memory usage.

Example:
In this picture - another program shows different values at the bottom of the window

[djmakinera]

FROZEN GUI


Private logfile:
Link Download:
http://jdownloader.org/pastebin/133312

[pspzockerscene]

Please wait for jiaz.
He will either help you tomorrow or on monday.

[djmakinera]

Okay...

Please see on the video.

Link Download VIDEO
http://jdownloader.org/pastebin/133313

[Jiaz]

JDownloader should not happen to "not possible to operate, not be selected".
I would like to check via JVisualvm and find out why it has so much memory need in your case.

[djmakinera]

See a video
Analysis time of 13 minutes and stopping critical.

logfile:
http://jdownloader.org/pastebin/133312

video:
http://jdownloader.org/pastebin/133313

Example link:

vk account - you ll find it somewhere in closed vk related tickets.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Soryy, I do not have a computer Java JDK.
But you can now reproduce the problem.

[pspzockerscene]

Merged vk threads.
dj please do NOT make new threads for vk stuff - your high memory usage also happens only via vk (I guess) so please simply stay in this thread.

GreeZ psp

[djmakinera]

I understand I will write only in this topic

50K = ~ 5 GB RAM :D

Spoiler:

[editestowy]

Originally Posted by djmakinera

Soryy, I do not have a computer Java JDK.
But you can now reproduce the problem.

You do not need to install JDK. I can create portable JDK and with standalone JVisualVM you can help Jiaz to track the problem.
Let me know if you need this then I'll give you further instructions.

[djmakinera]

pspzockerscene - "An Error occured"
**External links are only visible to Support Staff****External links are only visible to Support Staff**
07.03.15 11.29.35to07.03.15 11.30.39 jdlog://2860296763241/




editestowy -I downloaded and installed, and what do I do next?
http://www.oracle.com/technetwork/ja...s-2133151.html

Spoiler:

[pspzockerscene]

@djmakinera
I cannot find any "defect" errors in your latest log.
There are a lot of "file not found" errors but this is no bug.
Please add a screenshot.

GreeZ psp

[djmakinera]

There are a lot of "file not found" errors

Yes it is true.
Screen is not required because all files "File Not Found" (jd2 OFFLINE)

Via browser - Online

[pspzockerscene]

As said, audio links can time out...

[djmakinera]

Hm, but it should be possible to resume...

Spoiler:

[pspzockerscene]

Indeed this might work.
I'll check this once I got time.

[pspzockerscene]

Implemented 2nd handling to refresh timed out audio-directlinks.
You will have to re-add your audiolinks to check if this works.
---

[Please read]=>How/when do I get updates?
Read it carefully BEFORE complaining that something would still be not working!
Update cycles depend on your installed JD version!!

---
[Bitte lesen]=>Wie/wann bekomme ich Updates?
Bitte vollständig lesen bevor ihr euch beschwert, dass etwas immernoch nicht funktionieren würde!
Die Updatezyklen variieren je nach installierter JD Version!!

GreeZ pspzockerscene

[djmakinera]

Mr. King Of The Plug - For the tests - I will add links today and I tomorrow see(check) if the session has not expired;)
Kay ?;)

[pspzockerscene]

Yes this should be okay fir testing.
In my tests however, it worked well

[djmakinera]

CrawlerPlugin out of date



PluginErrors.log

--ID:883TS:1425994708555-3/10/15 2:38:28 PM - [jd.plugins.PluginForDecrypt(errLog)] -> CrawlerPlugin out of date: jd.plugins.decrypter.VKontakteRu@5741884a :29472

--ID:883TS:1425994708556-3/10/15 2:38:28 PM - [jd.plugins.PluginForDecrypt(errLog)] -> URL was: **External links are only visible to Support Staff****External links are only visible to Support Staff**

[djmakinera]

Very Old problem. Back again?
Or the fault of the new JD2 or plugin or change site VK?
Issue: Download files


19.03.15 10.20.50to19.03.15 13.02.52 jdlog://9629209066241/
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**





------------------------Thread: 34364:vkontakte.ru.log-----------------------
--ID:34364TS:1426766605854-3/19/15 1:03:25 PM - [] -> jd.plugins.PluginException: Too many requests in a short time
at jd.plugins.hoster.VKontakteRuHoster.generalErrorhandling(VKontakteRuHoster.java:391)
at jd.plugins.hoster.VKontakteRuHoster.getPageSafe(VKontakteRuHoster.java:436)
at jd.plugins.hoster.VKontakteRuHoster.requestFileInformation(VKontakteRuHoster.java:288)
at jd.plugins.hoster.VKontakteRuHoster.handleFree(VKontakteRuHoster.java:447)
at jd.plugins.PluginForHost.handle(PluginForHost.java:813)
at jd.controlling.downloadcontroller.SingleDownloadController.download(SingleDownloadController.java:38 5)
at jd.controlling.downloadcontroller.SingleDownloadController.run(SingleDownloadController.java:561)

--ID:34364TS:1426766605854-3/19/15 1:03:25 PM - [jd.controlling.downloadcontroller.SingleDownloadController(download)] -> Exception:
--ID:34364TS:1426766605854-3/19/15 1:03:25 PM - [] -> jd.plugins.PluginException: Too many requests in a short time
at jd.plugins.hoster.VKontakteRuHoster.generalErrorhandling(VKontakteRuHoster.java:391)
at jd.plugins.hoster.VKontakteRuHoster.getPageSafe(VKontakteRuHoster.java:436)
at jd.plugins.hoster.VKontakteRuHoster.requestFileInformation(VKontakteRuHoster.java:288)
at jd.plugins.hoster.VKontakteRuHoster.handleFree(VKontakteRuHoster.java:447)
at jd.plugins.PluginForHost.handle(PluginForHost.java:813)
at jd.controlling.downloadcontroller.SingleDownloadController.download(SingleDownloadController.java:38 5)
at jd.controlling.downloadcontroller.SingleDownloadController.run(SingleDownloadController.java:561)

[pspzockerscene]

Doesn't matter - JD will wait and retry in 1 minute!

[djmakinera]

I know what is causing the problem!!
Downloading files that no longer exist on the page (File Not Found)
Too Quickly check links = causes a short block


**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

1. Fixed vk.com/photos-1234...
2. "Too many requests in a short time" = NO PROBLEM = JD will wait a minute and retry!

GreeZ psp

[djmakinera]

Not work for me:
http://www.imagebam.com/image/b9d05c398486112

[pspzockerscene]

Use less max simultan downloads!

[djmakinera]

1 simultan -
0B/s - does not start download files.






jd.plugins.PluginException: Too many requests in a short time
at jd.plugins.hoster.VKontakteRuHoster.generalErrorhandling(VKontakteRuHoster.java:391)
at jd.plugins.hoster.VKontakteRuHoster.getPageSafe(VKontakteRuHoster.java:436)
at jd.plugins.hoster.VKontakteRuHoster.requestFileInformation(VKontakteRuHoster.java:288)
at jd.plugins.hoster.VKontakteRuHoster.handleFree(VKontakteRuHoster.java:447)
at jd.plugins.PluginForHost.handle(PluginForHost.java:813)
at jd.controlling.downloadcontroller.SingleDownloadController.download(SingleDownloadController.java:38 5)
at jd.controlling.downloadcontroller.SingleDownloadController.run(SingleDownloadController.java:561)

java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at jd.controlling.downloadcontroller.DownloadWatchDog.localFileCheck(DownloadWatchDog.java:3881)
at jd.plugins.download.DownloadLinkDownloadable.checkIfWeCanWrite(DownloadLinkDownloadable.java:173)
at jd.plugins.download.raf.OldRAFDownload.startDownload(OldRAFDownload.java:525)
at jd.plugins.hoster.VKontakteRuHoster.doFree(VKontakteRuHoster.java:332)
at jd.plugins.hoster.VKontakteRuHoster.handleFree(VKontakteRuHoster.java:448)
at jd.plugins.PluginForHost.handle(PluginForHost.java:813)
at jd.controlling.downloadcontroller.SingleDownloadController.download(SingleDownloadController.java:38 5)
at jd.controlling.downloadcontroller.SingleDownloadController.run(SingleDownloadController.java:561)