[Jiaz]

@Neurotoxin001: thanks for the report. will be fixed with next update

[Student im ersten Jahr]

Packagename - Current problem.
The important character "_" disappears

Example:
12345 12345
Shouldbe:
12345_12345

Example link decrypt:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[Jiaz]

@Student im ersten Jahr: disable Settings->Advanced Settings->GeneralSettings.cleanupfilenames

[StefanM]

VK.com problem: Analysis

VK.com plugin is working again, but when I add e.g. 100 video links to LinkGrabber, only 20 to 30 % will be found. I can add those links once again and let LinkGrabber grab again, a few more links are found. Have to repeat this several times until all links are found/grabbed.

Reason is: VK.com introduced a change which will refuse answering too many requests in a short period of time.

I can use Extreme Picture Finder, configure a random delay, between the requests/downloads, and everything works fine.

To solve the problem in JD, also here a delay has to be introduced. This is the only solution I found so far.

[Jiaz]

@StefanM: Please don't open multiple threads with same topic!
JDownloader already has a default wait of 1 secs for pagination/crawling (see Settings->Plugins->vk.com)
Please provide a debug log, see https://support.jdownloader.org/Know...d-session-logs
Enable Debug first, see Settings->Advanced Settings->Log.debugmodeenabled
and restart JDownloader. Now wait for issue to happen and then create log and post shown logID here
vk.com now has a WAF (web application firewall) that we added support for recently and I could not reproduce any issues. So please provide a log, so we can see why it fails for you

[elvis_ef]

Hello.

I have few galleries on my list.
Unfortunately with number of download files (that I can get in the same time) set for instance to 5, VK says immediately that I want files to be downloaded too quickly. That the time is too short.

I wonder if there is a setting, that can make JD download files with any download transfer limit or sth ? But... only for galleries, image galleries. Videos must be downloaded without such settings. The thing is videos have no such errors while downloading. Only images.

These galleries have 3k images, 4k images.. but some have 860, or just 430.. the same error occurs with all of them. Only with these images, not videos.

Link (gallery) example:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Merged vk threads.

@elvis_ef
We got one generic vk.com thread.
Please do not open new threads for vk.com related issues.

To answer your question:
No there is no general limits you can apply.

We will simply have to update the plugin in order to recognize that limit so it can wait- and try again later.

Please post your log-ID here | bitte poste deine Log-ID hier.

-psp-
EDIT

Auto-retry-handling for "Too many requests" has already been implemented.
Currently it will wait 60 seconds for all vk.com URLs and continue afterwards...

[StefanM]

Originally Posted by Jiaz

@StefanM: Please don't open multiple threads with same topic!
JDownloader already has a default wait of 1 secs for pagination/crawling (see Settings->Plugins->vk.com)
...
vk.com now has a WAF (web application firewall) that we added support for recently and I could not reproduce any issues. So please provide a log, so we can see why it fails for you

Thanks for your reply. Did not see it, because instant email notification was disabled again, even though I did enable it :-(

Will conduct a few tests and then send you results and more details later this week.

[StefanM]

Originally Posted by Jiaz

@StefanM:
JDownloader already has a default wait of 1 secs for pagination/crawling (see Settings->Plugins->vk.com)
...

Please find a detailed test documentation attached as zip archive. I invested some 90 minutes for that documentation. Hope, it helps solving the issue.

Excerpt as plain text:

Test documentation

Notes:
I'm logged in with my VK.com credentials during this test.
The behavior shown here is typical. Had dozens of cases such as this one - on two different machines using two different Internet providers. Both of them were 16 Mbps accounts.

How the test was run: (I'm quite convinced, you will be able to reproduce this)
First, I copied all 2,166 video links from this example page: **External links are only visible to Support Staff****External links are only visible to Support Staff** - grabbed with LinkGopher in Firefox.
You will find all 2,166 links in Sheet 1 of the attached Excel book. In sheet 2 you will find all links found by LinkGrabber in the first run.
For the next runs, each time I copied all 2,166 links again to LinkGrabber. I used the order as in the Excel book.
In sheet 3 you will find all additional links found by LinkGrabber in the second run - i.e. only those links, LinkGrabber did not find in the first run. Same applies to the third run…
1st run: 863 found
2nd run: another 468 found
3rd run: another 310 found
4th run: another 210 found
total 1,851 links found after 4 runs, would need a few more to find all online links.

In plugin settings, I left those 'sleep time' settings alone, as default that is.

But I noticed, that there is no 1 second delay between grabbing individual video links. As you can see from the attached bubble screenshots (time taken and number of online links found), more than just one link is grabbed within a second. E.g. 973 links found online in 13:34 (814 seconds). This is just an average value. I often see more than 3 or 4 links added as online within one single second.
Maybe I misunderstand that setting? If yes, can you please explain what this 'sleep time' is about.

[Jiaz]

Originally Posted by StefanM

Thanks for your reply.

Nothing to thank me for. And please understand and take it personal that I did delete your other post. Just please don't open multiple threads/posts about the same topic. We see your post


@StefanM: It would be nice if you could just provide a debug log. Please understand that it's much FASTER and easier to have read to analyze logs that show the issue than us having to reproduce the issue and wait for it to happen. So instead of spending 90 minutes in documentation, just reproduce the issue once (crawler abort) and then create the log.
Also you should not crawl/find the individual links by yourself because adding 2166 links will result in minimum 2166 requests (likely even more) . You should instead just let JDownloader crawl/find the links as it can find multiiple links with less requests. Crawler in JDownloader will be much faster than adding thousands of individual links by yourself where no optimization (api with multiple results) can take place.

Originally Posted by StefanM

But I noticed, that there is no 1 second delay between grabbing individual video links.

This delay is for the crawler plugin! pagination. But you're adding single/individual links.
vk.com has a generic request limit of 500 ms between each request and max 15 requests every 30 secs

Browser.setBurstRequestIntervalLimitGlobal("vk.com", 500, 15, 30000);

so the issue is more likely caused by some error/errormessage that causes the whole crawler plugin to fail and that's why we're asking for a debug log.

[pspzockerscene]

Another explanation for the wording "pagination" and our related setting "Sleep time for general pagination":
Pagination is when a website either contains multiple items per visible page or e.g. uses "endless scrolling" like vk.com does so every time you scroll down more, X new items get loaded.
Our setting defines the wait-time in between.

By using "Link gopher" to manually collect all single video items, the following things will happen:

• Crawling all videos one by one will take much longer
• --> It will also cause more requests so the chance of running into any limits is higher
• Pagination will never happen thus that setting is irrelevant as long as you add links this way

Is there any good reason why you do not let JD crawl this "wall" link?

vk.com/video/@CENSORED

?

Letting JD do the job you now did by hand using "Link Gopher" should make things much faster/easier.

Nevertheless I'm currently adding all of these 2166 items from your excel sheet to my test-JD and I will post my results here afterwards.

-psp-
EDIT

Here are my results:
Packages: 1676
Links: 1730
Links online: 1679

EDIT2

Here are my results when I let our crawler do the job (with pagination etc...):
Packages: 6 (much less packages because all vk.com self-hosted videos of that profile will be grouped intop one package named after the username)
Links: 2527
Links online: 2466

[StefanM]

Originally Posted by Jiaz

@StefanM: It would be nice if you could just provide a debug log. Please understand that it's much FASTER and easier to have read to analyze logs that show the issue than us having to reproduce the issue and wait for it to happen.

Thanks for all the information given.
But couldn't you just add those links I sent you to JD's LinkGrabber and see for yourself what happens?

Opening Excel, copying the links from sheet 1, and adding them to LinkGrabber wouldn't take more than a minute.

And then you can see for yourself what happens.

I add those video links as single links for simple reasons:

1. In JD, I can only see the individual links in my download list, wenn I add them separately.

2. I can only have the link as part of the downloaded video file name, when I use individual links.

3. In most cases I don't want to download all videos. Here it was just a test.

4. And maybe later - e.g. after more videos have been uploaded - again I only want to download a few of them.

Now, once again my little request:
Before I create a log with sensitive data (maybe alleged copy right infringements, ...) could you please just add those links to your JD and see if you can reproduce the problem?

It won't take you more than a minute and then waiting what happens.
I would really appreciate this!

I can tell you that the problem did not exist before the VK.com change. And I can tell you, that this problem does not exist, when I use Extreme Picture Finder (**External links are only visible to Support Staff****External links are only visible to Support Staff**). I have a working template which worked before and after the changes VK.com made.

This template also selects the highest resolution only, and the log provides me with the direct links, which are valid for some 5 days.

No problem occurs, when I feed those links to LinkGrabber. Just the file naming won't work then.

Addendum:
When I let JD crawl the video page, it stops crawling after 1,001 links.

[Jiaz]

@StefanM: I can reproduce the issue but I did not ran into any plugin issue. For me it stopped at about 1900 found links. The issue must lie somewhere between. That means either some links are getting *lost* from clipboard to plugin or there are some error messages in plugin that are silently caught.
We will do some more testing by tomorrow

Originally Posted by StefanM

Opening Excel, copying the links from sheet 1, and adding them to LinkGrabber wouldn't take more than a minute.

That's correct but due to set request limits in plugin, it takes a while to finish and as our assumption was about a plugin issue, it would be MUCH faster if you could just provide a log that contains the issue. But as we reproced without any issue, a log won't help and we have to dig deeper into the plugin

Originally Posted by StefanM

could you please just add those links to your JD and see if you can reproduce the problem?

pspzockerscene and I already did hence my comments above

[Jiaz]

Originally Posted by StefanM

When I let JD crawl the video page, it stops crawling after 1,001 links.

We will first check if we can reproduce the issue ourselves as it worked fine for pspzockerscene.
Just out of interest, did you also do testing with the account disabled in JDownloader?

[StefanM]

Originally Posted by Jiaz

@StefanM: ... The issue must lie somewhere between. That means either some links are getting *lost* from clipboard to plugin or there are some error messages in plugin that are silently caught.
We will do some more testing by tomorrow.

It is not a clipboard issue, as I see all links in LinkGrabber window. I paste them there manually.

And If you look at the different runs in my Excel book, you can see that each run some other links are found.

But everytime, each run, all links are in the LinkGrabber window.

[StefanM]

Originally Posted by Jiaz

We will first check if we can reproduce the issue ourselves as it worked fine for pspzockerscene.
Just out of interest, did you also do testing with the account disabled in JDownloader?

Will check that tomorrow.

And I/we should not not mix up two issues:

1. Pasting individual video links into the LinkGrabber window. (many links are not found, more are found in a second, third, ... run)

2. Pasting the Video gallery link into the LinkGrabber window. (Here Grabbing stops too early, before all links have been found)

[StefanM]

Originally Posted by pspzockerscene

Another explanation for the wording "pagination" and our related setting "Sleep time for general pagination":
Pagination is when a website either contains multiple items per visible page or e.g. uses "endless scrolling" like vk.com does so every time you scroll down more, X new items get loaded.
Our setting defines the wait-time in between...

Sorry about not answering earlier. I simply overlooked your reply.

Will answer all questions in a new post later today.

Just one hint right now. I learned from you that VK.com only accepts 15 requests within a 30 sec interval.

For sure, my test with copying more than 2,000 links to LinkGrabber sent more requests, than 15 per 30 sec.

But I will collect all details now and give you a summary including some analysis later today.

Thanks again for taking my report seriously. :-)

[pspzockerscene]

Originally Posted by StefanM

1. In JD, I can only see the individual links in my download list, wenn I add them separately.

Wrong, see:
https://support.jdownloader.org/Know...iew-added-urls

Originally Posted by StefanM

2. I can only have the link as part of the downloaded video file name, when I use individual links.

I don't understand that.
Please add a more detailed explanation.

Originally Posted by StefanM

3. In most cases I don't want to download all videos. Here it was just a test.
...
4. And maybe later - e.g. after more videos have been uploaded - again I only want to download a few of them.

Valid points.

Originally Posted by StefanM

I can tell you that the problem did not exist before the VK.com change. And I can tell you, that this problem does not exist, when I use Extreme Picture Finder (**External links are only visible to Support Staff****External links are only visible to Support Staff**). I have a working template which worked before and after the changes VK.com made.

Those tools work totally different than JD - comparing them to JDs functionality doesn't make any sense.

Originally Posted by StefanM

2. Pasting the Video gallery link into the LinkGrabber window. (Here Grabbing stops too early, before all links have been found)

No.
It worked fine here.
Please simply provide a log.
We do not process your data in any way.
If you want you can also manually grab your logs from the "logs" folder inside your JD folder, zip them and upload them to a cloud of your choice and delete them once we got them.

Originally Posted by StefanM

Just one hint right now. I learned from you that VK.com only accepts 15 requests within a 30 sec interval.

We've never claimed that.
Afaik vk definitely allows more requests but our plugin limits them atm.

Originally Posted by StefanM

But I will collect all details now and give you a summary including some analysis later today.

Again:
Please provide a log...

-psp-

[StefanM]

Originally Posted by Jiaz

We will first check if we can reproduce the issue ourselves as it worked fine for pspzockerscene.
Just out of interest, did you also do testing with the account disabled in JDownloader?

Additional findings and questions regarding those findings

I'm referring to two different issues:

Issue 1: Pasting individual video links into the LinkGrabber window. (many links are not found, more are found in a second, third, fourth, ... run)

Issue 2: Pasting the Video gallery link into the LinkGrabber window. (Here Grabbing stops too early, before all links have been found)


A) Logged in / not logged in on VK.com (tested with issue 2)

Tested several times on two different machines with two different providers.

Yes, there is a reproducible difference:
When I'm logged in, crawling is much slower and in average only some 50 % of the links are found.
When I'm not logged in, crawling is about twice as fast and in average over 90 % of the links are found.


B) Different providers (tested with issue 2)

Even though provider speed tests show almost the same speed, crawling speed is reproducible different.


C) Your info: VK.com only accepts max. 15 request in 30 sec (tested with issue 1)

While crawling (Finding Links…) I see a number of offline links being reported. This means to me that already in this stage requests are sent to VK.com. E.g. after 3 minutes 547 online and 32 offline links are displayed in the bubble.

Isn't this already 579 requests within 180 seconds?
This is for sure above those 15 accepted requests per 30 sec interval. Can this be the reason for issue 1. Please have a look at the attached Excel book which might give you an explanation. I marked all files by colors, in which run they were found. So you can see when which links were skipped / not found.


D) Browser.setBurstRequestIntervalLimitGlobal("vk.com", 500, 15, 30000)

Is there any way for me to edit those numbers for conducting more tests? Cannot find it in advanced settings.


E) Why do I prefer to copy individual links to LinkGrabber?

It is because of a limitation in JD :-)

1. In JD, I can only see the individual links in my download list, when I add them separately.
   You say "wrong: Wrong, see:
   https://support.jdownloader.org/Know...iew-added-urls
   I'm afraid you are mistaken here. In my test case you can only see **External links are only visible to Support Staff****External links are only visible to Support Staff** - the same video gallery link for each of those 2,000+ videos.
2. I can only have the link as part of the downloaded video file name, when I use individual links.
   More detailed explanation: Example: Video-Name_-12345678_987654321_720p.mp4. The numbers are the numbers from the individual video link. These can be amended to the file name, but only when I paste individual links.
3. In most cases I don't want to download all videos. Those 2,000+ links were just a test.
4. Maybe a few weeks after my download - e.g. after more videos have been uploaded - again I only want to download a few of them.

It is too bad, that you cannot see/copy links of individual videos when using the video gallery link to let HD grab all videos. This is what I also need for archiving. For that I would have to parse DownloadList*.zip files :-(


F) You mentioned reducing number of requests by dupe checking

As pointed out in C) there seem to be many occasions, where requests are being sent in high frequency. Please correct me if I'm wrong.

What I would like to know is, how and when dupe checking is being performed.

E.1) Let's assume, I copy links to files to LinkGrabber, which already had been downloaded earlier. Does JD dupe checker first check DownloadList and LinkCollector zip-files, whether or not they contain one or more of the added links? Or does JD send requests anyway?

E.2) Let's assume I add 20 links to LinkGrabber and links 11…20 are dupes of links 1…10.
Now let's assume, while crawling for links 1…10, JD only finds links 1…5. links 6…10 are not found. Now, what happens with links 11…20? Will JD know that it already found links 11…15? And will JD try to find links 16…20 once again as it had not found links 6…10. Again, please note that 16…20 are identical with 6…10.


G) Dupe checker doesn't work in a consistent manner

I'm mentioning this here, because you were referring to dupe checker. What 'annoys' me is the fact, that - depending on where the dupes are - and what I did with them, they can be displayed or they cannot be displayed (checking the box 'Already in Downloadlist'). Links that were already downloaded won't necessarily show up in the LinkGrabber pane. So, the user won't know, if JD simply 'skipped' those links due to an error or because they already had been downloaded earlier.
Or is there a way to have them displayed as 'already downloaded earlier'?


And one more question regarding bug reports or requests for improvement:

To give you just one more example:
The bubble notifier position settings (in advanced settings) do not work correctly when I use a display scaling different from 100 % - tested on different machines.

Where should/would I report this? In fact, over the years I found quite a few issues, but never took the time to report them here. Maybe I should do that… :-)

Thanks again for any efforts taken. You guys are really doing a great job!

[StefanM]

Originally Posted by pspzockerscene

Wrong, see:
**External links are only visible to Support Staff**...

I don't understand that.
Please add a more detailed explanation...

Will embed answers in the post I just sent by editing it. First answer is already there :-)

More to come within the next hour.

[StefanM]

Originally Posted by pspzockerscene

If you want you can also manually grab your logs from the "logs" folder inside your JD folder, zip them and upload them to a cloud of your choice and delete them once we got them.

Again:
Please provide a log...

-psp-

OK, I'm prepared to do that. Just tell me, which logs and which parts you need:

I see different logs such as:
vk.com_jd.plugins.decrypter.VKontakteRu.log.0 8.648.264 bytes
vk.com_jd.plugins.decrypter.VKontakteRu.log.1 11.201.511 bytes
vk.com_jd.plugins.decrypter.VKontakteRu.log.2 11.131.088 bytes

vk.com_jd.plugins.hoster.VKontakteRuHoster.log.0
vk.com_jd.plugins.hoster.VKontakteRuHoster.log.0.lck
vk.com_jd.plugins.hoster.VKontakteRuHoster.log.1
vk.com_jd.plugins.hoster.VKontakteRuHoster.log.2
vk.com_jd.plugins.hoster.VKontakteRuHoster.log.3
vk.com_jd.plugins.hoster.VKontakteRuHoster.log.4

....

Tons of data... :-)

Before I upload anything, I want to be able to verify what I'm sending. And limiting the amount of data would save me a lot of time.

PS: Worked for an IT company, would even call myself an IT-specialist, and I'm also a computer forensics guy,...

So, please understand that I may be overcautious.

PPS: I'm a good friend of the EPF developer, Maxim from Ukraine. That's why I'm pretty familiar with EPF and also because I'm writing the German GUI for all his programs, same as some of his web pages :-)

[pspzockerscene]

Originally Posted by StefanM

I'm referring to two different issues:
...

Both understood.

Originally Posted by StefanM

Yes, there is a reproducible difference:

Can't confirm.

Originally Posted by StefanM

Isn't this already 579 requests within 180 seconds?

Yap. I haven't denied that.

Originally Posted by StefanM

Can this be the reason for issue 1. Please have a look at the attached Excel book which might give you an explanation.

No. Please provide a log.

Originally Posted by StefanM

Is there any way for me to edit those numbers for conducting more tests? Cannot find it in advanced settings.

Yes and no.
You can grab our source code and edit those as explained here:
https://support.jdownloader.org/Know...up-ide-eclipse
Our currently set limits are not linear.
Here is a quote of the comment of that line of code we referenced:

sets request thresholds based on upper burstable limit. eg. 20(x) requests over 60000(y)[=1min]. Then on after it sets limit betweeninterval.

Originally Posted by StefanM

E) Why do I prefer to copy individual links to LinkGrabber?

It is because of a limitation in JD :-)
...
In JD, I can only see the individual links in my download list, when I add them separately.
You say "wrong: Wrong, see:

I'm afraid you are mistaken here. In my test case you can only see **External links are only visible to Support Staff****External links are only visible to Support Staff** - the same video gallery link for each of those 2,000+ videos.

Again wrong!
The mentioned support article describes what you need to do in order to see the individual links even when adding content via "vk.com/video/@CENSORED".
This is working fine here.

Originally Posted by StefanM

I can only have the link as part of the downloaded video file name, when I use individual links.
More detailed explanation: Example: Video-Name_-12345678_987654321_720p.mp4. The numbers are the numbers from the individual video link. These can be amended to the file name, but only when I past individual links.

You can modify filenames as you wish, see explanation of our Packagizer feature:
https://support.jdownloader.org/Know...the-packagizer
If you still fail to do so, please provide detailed examples (URLs!) so we can help you.

Originally Posted by StefanM

In most cases I don't want to download all videos. Those 2,000+ links were just a test.

Understood.
While we will look into this I just want to point out that e.g. on github.com you will find specific scripts dedicated just for downloading from vk.com. You might be better off using them in this case...

Originally Posted by StefanM

Maybe a few weeks after my download - e.g. after more videos have been uploaded - again I only want to download a few of them.

Understood.

Originally Posted by StefanM

It is too bad, that you cannot see/copy links of individual videos when using the video gallery link to let HD grab all videos. This is what I also need for archiving. For that I would have to parse DownloadList*.zip files :-(

As explained: Yes you can!

Originally Posted by StefanM

What I would like to know is, how and when dupe checking is being performed.
...
Let's assume, I copy links to files to LinkGrabber, which already had been downloaded earlier. Does JD dupe checker first check DownloadList and LinkCollector zip-files, whether or not they contain one or more of the added links? Or does JD send requests anyway?

No links are always re-processed.
In this case, links first go through a crawler to find the video direct-links/external links.
All of this will always be done and causes http requests.
There is no way to avoid this other than maybe using EventScripter scripts especially written for this:
https://support.jdownloader.org/Know...event-scripter

Originally Posted by StefanM

Let's assume I add 20 links to LinkGrabber and links 11…20 are dupes of links 1…10.
Now let's assume, while crawling for links 1…10, JD only finds links 1…5. links 6…10 are not found. Now, what happens with links 11…20? Will JD know that it already found links 11…15? And will JD try to find links 16…20 once again as it had not found links 6…10. Again, please note that 16…20 are identical with 6…10.

While adding links, JD will filter out dupes on the fly so if you add 20 items and 10 of them are contained twice in there, JD will only process 10.

Originally Posted by StefanM

I'm mentioning this here, because you were referring to dupe checker. What 'annoys' me is the fact, that - depending on where the dupes are - and what I did with them, they can be displayed or they cannot be displayed (checking the box 'Already in Downloadlist'). Links that were already downloaded won't necessarily show up in the LinkGrabber pane. So, the user won't know, if JD simply 'skipped' those links due to an error or because they already had been downloaded earlier.
Or is there a way to have them displayed as 'already downloaded earlier'?

Depends.
First of all, JD only checks against items that are still in your downloadlist.
If you e.g. remove them from your downloadlist but the files themselves still exist in your download destination, JD will not detect those as dupes. This makes sense because of multiple factors that I won't explain right now.
A user once wrote an EventScripter script for enhanced dupe checking.
That might be helpful for you too:
https://board.jdownloader.org/showthread.php?t=83048

Originally Posted by StefanM

To give you just one more example:
The bubble notifier position settings (in advanced settings) do not work correctly when I use a display scaling different from 100 % - tested on different machines.

Where should/would I report this? In fact, over the years I found quite a few issues, but never took the time to report them here. Maybe I should do that… :-)

Report it in the following subforum:
https://board.jdownloader.org/forumdisplay.php?f=12

Originally Posted by StefanM

Thanks again for any efforts taken. You guys are really doing a great job!

Thanks for your feedback.

...but please save yourself and us a lot of time by simply providing logs:
- You don't have to analyze JDs behavior from the outside as it won't necessarily help finding the cause of that issue
- We can work on it more efficiently by having logs

Originally Posted by StefanM

OK, I'm prepared to do that. Just tell me, which logs and which parts you need:

We need all of those:

vk.com_jd.plugins.decrypter.VKontakteRu

Originally Posted by StefanM

Before I upload anything, I want to be able to verify what I'm sending. And limiting the amount of data would save me a lot of time.

Sure you can do that.

Originally Posted by StefanM

Worked for an IT company, would even call myself an IT-specialist, and I'm also a computer forensics guy,...

Great! If you want you can jump right in and help finding the issue

[StefanM]

Originally Posted by pspzockerscene

Yes and no.
You can grab our source code and edit those as explained here:
**External links are only visible to Support Staff**...
Our currently set limits are not linear.
Here is a quote of the comment of that line of code we referenced:
...

I'm an IT guy, not a software developer :-(
But I will take a look at it...

Originally Posted by pspzockerscene

Again wrong!
The mentioned support article describes what you need to do in order to see the individual links even when adding content via "vk.com/video/@CENSORED".
This is working fine here.

Tried that already earlier. Didn't work out for me then.
It works after I unchecked everything, but 'the address to the actual content'
However, the wording is not correct. It is the address I want to have, so I'm happy with that.
But the address to the actual content is something like this:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Originally Posted by pspzockerscene

We need all of those:

vk.com_jd.plugins.decrypter.VKontakteRu

Ok, will prepare new logs tomorrow:

1. Logged in / not logged in
2. Pasting individual links / pasting video gallery link only.

PS: Being already retired, indeed I would be prepared to help (if I could), which I've already been doing voluntarily for a number of software developers in the past years. But again, I'm not a software developer, just a retired IT guy... ;-)

[pspzockerscene]

Originally Posted by StefanM

I'm an IT guy, not a software developer :-(

Things can always change

Originally Posted by StefanM

It works after I unchecked everything, but 'the address to the actual content'

Nice.

Originally Posted by StefanM

However, the wording is not correct. It is the address I want to have, so I'm happy with that.
But the address to the actual content is something like this:...

I disagree!
In a lot of cases it is impossible to direct-link content e.g. because
- Direct-URL can only be used once
- Direct-URL expires after a short time
- Direct-URL is only valid for current IP address
- Content is hidden behind a POST-request and not a GET request -> In this case it would be impossible for the user to use the, by your definition, "content-url"

--> This means by our definition the current "content-URL" is an URL which, if you open it, links either directly or indirectly (as closest as possible, most times 'indirectly') to the content behind.
E.g. for an instagram post containing multiple pictures, the cuntent-URL of any of those pictures would simply open the main post as it is not possible to "officially" link any image of a multi-image-post.
It's similar for twitter and we're handling it the same way for vk.com.
Also the content-URL shall not be a temporary URL because again, if the user tries to use that one and it doesn't work anymore -> Problem.
Ultimately at this moment it's up to the plugin developer which URL to choose as content-URL.
EDIT
So if you got a better wording than "content URL", let us know

Originally Posted by StefanM

PS: Being already retired, indeed I would be prepared to help (if I could), which I've already been doing voluntarily for a number of software developers in the past years. But again, I'm not a software developer, just a retired IT guy... ;-)

No worries.

-psp-

[Jiaz]

Originally Posted by StefanM

C) Your info: VK.com only accepts max. 15 request in 30 sec (tested with issue 1)

While crawling (Finding Links…) I see a number of offline links being reported. This means to me that already in this stage requests are sent to VK.com. E.g. after 3 minutes 547 online and 32 offline links are displayed in the bubble.

Isn't this already 579 requests within 180 seconds?
This is for sure above those 15 accepted requests per 30 sec interval.

Thanks for the info! I'll look into that!

[pspzockerscene]

Results of my investigation so far:
The issues were caused by a mix of different uncaught errors some of which were simple website-errormessages upon which a page reload is required:

- "You tried to load the same page more than once in one second" -> While afaik we didn't do that, this message can happen for about 1% of all videos. I've implemented up to 10 retries in this case but it doesn't yet work as it should
- In some very rare cases the regular expression used to filter the json containing the video information ws not precise enough and failed to do so
- No matter what Jiaz will find out the issue was not caused by our eventually failing request-limiting as I've removed that completely for my tests
- Another unhandled error (but that only affected one of your 2166 items): "This content is blocked in your country"

It's still not working as it should but here are my current results when adding all of your 2166 links without account:
Packages: 2525
Links: 2592
Online: 2528

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[StefanM]

Originally Posted by Jiaz

Thanks for the info! I'll look into that!

If needed, I made a couple of screenshots showing this. It happened in every test run I made today.

[StefanM]

Originally Posted by Jiaz

Thanks for the info! I'll look into that!

And it's easy to see, that the 15 requests per 30 sec doesn't work:

1. The tests were conducted with a vk.com gallery with 2,000+ videos.
2. With max. 15 requests per 30 sec, crawling would take at least 4,000+ sec = 1:07 hours
3. But time taken was under 15 min. in each of my test runs

Will now setup a clean JD2installation to produce the logs you requested

[pspzockerscene]

After a huge refactoring of our vk.com plugin, the "You tried to load the same page more than once in one second" error should be successfully workarounded with the release of our next set of CORE-updates so all of your 2000+ vk URLs should get processed just fine.

Keep in mind that once the mentioned error appears once, all crawler instances will wait for it to be cleared -> vk.com crawl process may seem slower than usual but therefore it should be able to process all items.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Jiaz]

Originally Posted by StefanM

And it's easy to see, that the 15 requests per 30 sec doesn't work:

1. The tests were conducted with a vk.com gallery with 2,000+ videos.
2. With max. 15 requests per 30 sec, crawling would take at least 4,000+ sec = 1:07 hours
3. But time taken was under 15 min. in each of my test runs

Will now setup a clean JD2installation to produce the logs you requested

No logs required here. I can reproduce the issue and working on it, but request limit is not required, see post from pspzockerscene

[StefanM]

Originally Posted by pspzockerscene

Results of my investigation so far:
The issues were caused by a mix of different uncaught errors some of which were simple website-errormessages upon which a page reload is required:
...
-psp-

After spending a couple of hours of testing today, here are the latest results:

I set up a clean JD installation using JDownloader2Setup_windows-x64_jre8.exe 'offline installer'.

Notes:

1. The naming jre8 should be replaced with jre1.8
2. It is still not an offline installer as it starts the web installer, anyway :-(
3. There were some updates this morning, so you cannot really compare the results :-(

I started my tests using those 2,166 separate links from the Excel book I sent you earlier. I did NOT use my VK.com credentials.

Result:
2,021 Packages
2,525 Links
2,459 Online
63 Offline
Hosters 3
Unknown Status 3

Time taken: less than 10 minutes
Silly me: I forgot to copy the log files before re-starting with a new clean program folder


As a next step I applied all my settings (quite a few!) of my other JD installations to see if that would make a difference.

Result:
1 Package (my setting now was <jd:simpledate:yyyy-MM-dd>)
2,609 Links
2,538 Online
0 Offline
Hosters 3
Unknown Status 71

Time taken: less than 10 minutes

Finally, I conducted the same test with my 'Work Version' of JD

Notes:
This version has 150,000+ files in the downloads pane - most of them downloaded in the past years.
This time I was logged in to VK.com

Result:
1 Package (my setting now was <jd:simpledate:yyyy-MM-dd>)
2,605 Links
2,535 Online
0 Offline
Hosters 3
Unknown Status 70

Time taken: 41 minutes


Started the same LinkGrabbing once again, but NOT logged in.

Result:
1 Package (my setting now was <jd:simpledate:yyyy-MM-dd>)
2,607 Links
2,537 Online
66 Offline
Hosters 3
Unknown Status 4

Time taken: 20 minutes

As you can see,

1. when I'm not logged in, the number of offline files is 66 instead of 0
2. and the speed is twice as high

I assume that speed is still slower than with a clean installation, as JD needs some time for dupe checking within 150,000+ links. Is my assumption correct?


Could add more results later, but what I can say already now is, that those severe problems (finding less than 50 % of links copied to LinkGrabber) seem to be gone.

Just let me know which scenarios I should test.

Question: Did you release any updates in the meantime?
If yes, I should repeat my test some other day and disable updating then. However, I better wait until you're done with your updates of the VK.com plugin.

[Jiaz]

Originally Posted by StefanM

[*]It is still not an offline installer as it starts the web installer, anyway :-(

We don't provide any offline installer and never will. Where does it state *offline* ? (will have to fix wordning)
Please use forum search on this.

Short: We update JDownloader VERY often (10-20-30 updates a day can be normal) that an offline installer would be outdated a few minutes/hours after its creation.
Also the installer only installs files that are required (eg OS, CPU architecture, bitness...), so we would need multiple
offline installer. Wasted time of work/traffic and ppl just will hotlink/spread an outdated installer.
Our installers are webinstallers that always install the latest available version.

[pspzockerscene]

Originally Posted by StefanM

Question: Did you release any updates in the meantime?
If yes, I should repeat my test some other day and disable updating then. However, I better wait until you're done with your updates of the VK.com plugin.

Please re-read my last reply.

Also even though Jiaz stated that we do not need any new logs right now:
In the future please rather include logs than doing some hour long tests for yourself: It doesn't help us...

-psp-
EDIT

Also, I did all of my tests without vk.com account.
As stated the problems were not account related.

[Jiaz]

Originally Posted by StefanM

Question: Did you release any updates in the meantime?
If yes, I should repeat my test some other day and disable updating then. However, I better wait until you're done with your updates of the VK.com plugin.

There have been several updates since then, see https://support.jdownloader.org/Know...pdate-behavior

[pspzockerscene]

Originally Posted by Jiaz

There have been several updates since then, see https://support.jdownloader.org/Know...pdate-behavior

Yes but not the latest one which should fix all- or the majority of his reported vk.com related issues.

[Jiaz]

@StefanM: new set of updates are live

[Jiaz]

Originally Posted by StefanM

I assume that speed is still slower than with a clean installation, as JD needs some time for dupe checking within 150,000+ links. Is my assumption correct?

Please provide a screenshot of your about dialog, see https://support.jdownloader.org/Know...tion-directory
many links might need memory tweak/optimizations. but 150k is no problem with good memory settings, even million and more

[StefanM]

Originally Posted by pspzockerscene

Please re-read my last reply.

Also even though Jiaz stated that we do not need any new logs right now:
In the future please rather include logs than doing some hour long tests for yourself: It doesn't help us...

Well, even when I send logs - which my hour long tests were for - I need some time for different test cases:

• logged in / not logged in
• video gallery link / separate video links

All possible combinations of those are at least 4 different tests. And - as you can see - you need to repeat tests several times to achieve a representative result - for example to make sure that no other factors may have influenced the result. For instance caused by VK.com.

-psp-

Originally Posted by pspzockerscene

EDIT

Also, I did all of my tests without vk.com account.
As stated the problems were not account related.

Well, as you can see from my test results - and I made many(!) tests, not only the ones I reported - there is a difference in speed. These delays, I think, are caused by VK.com.

You can also observe another difference: Whether or not files are being reported as offline by JD depends on whether or not I am logged in. One reason might be, that in some video galleries you do not have access to all videos when you are not logged in.

According to my experience, these restrictions not only apply to VK.com but to many other websites as well!

And again, speed has an influence whether or not a timing related problem shows up.

And according to my observation it also makes a big(!) difference when JD has to scan a large amount of already downloaded files. In my case this led to a tremendous crawling time increase - reproducible!

[Jiaz]

Originally Posted by StefanM

All possible combinations of those are at least 4 different tests. And - as you can see - you need to repeat tests several times to achieve a representative result - for example to make sure that no other factors may have influenced the result. For instance caused by VK.com!

No you don't have to put that much effort in it! We just need a log that contains the issue. In case something went wrong(eg plugin error) or there are less links, then stop and create the log. Then we will check/fix it and then you can test it. Most often the same issue is the cause for different combinations so no need to spend so much time into it.
For example the initial report of yours -> start Jdownloader -> add the links -> wait for crawler to stop -> less links than insert = issue, create log -> that's all

[Jiaz]

Originally Posted by StefanM

And according to my observation it also makes a big(!) difference when JD has to scan a large amount of already downloaded files. In my case this led to a tremendous crawling time increase - reproducible!

The whole crawling process does not check against *already downloaded files* nor any *duplicate checks in Linkgrabber*.
What has massive influence on speed is this, see https://board.jdownloader.org/showpo...postcount=3237
Most influence is caused by rate limits/server response/latency and what kind of links you're adding (folder links with 1 requests = 1000 results vs 1000 single links which
require 1000 requests)