[Jiaz]

@Neurotoxin001: thanks for the report. will be fixed with next update

[Student im ersten Jahr]

Packagename - Current problem.
The important character "_" disappears

Example:
12345 12345
Shouldbe:
12345_12345

Example link decrypt:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[Jiaz]

@Student im ersten Jahr: disable Settings->Advanced Settings->GeneralSettings.cleanupfilenames

[StefanM]

VK.com problem: Analysis

VK.com plugin is working again, but when I add e.g. 100 video links to LinkGrabber, only 20 to 30 % will be found. I can add those links once again and let LinkGrabber grab again, a few more links are found. Have to repeat this several times until all links are found/grabbed.

Reason is: VK.com introduced a change which will refuse answering too many requests in a short period of time.

I can use Extreme Picture Finder, configure a random delay, between the requests/downloads, and everything works fine.

To solve the problem in JD, also here a delay has to be introduced. This is the only solution I found so far.

[Jiaz]

@StefanM: Please don't open multiple threads with same topic!
JDownloader already has a default wait of 1 secs for pagination/crawling (see Settings->Plugins->vk.com)
Please provide a debug log, see https://support.jdownloader.org/Know...d-session-logs
Enable Debug first, see Settings->Advanced Settings->Log.debugmodeenabled
and restart JDownloader. Now wait for issue to happen and then create log and post shown logID here
vk.com now has a WAF (web application firewall) that we added support for recently and I could not reproduce any issues. So please provide a log, so we can see why it fails for you

[elvis_ef]

Hello.

I have few galleries on my list.
Unfortunately with number of download files (that I can get in the same time) set for instance to 5, VK says immediately that I want files to be downloaded too quickly. That the time is too short.

I wonder if there is a setting, that can make JD download files with any download transfer limit or sth ? But... only for galleries, image galleries. Videos must be downloaded without such settings. The thing is videos have no such errors while downloading. Only images.

These galleries have 3k images, 4k images.. but some have 860, or just 430.. the same error occurs with all of them. Only with these images, not videos.

Link (gallery) example:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Merged vk threads.

@elvis_ef
We got one generic vk.com thread.
Please do not open new threads for vk.com related issues.

To answer your question:
No there is no general limits you can apply.

We will simply have to update the plugin in order to recognize that limit so it can wait- and try again later.

Please post your log-ID here | bitte poste deine Log-ID hier.

-psp-
EDIT

Auto-retry-handling for "Too many requests" has already been implemented.
Currently it will wait 60 seconds for all vk.com URLs and continue afterwards...

[StefanM]

Originally Posted by Jiaz

@StefanM: Please don't open multiple threads with same topic!
JDownloader already has a default wait of 1 secs for pagination/crawling (see Settings->Plugins->vk.com)
...
vk.com now has a WAF (web application firewall) that we added support for recently and I could not reproduce any issues. So please provide a log, so we can see why it fails for you

Thanks for your reply. Did not see it, because instant email notification was disabled again, even though I did enable it :-(

Will conduct a few tests and then send you results and more details later this week.

[StefanM]

Originally Posted by Jiaz

@StefanM:
JDownloader already has a default wait of 1 secs for pagination/crawling (see Settings->Plugins->vk.com)
...

Please find a detailed test documentation attached as zip archive. I invested some 90 minutes for that documentation. Hope, it helps solving the issue.

Excerpt as plain text:

Test documentation

Notes:
I'm logged in with my VK.com credentials during this test.
The behavior shown here is typical. Had dozens of cases such as this one - on two different machines using two different Internet providers. Both of them were 16 Mbps accounts.

How the test was run: (I'm quite convinced, you will be able to reproduce this)
First, I copied all 2,166 video links from this example page: **External links are only visible to Support Staff****External links are only visible to Support Staff** - grabbed with LinkGopher in Firefox.
You will find all 2,166 links in Sheet 1 of the attached Excel book. In sheet 2 you will find all links found by LinkGrabber in the first run.
For the next runs, each time I copied all 2,166 links again to LinkGrabber. I used the order as in the Excel book.
In sheet 3 you will find all additional links found by LinkGrabber in the second run - i.e. only those links, LinkGrabber did not find in the first run. Same applies to the third run…
1st run: 863 found
2nd run: another 468 found
3rd run: another 310 found
4th run: another 210 found
total 1,851 links found after 4 runs, would need a few more to find all online links.

In plugin settings, I left those 'sleep time' settings alone, as default that is.

But I noticed, that there is no 1 second delay between grabbing individual video links. As you can see from the attached bubble screenshots (time taken and number of online links found), more than just one link is grabbed within a second. E.g. 973 links found online in 13:34 (814 seconds). This is just an average value. I often see more than 3 or 4 links added as online within one single second.
Maybe I misunderstand that setting? If yes, can you please explain what this 'sleep time' is about.

[Jiaz]

Originally Posted by StefanM

Thanks for your reply.

Nothing to thank me for. And please understand and take it personal that I did delete your other post. Just please don't open multiple threads/posts about the same topic. We see your post


@StefanM: It would be nice if you could just provide a debug log. Please understand that it's much FASTER and easier to have read to analyze logs that show the issue than us having to reproduce the issue and wait for it to happen. So instead of spending 90 minutes in documentation, just reproduce the issue once (crawler abort) and then create the log.
Also you should not crawl/find the individual links by yourself because adding 2166 links will result in minimum 2166 requests (likely even more) . You should instead just let JDownloader crawl/find the links as it can find multiiple links with less requests. Crawler in JDownloader will be much faster than adding thousands of individual links by yourself where no optimization (api with multiple results) can take place.

Originally Posted by StefanM

But I noticed, that there is no 1 second delay between grabbing individual video links.

This delay is for the crawler plugin! pagination. But you're adding single/individual links.
vk.com has a generic request limit of 500 ms between each request and max 15 requests every 30 secs

Browser.setBurstRequestIntervalLimitGlobal("vk.com", 500, 15, 30000);

so the issue is more likely caused by some error/errormessage that causes the whole crawler plugin to fail and that's why we're asking for a debug log.

[pspzockerscene]

Another explanation for the wording "pagination" and our related setting "Sleep time for general pagination":
Pagination is when a website either contains multiple items per visible page or e.g. uses "endless scrolling" like vk.com does so every time you scroll down more, X new items get loaded.
Our setting defines the wait-time in between.

By using "Link gopher" to manually collect all single video items, the following things will happen:

• Crawling all videos one by one will take much longer
• --> It will also cause more requests so the chance of running into any limits is higher
• Pagination will never happen thus that setting is irrelevant as long as you add links this way

Is there any good reason why you do not let JD crawl this "wall" link?

vk.com/video/@CENSORED

?

Letting JD do the job you now did by hand using "Link Gopher" should make things much faster/easier.

Nevertheless I'm currently adding all of these 2166 items from your excel sheet to my test-JD and I will post my results here afterwards.

-psp-
EDIT

Here are my results:
Packages: 1676
Links: 1730
Links online: 1679

EDIT2

Here are my results when I let our crawler do the job (with pagination etc...):
Packages: 6 (much less packages because all vk.com self-hosted videos of that profile will be grouped intop one package named after the username)
Links: 2527
Links online: 2466

[StefanM]

Originally Posted by Jiaz

@StefanM: It would be nice if you could just provide a debug log. Please understand that it's much FASTER and easier to have read to analyze logs that show the issue than us having to reproduce the issue and wait for it to happen.

Thanks for all the information given.
But couldn't you just add those links I sent you to JD's LinkGrabber and see for yourself what happens?

Opening Excel, copying the links from sheet 1, and adding them to LinkGrabber wouldn't take more than a minute.

And then you can see for yourself what happens.

I add those video links as single links for simple reasons:

1. In JD, I can only see the individual links in my download list, wenn I add them separately.

2. I can only have the link as part of the downloaded video file name, when I use individual links.

3. In most cases I don't want to download all videos. Here it was just a test.

4. And maybe later - e.g. after more videos have been uploaded - again I only want to download a few of them.

Now, once again my little request:
Before I create a log with sensitive data (maybe alleged copy right infringements, ...) could you please just add those links to your JD and see if you can reproduce the problem?

It won't take you more than a minute and then waiting what happens.
I would really appreciate this!

I can tell you that the problem did not exist before the VK.com change. And I can tell you, that this problem does not exist, when I use Extreme Picture Finder (**External links are only visible to Support Staff****External links are only visible to Support Staff**). I have a working template which worked before and after the changes VK.com made.

This template also selects the highest resolution only, and the log provides me with the direct links, which are valid for some 5 days.

No problem occurs, when I feed those links to LinkGrabber. Just the file naming won't work then.

Addendum:
When I let JD crawl the video page, it stops crawling after 1,001 links.

[Jiaz]

@StefanM: I can reproduce the issue but I did not ran into any plugin issue. For me it stopped at about 1900 found links. The issue must lie somewhere between. That means either some links are getting *lost* from clipboard to plugin or there are some error messages in plugin that are silently caught.
We will do some more testing by tomorrow

Originally Posted by StefanM

Opening Excel, copying the links from sheet 1, and adding them to LinkGrabber wouldn't take more than a minute.

That's correct but due to set request limits in plugin, it takes a while to finish and as our assumption was about a plugin issue, it would be MUCH faster if you could just provide a log that contains the issue. But as we reproced without any issue, a log won't help and we have to dig deeper into the plugin

Originally Posted by StefanM

could you please just add those links to your JD and see if you can reproduce the problem?

pspzockerscene and I already did hence my comments above

[Jiaz]

Originally Posted by StefanM

When I let JD crawl the video page, it stops crawling after 1,001 links.

We will first check if we can reproduce the issue ourselves as it worked fine for pspzockerscene.
Just out of interest, did you also do testing with the account disabled in JDownloader?

[StefanM]

Originally Posted by Jiaz

@StefanM: ... The issue must lie somewhere between. That means either some links are getting *lost* from clipboard to plugin or there are some error messages in plugin that are silently caught.
We will do some more testing by tomorrow.

It is not a clipboard issue, as I see all links in LinkGrabber window. I paste them there manually.

And If you look at the different runs in my Excel book, you can see that each run some other links are found.

But everytime, each run, all links are in the LinkGrabber window.

[StefanM]

Originally Posted by Jiaz

We will first check if we can reproduce the issue ourselves as it worked fine for pspzockerscene.
Just out of interest, did you also do testing with the account disabled in JDownloader?

Will check that tomorrow.

And I/we should not not mix up two issues:

1. Pasting individual video links into the LinkGrabber window. (many links are not found, more are found in a second, third, ... run)

2. Pasting the Video gallery link into the LinkGrabber window. (Here Grabbing stops too early, before all links have been found)

[StefanM]

Originally Posted by pspzockerscene

Another explanation for the wording "pagination" and our related setting "Sleep time for general pagination":
Pagination is when a website either contains multiple items per visible page or e.g. uses "endless scrolling" like vk.com does so every time you scroll down more, X new items get loaded.
Our setting defines the wait-time in between...

Sorry about not answering earlier. I simply overlooked your reply.

Will answer all questions in a new post later today.

Just one hint right now. I learned from you that VK.com only accepts 15 requests within a 30 sec interval.

For sure, my test with copying more than 2,000 links to LinkGrabber sent more requests, than 15 per 30 sec.

But I will collect all details now and give you a summary including some analysis later today.

Thanks again for taking my report seriously. :-)

[pspzockerscene]

Originally Posted by StefanM

1. In JD, I can only see the individual links in my download list, wenn I add them separately.

Wrong, see:
https://support.jdownloader.org/Know...iew-added-urls

Originally Posted by StefanM

2. I can only have the link as part of the downloaded video file name, when I use individual links.

I don't understand that.
Please add a more detailed explanation.

Originally Posted by StefanM

3. In most cases I don't want to download all videos. Here it was just a test.
...
4. And maybe later - e.g. after more videos have been uploaded - again I only want to download a few of them.

Valid points.

Originally Posted by StefanM

I can tell you that the problem did not exist before the VK.com change. And I can tell you, that this problem does not exist, when I use Extreme Picture Finder (**External links are only visible to Support Staff****External links are only visible to Support Staff**). I have a working template which worked before and after the changes VK.com made.

Those tools work totally different than JD - comparing them to JDs functionality doesn't make any sense.

Originally Posted by StefanM

2. Pasting the Video gallery link into the LinkGrabber window. (Here Grabbing stops too early, before all links have been found)

No.
It worked fine here.
Please simply provide a log.
We do not process your data in any way.
If you want you can also manually grab your logs from the "logs" folder inside your JD folder, zip them and upload them to a cloud of your choice and delete them once we got them.

Originally Posted by StefanM

Just one hint right now. I learned from you that VK.com only accepts 15 requests within a 30 sec interval.

We've never claimed that.
Afaik vk definitely allows more requests but our plugin limits them atm.

Originally Posted by StefanM

But I will collect all details now and give you a summary including some analysis later today.

Again:
Please provide a log...

-psp-

[StefanM]

Originally Posted by Jiaz

We will first check if we can reproduce the issue ourselves as it worked fine for pspzockerscene.
Just out of interest, did you also do testing with the account disabled in JDownloader?

Additional findings and questions regarding those findings

I'm referring to two different issues:

Issue 1: Pasting individual video links into the LinkGrabber window. (many links are not found, more are found in a second, third, fourth, ... run)

Issue 2: Pasting the Video gallery link into the LinkGrabber window. (Here Grabbing stops too early, before all links have been found)


A) Logged in / not logged in on VK.com (tested with issue 2)

Tested several times on two different machines with two different providers.

Yes, there is a reproducible difference:
When I'm logged in, crawling is much slower and in average only some 50 % of the links are found.
When I'm not logged in, crawling is about twice as fast and in average over 90 % of the links are found.


B) Different providers (tested with issue 2)

Even though provider speed tests show almost the same speed, crawling speed is reproducible different.


C) Your info: VK.com only accepts max. 15 request in 30 sec (tested with issue 1)

While crawling (Finding Links…) I see a number of offline links being reported. This means to me that already in this stage requests are sent to VK.com. E.g. after 3 minutes 547 online and 32 offline links are displayed in the bubble.

Isn't this already 579 requests within 180 seconds?
This is for sure above those 15 accepted requests per 30 sec interval. Can this be the reason for issue 1. Please have a look at the attached Excel book which might give you an explanation. I marked all files by colors, in which run they were found. So you can see when which links were skipped / not found.


D) Browser.setBurstRequestIntervalLimitGlobal("vk.com", 500, 15, 30000)

Is there any way for me to edit those numbers for conducting more tests? Cannot find it in advanced settings.


E) Why do I prefer to copy individual links to LinkGrabber?

It is because of a limitation in JD :-)

1. In JD, I can only see the individual links in my download list, when I add them separately.
   You say "wrong: Wrong, see:
   https://support.jdownloader.org/Know...iew-added-urls
   I'm afraid you are mistaken here. In my test case you can only see **External links are only visible to Support Staff****External links are only visible to Support Staff** - the same video gallery link for each of those 2,000+ videos.
2. I can only have the link as part of the downloaded video file name, when I use individual links.
   More detailed explanation: Example: Video-Name_-12345678_987654321_720p.mp4. The numbers are the numbers from the individual video link. These can be amended to the file name, but only when I paste individual links.
3. In most cases I don't want to download all videos. Those 2,000+ links were just a test.
4. Maybe a few weeks after my download - e.g. after more videos have been uploaded - again I only want to download a few of them.

It is too bad, that you cannot see/copy links of individual videos when using the video gallery link to let HD grab all videos. This is what I also need for archiving. For that I would have to parse DownloadList*.zip files :-(


F) You mentioned reducing number of requests by dupe checking

As pointed out in C) there seem to be many occasions, where requests are being sent in high frequency. Please correct me if I'm wrong.

What I would like to know is, how and when dupe checking is being performed.

E.1) Let's assume, I copy links to files to LinkGrabber, which already had been downloaded earlier. Does JD dupe checker first check DownloadList and LinkCollector zip-files, whether or not they contain one or more of the added links? Or does JD send requests anyway?

E.2) Let's assume I add 20 links to LinkGrabber and links 11…20 are dupes of links 1…10.
Now let's assume, while crawling for links 1…10, JD only finds links 1…5. links 6…10 are not found. Now, what happens with links 11…20? Will JD know that it already found links 11…15? And will JD try to find links 16…20 once again as it had not found links 6…10. Again, please note that 16…20 are identical with 6…10.


G) Dupe checker doesn't work in a consistent manner

I'm mentioning this here, because you were referring to dupe checker. What 'annoys' me is the fact, that - depending on where the dupes are - and what I did with them, they can be displayed or they cannot be displayed (checking the box 'Already in Downloadlist'). Links that were already downloaded won't necessarily show up in the LinkGrabber pane. So, the user won't know, if JD simply 'skipped' those links due to an error or because they already had been downloaded earlier.
Or is there a way to have them displayed as 'already downloaded earlier'?


And one more question regarding bug reports or requests for improvement:

To give you just one more example:
The bubble notifier position settings (in advanced settings) do not work correctly when I use a display scaling different from 100 % - tested on different machines.

Where should/would I report this? In fact, over the years I found quite a few issues, but never took the time to report them here. Maybe I should do that… :-)

Thanks again for any efforts taken. You guys are really doing a great job!

[StefanM]

Originally Posted by pspzockerscene

Wrong, see:
**External links are only visible to Support Staff**...

I don't understand that.
Please add a more detailed explanation...

Will embed answers in the post I just sent by editing it. First answer is already there :-)

More to come within the next hour.