[StefanM]

Originally Posted by pspzockerscene

If you want you can also manually grab your logs from the "logs" folder inside your JD folder, zip them and upload them to a cloud of your choice and delete them once we got them.

Again:
Please provide a log...

-psp-

OK, I'm prepared to do that. Just tell me, which logs and which parts you need:

I see different logs such as:
vk.com_jd.plugins.decrypter.VKontakteRu.log.0 8.648.264 bytes
vk.com_jd.plugins.decrypter.VKontakteRu.log.1 11.201.511 bytes
vk.com_jd.plugins.decrypter.VKontakteRu.log.2 11.131.088 bytes

vk.com_jd.plugins.hoster.VKontakteRuHoster.log.0
vk.com_jd.plugins.hoster.VKontakteRuHoster.log.0.lck
vk.com_jd.plugins.hoster.VKontakteRuHoster.log.1
vk.com_jd.plugins.hoster.VKontakteRuHoster.log.2
vk.com_jd.plugins.hoster.VKontakteRuHoster.log.3
vk.com_jd.plugins.hoster.VKontakteRuHoster.log.4

....

Tons of data... :-)

Before I upload anything, I want to be able to verify what I'm sending. And limiting the amount of data would save me a lot of time.

PS: Worked for an IT company, would even call myself an IT-specialist, and I'm also a computer forensics guy,...

So, please understand that I may be overcautious.

PPS: I'm a good friend of the EPF developer, Maxim from Ukraine. That's why I'm pretty familiar with EPF and also because I'm writing the German GUI for all his programs, same as some of his web pages :-)

[pspzockerscene]

Originally Posted by StefanM

I'm referring to two different issues:
...

Both understood.

Originally Posted by StefanM

Yes, there is a reproducible difference:

Can't confirm.

Originally Posted by StefanM

Isn't this already 579 requests within 180 seconds?

Yap. I haven't denied that.

Originally Posted by StefanM

Can this be the reason for issue 1. Please have a look at the attached Excel book which might give you an explanation.

No. Please provide a log.

Originally Posted by StefanM

Is there any way for me to edit those numbers for conducting more tests? Cannot find it in advanced settings.

Yes and no.
You can grab our source code and edit those as explained here:
https://support.jdownloader.org/Know...up-ide-eclipse
Our currently set limits are not linear.
Here is a quote of the comment of that line of code we referenced:

sets request thresholds based on upper burstable limit. eg. 20(x) requests over 60000(y)[=1min]. Then on after it sets limit betweeninterval.

Originally Posted by StefanM

E) Why do I prefer to copy individual links to LinkGrabber?

It is because of a limitation in JD :-)
...
In JD, I can only see the individual links in my download list, when I add them separately.
You say "wrong: Wrong, see:

I'm afraid you are mistaken here. In my test case you can only see **External links are only visible to Support Staff****External links are only visible to Support Staff** - the same video gallery link for each of those 2,000+ videos.

Again wrong!
The mentioned support article describes what you need to do in order to see the individual links even when adding content via "vk.com/video/@CENSORED".
This is working fine here.

Originally Posted by StefanM

I can only have the link as part of the downloaded video file name, when I use individual links.
More detailed explanation: Example: Video-Name_-12345678_987654321_720p.mp4. The numbers are the numbers from the individual video link. These can be amended to the file name, but only when I past individual links.

You can modify filenames as you wish, see explanation of our Packagizer feature:
https://support.jdownloader.org/Know...the-packagizer
If you still fail to do so, please provide detailed examples (URLs!) so we can help you.

Originally Posted by StefanM

In most cases I don't want to download all videos. Those 2,000+ links were just a test.

Understood.
While we will look into this I just want to point out that e.g. on github.com you will find specific scripts dedicated just for downloading from vk.com. You might be better off using them in this case...

Originally Posted by StefanM

Maybe a few weeks after my download - e.g. after more videos have been uploaded - again I only want to download a few of them.

Understood.

Originally Posted by StefanM

It is too bad, that you cannot see/copy links of individual videos when using the video gallery link to let HD grab all videos. This is what I also need for archiving. For that I would have to parse DownloadList*.zip files :-(

As explained: Yes you can!

Originally Posted by StefanM

What I would like to know is, how and when dupe checking is being performed.
...
Let's assume, I copy links to files to LinkGrabber, which already had been downloaded earlier. Does JD dupe checker first check DownloadList and LinkCollector zip-files, whether or not they contain one or more of the added links? Or does JD send requests anyway?

No links are always re-processed.
In this case, links first go through a crawler to find the video direct-links/external links.
All of this will always be done and causes http requests.
There is no way to avoid this other than maybe using EventScripter scripts especially written for this:
https://support.jdownloader.org/Know...event-scripter

Originally Posted by StefanM

Let's assume I add 20 links to LinkGrabber and links 11…20 are dupes of links 1…10.
Now let's assume, while crawling for links 1…10, JD only finds links 1…5. links 6…10 are not found. Now, what happens with links 11…20? Will JD know that it already found links 11…15? And will JD try to find links 16…20 once again as it had not found links 6…10. Again, please note that 16…20 are identical with 6…10.

While adding links, JD will filter out dupes on the fly so if you add 20 items and 10 of them are contained twice in there, JD will only process 10.

Originally Posted by StefanM

I'm mentioning this here, because you were referring to dupe checker. What 'annoys' me is the fact, that - depending on where the dupes are - and what I did with them, they can be displayed or they cannot be displayed (checking the box 'Already in Downloadlist'). Links that were already downloaded won't necessarily show up in the LinkGrabber pane. So, the user won't know, if JD simply 'skipped' those links due to an error or because they already had been downloaded earlier.
Or is there a way to have them displayed as 'already downloaded earlier'?

Depends.
First of all, JD only checks against items that are still in your downloadlist.
If you e.g. remove them from your downloadlist but the files themselves still exist in your download destination, JD will not detect those as dupes. This makes sense because of multiple factors that I won't explain right now.
A user once wrote an EventScripter script for enhanced dupe checking.
That might be helpful for you too:
https://board.jdownloader.org/showthread.php?t=83048

Originally Posted by StefanM

To give you just one more example:
The bubble notifier position settings (in advanced settings) do not work correctly when I use a display scaling different from 100 % - tested on different machines.

Where should/would I report this? In fact, over the years I found quite a few issues, but never took the time to report them here. Maybe I should do that… :-)

Report it in the following subforum:
https://board.jdownloader.org/forumdisplay.php?f=12

Originally Posted by StefanM

Thanks again for any efforts taken. You guys are really doing a great job!

Thanks for your feedback.

...but please save yourself and us a lot of time by simply providing logs:
- You don't have to analyze JDs behavior from the outside as it won't necessarily help finding the cause of that issue
- We can work on it more efficiently by having logs

Originally Posted by StefanM

OK, I'm prepared to do that. Just tell me, which logs and which parts you need:

We need all of those:

vk.com_jd.plugins.decrypter.VKontakteRu

Originally Posted by StefanM

Before I upload anything, I want to be able to verify what I'm sending. And limiting the amount of data would save me a lot of time.

Sure you can do that.

Originally Posted by StefanM

Worked for an IT company, would even call myself an IT-specialist, and I'm also a computer forensics guy,...

Great! If you want you can jump right in and help finding the issue

[StefanM]

Originally Posted by pspzockerscene

Yes and no.
You can grab our source code and edit those as explained here:
**External links are only visible to Support Staff**...
Our currently set limits are not linear.
Here is a quote of the comment of that line of code we referenced:
...

I'm an IT guy, not a software developer :-(
But I will take a look at it...

Originally Posted by pspzockerscene

Again wrong!
The mentioned support article describes what you need to do in order to see the individual links even when adding content via "vk.com/video/@CENSORED".
This is working fine here.

Tried that already earlier. Didn't work out for me then.
It works after I unchecked everything, but 'the address to the actual content'
However, the wording is not correct. It is the address I want to have, so I'm happy with that.
But the address to the actual content is something like this:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Originally Posted by pspzockerscene

We need all of those:

vk.com_jd.plugins.decrypter.VKontakteRu

Ok, will prepare new logs tomorrow:

1. Logged in / not logged in
2. Pasting individual links / pasting video gallery link only.

PS: Being already retired, indeed I would be prepared to help (if I could), which I've already been doing voluntarily for a number of software developers in the past years. But again, I'm not a software developer, just a retired IT guy... ;-)

[pspzockerscene]

Originally Posted by StefanM

I'm an IT guy, not a software developer :-(

Things can always change

Originally Posted by StefanM

It works after I unchecked everything, but 'the address to the actual content'

Nice.

Originally Posted by StefanM

However, the wording is not correct. It is the address I want to have, so I'm happy with that.
But the address to the actual content is something like this:...

I disagree!
In a lot of cases it is impossible to direct-link content e.g. because
- Direct-URL can only be used once
- Direct-URL expires after a short time
- Direct-URL is only valid for current IP address
- Content is hidden behind a POST-request and not a GET request -> In this case it would be impossible for the user to use the, by your definition, "content-url"

--> This means by our definition the current "content-URL" is an URL which, if you open it, links either directly or indirectly (as closest as possible, most times 'indirectly') to the content behind.
E.g. for an instagram post containing multiple pictures, the cuntent-URL of any of those pictures would simply open the main post as it is not possible to "officially" link any image of a multi-image-post.
It's similar for twitter and we're handling it the same way for vk.com.
Also the content-URL shall not be a temporary URL because again, if the user tries to use that one and it doesn't work anymore -> Problem.
Ultimately at this moment it's up to the plugin developer which URL to choose as content-URL.
EDIT
So if you got a better wording than "content URL", let us know

Originally Posted by StefanM

PS: Being already retired, indeed I would be prepared to help (if I could), which I've already been doing voluntarily for a number of software developers in the past years. But again, I'm not a software developer, just a retired IT guy... ;-)

No worries.

-psp-

[Jiaz]

Originally Posted by StefanM

C) Your info: VK.com only accepts max. 15 request in 30 sec (tested with issue 1)

While crawling (Finding Links…) I see a number of offline links being reported. This means to me that already in this stage requests are sent to VK.com. E.g. after 3 minutes 547 online and 32 offline links are displayed in the bubble.

Isn't this already 579 requests within 180 seconds?
This is for sure above those 15 accepted requests per 30 sec interval.

Thanks for the info! I'll look into that!

[pspzockerscene]

Results of my investigation so far:
The issues were caused by a mix of different uncaught errors some of which were simple website-errormessages upon which a page reload is required:

- "You tried to load the same page more than once in one second" -> While afaik we didn't do that, this message can happen for about 1% of all videos. I've implemented up to 10 retries in this case but it doesn't yet work as it should
- In some very rare cases the regular expression used to filter the json containing the video information ws not precise enough and failed to do so
- No matter what Jiaz will find out the issue was not caused by our eventually failing request-limiting as I've removed that completely for my tests
- Another unhandled error (but that only affected one of your 2166 items): "This content is blocked in your country"

It's still not working as it should but here are my current results when adding all of your 2166 links without account:
Packages: 2525
Links: 2592
Online: 2528

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[StefanM]

Originally Posted by Jiaz

Thanks for the info! I'll look into that!

If needed, I made a couple of screenshots showing this. It happened in every test run I made today.

[StefanM]

Originally Posted by Jiaz

Thanks for the info! I'll look into that!

And it's easy to see, that the 15 requests per 30 sec doesn't work:

1. The tests were conducted with a vk.com gallery with 2,000+ videos.
2. With max. 15 requests per 30 sec, crawling would take at least 4,000+ sec = 1:07 hours
3. But time taken was under 15 min. in each of my test runs

Will now setup a clean JD2installation to produce the logs you requested

[pspzockerscene]

After a huge refactoring of our vk.com plugin, the "You tried to load the same page more than once in one second" error should be successfully workarounded with the release of our next set of CORE-updates so all of your 2000+ vk URLs should get processed just fine.

Keep in mind that once the mentioned error appears once, all crawler instances will wait for it to be cleared -> vk.com crawl process may seem slower than usual but therefore it should be able to process all items.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Jiaz]

Originally Posted by StefanM

And it's easy to see, that the 15 requests per 30 sec doesn't work:

1. The tests were conducted with a vk.com gallery with 2,000+ videos.
2. With max. 15 requests per 30 sec, crawling would take at least 4,000+ sec = 1:07 hours
3. But time taken was under 15 min. in each of my test runs

Will now setup a clean JD2installation to produce the logs you requested

No logs required here. I can reproduce the issue and working on it, but request limit is not required, see post from pspzockerscene

[StefanM]

Originally Posted by pspzockerscene

Results of my investigation so far:
The issues were caused by a mix of different uncaught errors some of which were simple website-errormessages upon which a page reload is required:
...
-psp-

After spending a couple of hours of testing today, here are the latest results:

I set up a clean JD installation using JDownloader2Setup_windows-x64_jre8.exe 'offline installer'.

Notes:

1. The naming jre8 should be replaced with jre1.8
2. It is still not an offline installer as it starts the web installer, anyway :-(
3. There were some updates this morning, so you cannot really compare the results :-(

I started my tests using those 2,166 separate links from the Excel book I sent you earlier. I did NOT use my VK.com credentials.

Result:
2,021 Packages
2,525 Links
2,459 Online
63 Offline
Hosters 3
Unknown Status 3

Time taken: less than 10 minutes
Silly me: I forgot to copy the log files before re-starting with a new clean program folder


As a next step I applied all my settings (quite a few!) of my other JD installations to see if that would make a difference.

Result:
1 Package (my setting now was <jd:simpledate:yyyy-MM-dd>)
2,609 Links
2,538 Online
0 Offline
Hosters 3
Unknown Status 71

Time taken: less than 10 minutes

Finally, I conducted the same test with my 'Work Version' of JD

Notes:
This version has 150,000+ files in the downloads pane - most of them downloaded in the past years.
This time I was logged in to VK.com

Result:
1 Package (my setting now was <jd:simpledate:yyyy-MM-dd>)
2,605 Links
2,535 Online
0 Offline
Hosters 3
Unknown Status 70

Time taken: 41 minutes


Started the same LinkGrabbing once again, but NOT logged in.

Result:
1 Package (my setting now was <jd:simpledate:yyyy-MM-dd>)
2,607 Links
2,537 Online
66 Offline
Hosters 3
Unknown Status 4

Time taken: 20 minutes

As you can see,

1. when I'm not logged in, the number of offline files is 66 instead of 0
2. and the speed is twice as high

I assume that speed is still slower than with a clean installation, as JD needs some time for dupe checking within 150,000+ links. Is my assumption correct?


Could add more results later, but what I can say already now is, that those severe problems (finding less than 50 % of links copied to LinkGrabber) seem to be gone.

Just let me know which scenarios I should test.

Question: Did you release any updates in the meantime?
If yes, I should repeat my test some other day and disable updating then. However, I better wait until you're done with your updates of the VK.com plugin.

[Jiaz]

Originally Posted by StefanM

[*]It is still not an offline installer as it starts the web installer, anyway :-(

We don't provide any offline installer and never will. Where does it state *offline* ? (will have to fix wordning)
Please use forum search on this.

Short: We update JDownloader VERY often (10-20-30 updates a day can be normal) that an offline installer would be outdated a few minutes/hours after its creation.
Also the installer only installs files that are required (eg OS, CPU architecture, bitness...), so we would need multiple
offline installer. Wasted time of work/traffic and ppl just will hotlink/spread an outdated installer.
Our installers are webinstallers that always install the latest available version.

[pspzockerscene]

Originally Posted by StefanM

Question: Did you release any updates in the meantime?
If yes, I should repeat my test some other day and disable updating then. However, I better wait until you're done with your updates of the VK.com plugin.

Please re-read my last reply.

Also even though Jiaz stated that we do not need any new logs right now:
In the future please rather include logs than doing some hour long tests for yourself: It doesn't help us...

-psp-
EDIT

Also, I did all of my tests without vk.com account.
As stated the problems were not account related.

[Jiaz]

Originally Posted by StefanM

Question: Did you release any updates in the meantime?
If yes, I should repeat my test some other day and disable updating then. However, I better wait until you're done with your updates of the VK.com plugin.

There have been several updates since then, see https://support.jdownloader.org/Know...pdate-behavior

[pspzockerscene]

Originally Posted by Jiaz

There have been several updates since then, see https://support.jdownloader.org/Know...pdate-behavior

Yes but not the latest one which should fix all- or the majority of his reported vk.com related issues.

[Jiaz]

@StefanM: new set of updates are live

[Jiaz]

Originally Posted by StefanM

I assume that speed is still slower than with a clean installation, as JD needs some time for dupe checking within 150,000+ links. Is my assumption correct?

Please provide a screenshot of your about dialog, see https://support.jdownloader.org/Know...tion-directory
many links might need memory tweak/optimizations. but 150k is no problem with good memory settings, even million and more

[StefanM]

Originally Posted by pspzockerscene

Please re-read my last reply.

Also even though Jiaz stated that we do not need any new logs right now:
In the future please rather include logs than doing some hour long tests for yourself: It doesn't help us...

Well, even when I send logs - which my hour long tests were for - I need some time for different test cases:

• logged in / not logged in
• video gallery link / separate video links

All possible combinations of those are at least 4 different tests. And - as you can see - you need to repeat tests several times to achieve a representative result - for example to make sure that no other factors may have influenced the result. For instance caused by VK.com.

-psp-

Originally Posted by pspzockerscene

EDIT

Also, I did all of my tests without vk.com account.
As stated the problems were not account related.

Well, as you can see from my test results - and I made many(!) tests, not only the ones I reported - there is a difference in speed. These delays, I think, are caused by VK.com.

You can also observe another difference: Whether or not files are being reported as offline by JD depends on whether or not I am logged in. One reason might be, that in some video galleries you do not have access to all videos when you are not logged in.

According to my experience, these restrictions not only apply to VK.com but to many other websites as well!

And again, speed has an influence whether or not a timing related problem shows up.

And according to my observation it also makes a big(!) difference when JD has to scan a large amount of already downloaded files. In my case this led to a tremendous crawling time increase - reproducible!

[Jiaz]

Originally Posted by StefanM

All possible combinations of those are at least 4 different tests. And - as you can see - you need to repeat tests several times to achieve a representative result - for example to make sure that no other factors may have influenced the result. For instance caused by VK.com!

No you don't have to put that much effort in it! We just need a log that contains the issue. In case something went wrong(eg plugin error) or there are less links, then stop and create the log. Then we will check/fix it and then you can test it. Most often the same issue is the cause for different combinations so no need to spend so much time into it.
For example the initial report of yours -> start Jdownloader -> add the links -> wait for crawler to stop -> less links than insert = issue, create log -> that's all

[Jiaz]

Originally Posted by StefanM

And according to my observation it also makes a big(!) difference when JD has to scan a large amount of already downloaded files. In my case this led to a tremendous crawling time increase - reproducible!

The whole crawling process does not check against *already downloaded files* nor any *duplicate checks in Linkgrabber*.
What has massive influence on speed is this, see https://board.jdownloader.org/showpo...postcount=3237
Most influence is caused by rate limits/server response/latency and what kind of links you're adding (folder links with 1 requests = 1000 results vs 1000 single links which
require 1000 requests)