[StefanM]

Originally Posted by Jiaz

The whole crawling process does not check against *already downloaded files* nor any *duplicate checks in Linkgrabber*.

Well, but obviously a dupe check is performed every time, after I click on Continue in LinkGrabber.

And as I said, I have 150,000+ downloaded files.

So, for each and every link that I copy to LinkGrabber a dupe check will have to be performed against those 150,000+ files and also against those, which are still in the list in LinkGrabber pane.

How much time would that cost?
When I do that in Excel, ist is only a few seconds. But how fast is JD compared to that?

I can only tell, that it is reproducible here, that a clean JD - which I just set up for my tests - is much, much faster, than my Work-JD with that large number of 'known' files/links.
And this also applies to the start-up time JD needs, which is understandable.

On the other hand, the zip-archives in cfg contain a lot(!) of data. Don't know how JD handles this and how much this can slow JD down.

[Jiaz]

Originally Posted by StefanM

Well, but obviously a dupe check is performed every time, after I click on Continue in LinkGrabber.

ONLY when you use *Already in DownloadList* as packagizer/filter condition, an additional check against Downloads is done.
And I guess this might be the case for you, then yes.

[Jiaz]

Originally Posted by StefanM

And as I said, I have 150,000+ downloaded files.
So, for each and every link that I copy to LinkGrabber a dupe check will have to be performed against those 150,000+ files and also against those, which are still in the list in LinkGrabber pane.
How much time would that cost?

Of course 150k links in DownloadList and adding 150k new links, will take some time.
Checks are already optimized for good tradeoff between speed and memory consumption. Currently checks are implemented via HashMap of linkIdentifier.

Originally Posted by StefanM

I can only tell, that it is reproducible here, that a clean JD - which I just set up for my tests - is much, much faster, than my Work-JD with that large number of 'known' files/links.

Of course the check will be faster if there are less links in DownloadList.

Originally Posted by StefanM

On the other hand, the zip-archives in cfg contain a lot(!) of data. Don't know how JD handles this and how much this can slow JD down.

The zip archives are only for storage on disk. JDownloader holds all data in memory, thats why I'm asking

Originally Posted by Jiaz

Please provide a screenshot of your about dialog, see https://support.jdownloader.org/Know...tion-directory
many links might need memory tweak/optimizations. but 150k is no problem with good memory settings, even million and more

[StefanM]

Originally Posted by Jiaz

Please provide a screenshot of your about dialog,...

Here you go! Made it just after the core update.

Question: Does this core update also contain the changes you made to the VK.com PlugIn?

If yes, I would re-test again (first only for me).

This is from one of my PCs with only 12 GB RAM. Usually I have more.

[Student im ersten Jahr]

Decryption Defect. It only decrypts a maximum of 121 photos. And some albums have much more than 121 photos.

Example Link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[StefanM]

Originally Posted by Student im ersten Jahr

Decryption Defect. It only decrypts a maximum of 121 photos. And some albums have much more than 121 photos.

@Student im ersten Jahr

I can confirm, that we have an issue here.

Tested this myself:
Bubble Notifier reports 1,882 Found Links
And in the end I see also 1,882 Links in LinkGrabber pane. (see attachment)
But VK.com reports All photos: 26,553
Crawler finds 1,882 links, too, which is just 7 % of downloadable photos on the page you mentioned.

BTW: Have you already tried SaveFrom.net helper Firefox AddOn?
It is a great tool for batch downloading photos from VK.com

And you can also copy the search result instead of downloading it with Firefox. Instead you can copy it to JD. :-)

It can also be used for video downloading: You will see an additional download button on each video page, where you can select among all available resolutions. But that tool cannot perform batch downloading.

[pspzockerscene]

@StefanM
Again no need for extensive tests as this bug was obvious/easy to reproduce with provided example URLs

Changelog for the next vk.com plugin update:
- Fixed pagination (= missing items) for photo albums [regression caused by my recent refactoring]
- Photo album pagination now also respects "Sleep time for general pagination" vk.com plugin setting" (and yes, this may slow down photo album crawling compared to before)
- Lowered minimum configurable wait for pagination from 1000ms to 500ms (default remains 1000ms)

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Jiaz]

Originally Posted by StefanM

Question: Does this core update also contain the changes you made to the VK.com PlugIn?

See https://support.jdownloader.org/Know...pdate-behavior
core update just means it also contains updates for non plugin parts
pending core updates can delay plugin-only updates

[Student im ersten Jahr]

I have a question: can JD2 auto-save direct image links to a text file? Direct links do not expire! But it's very valuable because I can access a backup of any image file on the VK.

Example DIRECT LINK:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Source: **External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Originally Posted by Student im ersten Jahr

I have a question: can JD2 auto-save direct image links to a text file? Direct links do not expire! But it's very valuable because I can access a backup of any image file on the VK.

No.
But you can use an EventScripter script to save those URLs in textfiles.
They're stored on each DownloadLink as property "picturedirectlink" and/or "directurls_fallback".

[Student im ersten Jahr]

To be honest, I am not an expert to write java-script code.

[pspzockerscene]

Originally Posted by Student im ersten Jahr

To be honest, I am not an expert to write java-script code.

Neither am I.
You can ask for help in this thread.

[StefanM]

Originally Posted by Student im ersten Jahr

I have a question: can JD2 auto-save direct image links to a text file? Direct links do not expire! But it's very valuable because I can access a backup of any image file on the VK.

Example DIRECT LINK:
**External links are only visible to Support Staff**...

Source: **External links are only visible to Support Staff**...

@Student im ersten Jahr

Again, in the meantime just use the Firefox AddOn SaveFrom.net helper I mentioned. If you want, you can copy the links to the photos and paste them in Linkrabber.

It is the format as in your Example 'DIRECT LINK'

And that AddOn is sill working without any problems.
You can use it for all photos or just for galleries.

[pspzockerscene]

Originally Posted by StefanM

Again, in the meantime just use the Firefox AddOn SaveFrom.net helper I mentioned. If you want, you can copy the links to the photos and paste them in Linkrabber.

I guess he wants to have the filename options JD has

Originally Posted by StefanM

It is the format as in your Example 'DIRECT LINK'

Yeah but it's missing other metadata/filenames.

Originally Posted by StefanM

And that AddOn is sill working without any problems.

Our plugin too

[StefanM]

Originally Posted by pspzockerscene

Our plugin too

That's good. Also I would prefer using JD over the Firefox AddOn.

But what naming are you referring to?
Just tested this album: **External links are only visible to Support Staff****External links are only visible to Support Staff**
I get just these names: -27763046_323043687.jpg (example)
when downloading from that album

One more thing...
As there is no changelog and - as I had to learn - there will be no changelog, how do I know, in which core release an issue/bug has been fixed.

The last reaction I received (after I retested Student's findings) was: "No need to retest, we are aware of the problem and it will be fixed."

How, the devil can I find out, when an issue has been fixed and retesting makes sense?

[pspzockerscene]

Originally Posted by StefanM

I get just these names: -27763046_323043687.jpg (example)
when downloading from that album

Yap.
I was referring to the other capabilities that JD has (packagizer, plugin properties and so on) which your addon might not have.

Originally Posted by StefanM

As there is no changelog and - as I had to learn - there will be no changelog, how do I know, in which core release an issue/bug has been fixed.

It's all described in our update FAQ:
https://support.jdownloader.org/Know...pdate-behavior
Basically whenever a fix is proposed here, it will be included in the next revision so just wait until you get an update and the revision number of your JDownloader increases (+ build date).
You may also check/view our source:
https://support.jdownloader.org/Know...up-ide-eclipse
As explained, we do make use of commit messages and those may also include more detailed information about fixes in plugins and so on.

Originally Posted by StefanM

The last reaction I received (after I retested Student's findings) was: "No need to retest, we are aware of the problem and it will be fixed."

No not exactly.
First I announced the fix here:
https://board.jdownloader.org/showpo...postcount=3247
...and since then, several updates have been released so yes the changes I mentioned have been released.
You can also if you want, monitor the "last changed" date of the affected plugin classes.
In this case:
See [JD install dir]/plugins/hoster/VKontakteRuHoster and [JD install dir]/plugins/decrypted/VKontakteRu
You can also view my commit here:
github.com/mirror/jdownloader/commit/79a3b55fe00ddfc98ad79e0ad51da52ae58adab3
Usually if changes are related to forum bug reports we will include the threadID in our commit messages, also in this case
Each commit will increase the revision number in source code:

$Revision: <Numbers> $

--> As long as your stable version is below that revision, you obviously did not get those changes yet.

Originally Posted by StefanM

How, the devil can I find out, when an issue has been fixed and retesting makes sense?

Well now you know at least 3 ways to find out.

-psp-
EDIT

In some cases we do notify ticket/forum users separately about updates/CORE-updates but most of all times we rely on our update FAQ and users simply noticing the next update + restart of JD.
If you got suggestions to improve our Update FAQ please let us know.
EDIT2

What I forgot:
Now always but most of all times we do use our forum thread prefixes for update announcements.
In this case I didn't do this because the vk thread is so long and the issue you reported affected only you atm so I did not want other users to think that there was some kind of problem with the plugin and also the majority of it worked so I left the prefix of this thread on "Solved".
As you can see, we do not have any fixed guidelines for the announcement of updates/changes

[raztoki]

I recall a lot of the request issues in the past. What was the cause back then was multi-threading and plugin returning results from decrypter as it was finding results, and hoster plugin maybe doing online checking concurrently, and or user was downloading too. It took many hours to find those issues and lock it down. I haven't worked on the VK plugin for years but complexity was similar to facebook and constant website ui changes. Fun times.

[pspzockerscene]

Nowdays Facebook is / would be more complex and vk plugin is currently in a good state.
Still it would be much easier if we'd have just implemented the vk API but I was always afraid of bans.

[StefanM]

Don't know since when: Playlists can no longer be crawled.

Just tested these:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

vk.com/video/playlist/-9745_42124286
vk.com/video/playlist/-9745_43376314
vk.com/video/playlist/-9745_43376150
vk.com/video/playlist/-9745_42128214

'Crawling for Downloads' just pops up for a second. That's it!
Deep Scan doesn't make any difference.

[Jiaz]

@StefanM: minimal json layout change, should be working again with next plugin update