[StefanM]

Originally Posted by Jiaz

The whole crawling process does not check against *already downloaded files* nor any *duplicate checks in Linkgrabber*.

Well, but obviously a dupe check is performed every time, after I click on Continue in LinkGrabber.

And as I said, I have 150,000+ downloaded files.

So, for each and every link that I copy to LinkGrabber a dupe check will have to be performed against those 150,000+ files and also against those, which are still in the list in LinkGrabber pane.

How much time would that cost?
When I do that in Excel, ist is only a few seconds. But how fast is JD compared to that?

I can only tell, that it is reproducible here, that a clean JD - which I just set up for my tests - is much, much faster, than my Work-JD with that large number of 'known' files/links.
And this also applies to the start-up time JD needs, which is understandable.

On the other hand, the zip-archives in cfg contain a lot(!) of data. Don't know how JD handles this and how much this can slow JD down.

[Jiaz]

Originally Posted by StefanM

Well, but obviously a dupe check is performed every time, after I click on Continue in LinkGrabber.

ONLY when you use *Already in DownloadList* as packagizer/filter condition, an additional check against Downloads is done.
And I guess this might be the case for you, then yes.

[Jiaz]

Originally Posted by StefanM

And as I said, I have 150,000+ downloaded files.
So, for each and every link that I copy to LinkGrabber a dupe check will have to be performed against those 150,000+ files and also against those, which are still in the list in LinkGrabber pane.
How much time would that cost?

Of course 150k links in DownloadList and adding 150k new links, will take some time.
Checks are already optimized for good tradeoff between speed and memory consumption. Currently checks are implemented via HashMap of linkIdentifier.

Originally Posted by StefanM

I can only tell, that it is reproducible here, that a clean JD - which I just set up for my tests - is much, much faster, than my Work-JD with that large number of 'known' files/links.

Of course the check will be faster if there are less links in DownloadList.

Originally Posted by StefanM

On the other hand, the zip-archives in cfg contain a lot(!) of data. Don't know how JD handles this and how much this can slow JD down.

The zip archives are only for storage on disk. JDownloader holds all data in memory, thats why I'm asking

Originally Posted by Jiaz

Please provide a screenshot of your about dialog, see https://support.jdownloader.org/Know...tion-directory
many links might need memory tweak/optimizations. but 150k is no problem with good memory settings, even million and more

[StefanM]

Originally Posted by Jiaz

Please provide a screenshot of your about dialog,...

Here you go! Made it just after the core update.

Question: Does this core update also contain the changes you made to the VK.com PlugIn?

If yes, I would re-test again (first only for me).

This is from one of my PCs with only 12 GB RAM. Usually I have more.

[Student im ersten Jahr]

Decryption Defect. It only decrypts a maximum of 121 photos. And some albums have much more than 121 photos.

Example Link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[StefanM]

Originally Posted by Student im ersten Jahr

Decryption Defect. It only decrypts a maximum of 121 photos. And some albums have much more than 121 photos.

@Student im ersten Jahr

I can confirm, that we have an issue here.

Tested this myself:
Bubble Notifier reports 1,882 Found Links
And in the end I see also 1,882 Links in LinkGrabber pane. (see attachment)
But VK.com reports All photos: 26,553
Crawler finds 1,882 links, too, which is just 7 % of downloadable photos on the page you mentioned.

BTW: Have you already tried SaveFrom.net helper Firefox AddOn?
It is a great tool for batch downloading photos from VK.com

And you can also copy the search result instead of downloading it with Firefox. Instead you can copy it to JD. :-)

It can also be used for video downloading: You will see an additional download button on each video page, where you can select among all available resolutions. But that tool cannot perform batch downloading.

[pspzockerscene]

@StefanM
Again no need for extensive tests as this bug was obvious/easy to reproduce with provided example URLs

Changelog for the next vk.com plugin update:
- Fixed pagination (= missing items) for photo albums [regression caused by my recent refactoring]
- Photo album pagination now also respects "Sleep time for general pagination" vk.com plugin setting" (and yes, this may slow down photo album crawling compared to before)
- Lowered minimum configurable wait for pagination from 1000ms to 500ms (default remains 1000ms)

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Jiaz]

Originally Posted by StefanM

Question: Does this core update also contain the changes you made to the VK.com PlugIn?

See https://support.jdownloader.org/Know...pdate-behavior
core update just means it also contains updates for non plugin parts
pending core updates can delay plugin-only updates

[Student im ersten Jahr]

I have a question: can JD2 auto-save direct image links to a text file? Direct links do not expire! But it's very valuable because I can access a backup of any image file on the VK.

Example DIRECT LINK:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Source: **External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Originally Posted by Student im ersten Jahr

I have a question: can JD2 auto-save direct image links to a text file? Direct links do not expire! But it's very valuable because I can access a backup of any image file on the VK.

No.
But you can use an EventScripter script to save those URLs in textfiles.
They're stored on each DownloadLink as property "picturedirectlink" and/or "directurls_fallback".

[Student im ersten Jahr]

To be honest, I am not an expert to write java-script code.

[pspzockerscene]

Originally Posted by Student im ersten Jahr

To be honest, I am not an expert to write java-script code.

Neither am I.
You can ask for help in this thread.

[StefanM]

Originally Posted by Student im ersten Jahr

I have a question: can JD2 auto-save direct image links to a text file? Direct links do not expire! But it's very valuable because I can access a backup of any image file on the VK.

Example DIRECT LINK:
**External links are only visible to Support Staff**...

Source: **External links are only visible to Support Staff**...

@Student im ersten Jahr

Again, in the meantime just use the Firefox AddOn SaveFrom.net helper I mentioned. If you want, you can copy the links to the photos and paste them in Linkrabber.

It is the format as in your Example 'DIRECT LINK'

And that AddOn is sill working without any problems.
You can use it for all photos or just for galleries.

[pspzockerscene]

Originally Posted by StefanM

Again, in the meantime just use the Firefox AddOn SaveFrom.net helper I mentioned. If you want, you can copy the links to the photos and paste them in Linkrabber.

I guess he wants to have the filename options JD has

Originally Posted by StefanM

It is the format as in your Example 'DIRECT LINK'

Yeah but it's missing other metadata/filenames.

Originally Posted by StefanM

And that AddOn is sill working without any problems.

Our plugin too

[StefanM]

Originally Posted by pspzockerscene

Our plugin too

That's good. Also I would prefer using JD over the Firefox AddOn.

But what naming are you referring to?
Just tested this album: **External links are only visible to Support Staff****External links are only visible to Support Staff**
I get just these names: -27763046_323043687.jpg (example)
when downloading from that album

One more thing...
As there is no changelog and - as I had to learn - there will be no changelog, how do I know, in which core release an issue/bug has been fixed.

The last reaction I received (after I retested Student's findings) was: "No need to retest, we are aware of the problem and it will be fixed."

How, the devil can I find out, when an issue has been fixed and retesting makes sense?

[pspzockerscene]

Originally Posted by StefanM

I get just these names: -27763046_323043687.jpg (example)
when downloading from that album

Yap.
I was referring to the other capabilities that JD has (packagizer, plugin properties and so on) which your addon might not have.

Originally Posted by StefanM

As there is no changelog and - as I had to learn - there will be no changelog, how do I know, in which core release an issue/bug has been fixed.

It's all described in our update FAQ:
https://support.jdownloader.org/Know...pdate-behavior
Basically whenever a fix is proposed here, it will be included in the next revision so just wait until you get an update and the revision number of your JDownloader increases (+ build date).
You may also check/view our source:
https://support.jdownloader.org/Know...up-ide-eclipse
As explained, we do make use of commit messages and those may also include more detailed information about fixes in plugins and so on.

Originally Posted by StefanM

The last reaction I received (after I retested Student's findings) was: "No need to retest, we are aware of the problem and it will be fixed."

No not exactly.
First I announced the fix here:
https://board.jdownloader.org/showpo...postcount=3247
...and since then, several updates have been released so yes the changes I mentioned have been released.
You can also if you want, monitor the "last changed" date of the affected plugin classes.
In this case:
See [JD install dir]/plugins/hoster/VKontakteRuHoster and [JD install dir]/plugins/decrypted/VKontakteRu
You can also view my commit here:
github.com/mirror/jdownloader/commit/79a3b55fe00ddfc98ad79e0ad51da52ae58adab3
Usually if changes are related to forum bug reports we will include the threadID in our commit messages, also in this case
Each commit will increase the revision number in source code:

$Revision: <Numbers> $

--> As long as your stable version is below that revision, you obviously did not get those changes yet.

Originally Posted by StefanM

How, the devil can I find out, when an issue has been fixed and retesting makes sense?

Well now you know at least 3 ways to find out.

-psp-
EDIT

In some cases we do notify ticket/forum users separately about updates/CORE-updates but most of all times we rely on our update FAQ and users simply noticing the next update + restart of JD.
If you got suggestions to improve our Update FAQ please let us know.
EDIT2

What I forgot:
Now always but most of all times we do use our forum thread prefixes for update announcements.
In this case I didn't do this because the vk thread is so long and the issue you reported affected only you atm so I did not want other users to think that there was some kind of problem with the plugin and also the majority of it worked so I left the prefix of this thread on "Solved".
As you can see, we do not have any fixed guidelines for the announcement of updates/changes

[raztoki]

I recall a lot of the request issues in the past. What was the cause back then was multi-threading and plugin returning results from decrypter as it was finding results, and hoster plugin maybe doing online checking concurrently, and or user was downloading too. It took many hours to find those issues and lock it down. I haven't worked on the VK plugin for years but complexity was similar to facebook and constant website ui changes. Fun times.

[pspzockerscene]

Nowdays Facebook is / would be more complex and vk plugin is currently in a good state.
Still it would be much easier if we'd have just implemented the vk API but I was always afraid of bans.

[StefanM]

Don't know since when: Playlists can no longer be crawled.

Just tested these:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

vk.com/video/playlist/-9745_42124286
vk.com/video/playlist/-9745_43376314
vk.com/video/playlist/-9745_43376150
vk.com/video/playlist/-9745_42128214

'Crawling for Downloads' just pops up for a second. That's it!
Deep Scan doesn't make any difference.

[Jiaz]

@StefanM: minimal json layout change, should be working again with next plugin update

[StefanM]

Great! Again an extremely fast reaction!

[Jiaz]

@StefanM: you're welcome

[StefanM]

Originally Posted by Jiaz

@StefanM: you're welcome

@Jiaz:
Sorry, still not working correctly!

Tested this one **External links are only visible to Support Staff****External links are only visible to Support Staff** several times. It contains 93 videos, but JD only finds 53 videos :-(

It is reproducible.

[pspzockerscene]

Working on it...
EDIT
Works fine here.

I get:
88 vk.com items
5 YT videos (4 of them are offline)

Please post a screenshot of your vk.com plugin settings and enable debug mode and post a log:
Please post your log-ID here | bitte poste deine Log-ID hier.

-psp-

[StefanM]

Originally Posted by pspzockerscene

Working on it...
EDIT
Works fine here.

I get:
88 vk.com items
5 YT videos (4 of them are offline)

Please post a screenshot of your vk.com plugin settings and enable debug mode and post a log:
Please post **External links are only visible to Support Staff**... | bitte poste **External links are only visible to Support Staff**....

-psp-

Pls read my EDIT. Will continue testing after lunch...
... and report again, then.

[Jiaz]

Originally Posted by pspzockerscene

I get:
88 vk.com items
5 YT videos (4 of them are offline)

Same results here

[StefanM]

@Jiaz, @pspzockerscene

After a couple of lengthily tests...

I found some inconsistency in dupe checking, which was probably caused by links created/collected with different versions of VK.com plugin over the past weeks.

Also a mix of below formats had impact on the dupe check results:

1. **External links are only visible to Support Staff****External links are only visible to Support Staff**
instead of
2. **External links are only visible to Support Staff****External links are only visible to Support Staff**

Links were collected based on different LinkGrabber inputs:
1. as result from crawling a playlist
2. as result from grabbing links with Firefox LinkGopher

Anyway, when using the current JD version, after cleaning all old test 'link collections' from the past weeks there is no such problem anymore, as reported.

So, just disregard my report.

[Jiaz]

@StefanM: Thanks for the followup. the current internal linkID has not changes since 02.06.2021 but depends on fastlinkcheck enabled/disabled (in case of videos) as the selected video quality is part of the linkID

[Student im ersten Jahr]

Changes to the plug-in cause a decryption defect.
It should decrypt one picture, but it decrypts unexpected, different data (more pictures and mp3)

**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Originally Posted by Jiaz

but depends on fastlinkcheck enabled/disabled (in case of videos) as the selected video quality is part of the linkID

For "superfast checked" videos, ID may change once quality gets chosen by plugin e.g. during full linkcheck or download-attempt (linkcheck is auto-done before download attempt).

This is no news regarding the vk.com plugin though.
As long as you do not change quality settings in the middle and re-add vk.com profiles after changing settings, dupe-check should work just fine.

[pspzockerscene]

Originally Posted by Student im ersten Jahr

Changes to the plug-in cause a decryption defect.

Fixed.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-
EDIT

Your link will not only return one item because:
1. It doesn't lead to any specific content here - it just redirects to "vk.com/feed".
2. It is invalid when just everything except the "photoID" from that URL.
3. It is part of a wall-post and thus needs to be crawled as part of one.
Wall posts can contain multiple elements.

[Student im ersten Jahr]

@pspzockersceneIt seems that the decryption of one photo is still wrong:

vk.com/feed?z=photo-120780206_457288362%2Falbum-120780206_00%2Frev

Show Download URLs
Downlaod from
Copy download adress to clipboard
Invalid links
Example:

vk.com/wall**External links are only visible to Support Staff**

Defect decrypt. Deciphering very, very many photos!
And there should only be 9 photos.
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Originally Posted by Student im ersten Jahr

seems that the decryption of one photo is still wrong:

Yes I see that it's still not working the way it should.
Please provide more of such example URLs.
EDIT

Fixed!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

Originally Posted by Student im ersten Jahr

Copy download adress to clipboard
Invalid links
Example:

Looks fine to me.
@Jiaz
Please check this.

Originally Posted by Student im ersten Jahr

Defect decrypt. Deciphering very, very many photos!
And there should only be 9 photos.

I get 8 photos.

Please post your log-ID here | bitte poste deine Log-ID hier.

-psp-

[Student im ersten Jahr]

You need to tell me about the plugin settings, I guess it all depends on the plugin settings?

[pspzockerscene]

1. Your wall-post URL is now offline (website states "Post not found" when opening it).
2. Check Settings -> Plugins -> vk.com -> "Wall settings" and "Settings for comments inside ...".

I was unable to reproduce any issues.
If you can still reproduce some, please provide new example URLs and a new logID.

[StefanM]

Originally Posted by pspzockerscene

1. Your wall-post URL is now offline (website states "Post not found" when opening it).
2. Check Settings -> Plugins -> vk.com -> "Wall settings" and "Settings for comments inside ...".

I was unable to reproduce any issues.
If you can still reproduce some, please provide new example URLs and a new logID.

Just wanted to say, that I'm prepared to try to reproduce any problems/issues myself - just in case assistance is needed.

Only thing I would need is the URL to test with.

Just an offer...
nothing more...

Because with all galleries I downloaded recently JD-Plugin worked just fine.

[pspzockerscene]

More tests are not necessary and not possible - his example URL is down.

-psp-

[Neurotoxin001]

Core #46316
Video link **External links are only visible to Support Staff****External links are only visible to Support Staff**
Error: https://i.imgur.com/VWdILcw.png

[StefanM]

Originally Posted by Neurotoxin001

Core #46316

Just tested a couple of video links: All of them are working fine for me.
If you want, just send me the link in this format (video-12345678_87654321) by PM, so it won't get censored. And I can verify...
... if it is working for me.

[pspzockerscene]

@Neurotoxin001
Fixed.

@StefanM
As already explained it doesn't make any sense to do additional tests because:
1. Often plugin related issues are reproducable for all users.
2. Often URLs can only be reproduced with specific URLs -> Also in this case.
EDIT
Sure you can continue doing those but I'm just saying it will often be a waste of time

For all changes mentioned in my post, the following information applies:

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-