JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #3241  
Old 03.06.2022, 18:01
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 354
Default

Quote:
Originally Posted by Jiaz View Post
The whole crawling process does not check against *already downloaded files* nor any *duplicate checks in Linkgrabber*.
Well, but obviously a dupe check is performed every time, after I click on Continue in LinkGrabber.

And as I said, I have 150,000+ downloaded files.

So, for each and every link that I copy to LinkGrabber a dupe check will have to be performed against those 150,000+ files and also against those, which are still in the list in LinkGrabber pane.

How much time would that cost?
When I do that in Excel, ist is only a few seconds. But how fast is JD compared to that?

I can only tell, that it is reproducible here, that a clean JD - which I just set up for my tests - is much, much faster, than my Work-JD with that large number of 'known' files/links.
And this also applies to the start-up time JD needs, which is understandable.

On the other hand, the zip-archives in cfg contain a lot(!) of data. Don't know how JD handles this and how much this can slow JD down.
Reply With Quote
  #3242  
Old 03.06.2022, 18:40
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 77,929
Default

Quote:
Originally Posted by StefanM View Post
Well, but obviously a dupe check is performed every time, after I click on Continue in LinkGrabber.
ONLY when you use *Already in DownloadList* as packagizer/filter condition, an additional check against Downloads is done.
And I guess this might be the case for you, then yes.
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 03.06.2022 at 18:50.
Reply With Quote
  #3243  
Old 03.06.2022, 18:46
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 77,929
Default

Quote:
Originally Posted by StefanM View Post
And as I said, I have 150,000+ downloaded files.
So, for each and every link that I copy to LinkGrabber a dupe check will have to be performed against those 150,000+ files and also against those, which are still in the list in LinkGrabber pane.
How much time would that cost?
Of course 150k links in DownloadList and adding 150k new links, will take some time.
Checks are already optimized for good tradeoff between speed and memory consumption. Currently checks are implemented via HashMap of linkIdentifier.

Quote:
Originally Posted by StefanM View Post
I can only tell, that it is reproducible here, that a clean JD - which I just set up for my tests - is much, much faster, than my Work-JD with that large number of 'known' files/links.
Of course the check will be faster if there are less links in DownloadList.

Quote:
Originally Posted by StefanM View Post
On the other hand, the zip-archives in cfg contain a lot(!) of data. Don't know how JD handles this and how much this can slow JD down.
The zip archives are only for storage on disk. JDownloader holds all data in memory, thats why I'm asking
Quote:
Originally Posted by Jiaz View Post
Please provide a screenshot of your about dialog, see https://support.jdownloader.org/Know...tion-directory
many links might need memory tweak/optimizations. but 150k is no problem with good memory settings, even million and more
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3244  
Old 04.06.2022, 17:47
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 354
Default

Quote:
Originally Posted by Jiaz View Post
Please provide a screenshot of your about dialog,...
Here you go! Made it just after the core update.

Question: Does this core update also contain the changes you made to the VK.com PlugIn?

If yes, I would re-test again (first only for me).

This is from one of my PCs with only 12 GB RAM. Usually I have more.
Attached Thumbnails
About.jpg  

Last edited by StefanM; 04.06.2022 at 17:50.
Reply With Quote
  #3245  
Old 05.06.2022, 22:10
Student im ersten Jahr Student im ersten Jahr is offline
Peta Loader
 
Join Date: Nov 2020
Posts: 180
Default

Decryption Defect. It only decrypts a maximum of 121 photos. And some albums have much more than 121 photos.

Example Link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #3246  
Old 06.06.2022, 10:47
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 354
Default

Quote:
Originally Posted by Student im ersten Jahr View Post
Decryption Defect. It only decrypts a maximum of 121 photos. And some albums have much more than 121 photos.
@Student im ersten Jahr

I can confirm, that we have an issue here.

Tested this myself:
Bubble Notifier reports 1,882 Found Links
And in the end I see also 1,882 Links in LinkGrabber pane. (see attachment)
But VK.com reports All photos: 26,553
Crawler finds 1,882 links, too, which is just 7 % of downloadable photos on the page you mentioned.

BTW: Have you already tried SaveFrom.net helper Firefox AddOn?
It is a great tool for batch downloading photos from VK.com

And you can also copy the search result instead of downloading it with Firefox. Instead you can copy it to JD. :-)

It can also be used for video downloading: You will see an additional download button on each video page, where you can select among all available resolutions. But that tool cannot perform batch downloading.
Attached Thumbnails
student.jpg  

Last edited by Jiaz; 07.06.2022 at 16:33.
Reply With Quote
  #3247  
Old 07.06.2022, 16:13
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 63,166
Default

@StefanM
Again no need for extensive tests as this bug was obvious/easy to reproduce with provided example URLs

Changelog for the next vk.com plugin update:
- Fixed pagination (= missing items) for photo albums [regression caused by my recent refactoring]
- Photo album pagination now also respects "Sleep time for general pagination" vk.com plugin setting" (and yes, this may slow down photo album crawling compared to before)
- Lowered minimum configurable wait for pagination from 1000ms to 500ms (default remains 1000ms)

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 07.06.2022 at 16:38.
Reply With Quote
  #3248  
Old 07.06.2022, 16:39
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 77,929
Default

Quote:
Originally Posted by StefanM View Post
Question: Does this core update also contain the changes you made to the VK.com PlugIn?
See https://support.jdownloader.org/Know...pdate-behavior
core update just means it also contains updates for non plugin parts
pending core updates can delay plugin-only updates
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 07.06.2022 at 16:58.
Reply With Quote
  #3249  
Old 07.06.2022, 17:28
Student im ersten Jahr Student im ersten Jahr is offline
Peta Loader
 
Join Date: Nov 2020
Posts: 180
Default

I have a question: can JD2 auto-save direct image links to a text file? Direct links do not expire! But it's very valuable because I can access a backup of any image file on the VK.

Example DIRECT LINK:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Source: **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #3250  
Old 07.06.2022, 17:30
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 63,166
Default

Quote:
Originally Posted by Student im ersten Jahr View Post
I have a question: can JD2 auto-save direct image links to a text file? Direct links do not expire! But it's very valuable because I can access a backup of any image file on the VK.
No.
But you can use an EventScripter script to save those URLs in textfiles.
They're stored on each DownloadLink as property "picturedirectlink" and/or "directurls_fallback".
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3251  
Old 07.06.2022, 17:36
Student im ersten Jahr Student im ersten Jahr is offline
Peta Loader
 
Join Date: Nov 2020
Posts: 180
Default

To be honest, I am not an expert to write java-script code.
Reply With Quote
  #3252  
Old 07.06.2022, 17:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 63,166
Default

Quote:
Originally Posted by Student im ersten Jahr View Post
To be honest, I am not an expert to write java-script code.
Neither am I.
You can ask for help in this thread.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3253  
Old 07.06.2022, 20:04
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 354
Default

Quote:
Originally Posted by Student im ersten Jahr View Post
I have a question: can JD2 auto-save direct image links to a text file? Direct links do not expire! But it's very valuable because I can access a backup of any image file on the VK.

Example DIRECT LINK:
**External links are only visible to Support Staff**...

Source: **External links are only visible to Support Staff**...
@Student im ersten Jahr

Again, in the meantime just use the Firefox AddOn SaveFrom.net helper I mentioned. If you want, you can copy the links to the photos and paste them in Linkrabber.

It is the format as in your Example 'DIRECT LINK'

And that AddOn is sill working without any problems.
You can use it for all photos or just for galleries.

Last edited by StefanM; 07.06.2022 at 21:16.
Reply With Quote
  #3254  
Old 08.06.2022, 12:32
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 63,166
Default

Quote:
Originally Posted by StefanM View Post
Again, in the meantime just use the Firefox AddOn SaveFrom.net helper I mentioned. If you want, you can copy the links to the photos and paste them in Linkrabber.
I guess he wants to have the filename options JD has

Quote:
Originally Posted by StefanM View Post
It is the format as in your Example 'DIRECT LINK'
Yeah but it's missing other metadata/filenames.

Quote:
Originally Posted by StefanM View Post
And that AddOn is sill working without any problems.
Our plugin too
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3255  
Old 09.06.2022, 16:04
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 354
Default

Quote:
Originally Posted by pspzockerscene View Post
Our plugin too
That's good. Also I would prefer using JD over the Firefox AddOn.

But what naming are you referring to?
Just tested this album: **External links are only visible to Support Staff****External links are only visible to Support Staff**
I get just these names: -27763046_323043687.jpg (example)
when downloading from that album

One more thing...
As there is no changelog and - as I had to learn - there will be no changelog, how do I know, in which core release an issue/bug has been fixed.

The last reaction I received (after I retested Student's findings) was: "No need to retest, we are aware of the problem and it will be fixed."

How, the devil can I find out, when an issue has been fixed and retesting makes sense?
Reply With Quote
  #3256  
Old 09.06.2022, 16:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 63,166
Default

Quote:
Originally Posted by StefanM View Post
I get just these names: -27763046_323043687.jpg (example)
when downloading from that album
Yap.
I was referring to the other capabilities that JD has (packagizer, plugin properties and so on) which your addon might not have.

Quote:
Originally Posted by StefanM View Post
As there is no changelog and - as I had to learn - there will be no changelog, how do I know, in which core release an issue/bug has been fixed.
It's all described in our update FAQ:
https://support.jdownloader.org/Know...pdate-behavior
Basically whenever a fix is proposed here, it will be included in the next revision so just wait until you get an update and the revision number of your JDownloader increases (+ build date).
You may also check/view our source:
https://support.jdownloader.org/Know...up-ide-eclipse
As explained, we do make use of commit messages and those may also include more detailed information about fixes in plugins and so on.

Quote:
Originally Posted by StefanM View Post
The last reaction I received (after I retested Student's findings) was: "No need to retest, we are aware of the problem and it will be fixed."
No not exactly.
First I announced the fix here:
https://board.jdownloader.org/showpo...postcount=3247
...and since then, several updates have been released so yes the changes I mentioned have been released.
You can also if you want, monitor the "last changed" date of the affected plugin classes.
In this case:
See [JD install dir]/plugins/hoster/VKontakteRuHoster and [JD install dir]/plugins/decrypted/VKontakteRu
You can also view my commit here:
github.com/mirror/jdownloader/commit/79a3b55fe00ddfc98ad79e0ad51da52ae58adab3
Usually if changes are related to forum bug reports we will include the threadID in our commit messages, also in this case
Each commit will increase the revision number in source code:
Code:
$Revision: <Numbers> $
--> As long as your stable version is below that revision, you obviously did not get those changes yet.

Quote:
Originally Posted by StefanM View Post
How, the devil can I find out, when an issue has been fixed and retesting makes sense?
Well now you know at least 3 ways to find out.

-psp-
EDIT

In some cases we do notify ticket/forum users separately about updates/CORE-updates but most of all times we rely on our update FAQ and users simply noticing the next update + restart of JD.
If you got suggestions to improve our Update FAQ please let us know.
EDIT2

What I forgot:
Now always but most of all times we do use our forum thread prefixes for update announcements.
In this case I didn't do this because the vk thread is so long and the issue you reported affected only you atm so I did not want other users to think that there was some kind of problem with the plugin and also the majority of it worked so I left the prefix of this thread on "Solved".
As you can see, we do not have any fixed guidelines for the announcement of updates/changes
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by raztoki; 12.06.2022 at 02:43. Reason: spelling
Reply With Quote
  #3257  
Old 12.06.2022, 02:46
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,460
Default

I recall a lot of the request issues in the past. What was the cause back then was multi-threading and plugin returning results from decrypter as it was finding results, and hoster plugin maybe doing online checking concurrently, and or user was downloading too. It took many hours to find those issues and lock it down. I haven't worked on the VK plugin for years but complexity was similar to facebook and constant website ui changes. Fun times.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #3258  
Old 14.06.2022, 11:45
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 63,166
Default

Nowdays Facebook is / would be more complex and vk plugin is currently in a good state.
Still it would be much easier if we'd have just implemented the vk API but I was always afraid of bans.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3259  
Old 19.06.2022, 12:59
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 354
Default VK.com Playlists: Plugin not working

Don't know since when: Playlists can no longer be crawled.

Just tested these:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

vk.com/video/playlist/-9745_42124286
vk.com/video/playlist/-9745_43376314
vk.com/video/playlist/-9745_43376150
vk.com/video/playlist/-9745_42128214

'Crawling for Downloads' just pops up for a second. That's it!
Deep Scan doesn't make any difference.
Reply With Quote
  #3260  
Old 19.06.2022, 16:36
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 77,929
Default

@StefanM: minimal json layout change, should be working again with next plugin update
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 05:55.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.