[WRESTLiNGHDD]

Danke für deine Arbeit.
Für YouTube benötige ich es, weil JD "nur" ~1000 Videos eines Kanals crawlt und dann aufhört. Benutze ich, wenn es weitaus mehr sind.

Habe JD geupdated:

Meine Tests:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Crawlt 999 von 14504
Das gleiche Profil, nur die "added" Sektion statt "uploaded":
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Crawlt 1 von 14504

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Crawlt alles
Das gleiche Profil, nur die "added" Sektion statt "uploaded":
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Passiert nichts

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Crawlt 993 von 1876
Das gleiche Profil, nur die "added" Sektion statt "uploaded":
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Passiert nichts

**External links are only visible to Support Staff****External links are only visible to Support Staff**
crawlt alles
**External links are only visible to Support Staff****External links are only visible to Support Staff**
11 von 1004

Dass beim Crawlen von "added" nur die restlichen Videos hinzugefügt werden, die bei "uploaded" nicht dabei waren, erscheibt mir logisch.

Ich erkenne leider keine Systematik, wieso es nicht 100%ig fuktioniert.
Aber es ist auf jeden Fall ein großer Fortschritt. Kann man da noch was machen evtl.?
Danke nochmal.
09.08.24 08.01.36 <--> 09.08.24 08.14.21 jdlog://3274411370661/

[Jiaz]

Originally Posted by WRESTLiNGHDD

Aber es ist auf jeden Fall ein großer Fortschritt. Kann man da noch was machen evtl.?

Danke für die Beispiele! Wir werden uns das natürlich anschauen

[pspzockerscene]

Originally Posted by WRESTLiNGHDD

Für YouTube benötige ich es, weil JD "nur" ~1000 Videos eines Kanals crawlt und dann aufhört.

Das sollte nicht passieren. Melde das Problem mit Testlinks gerne im YT Thread:
https://board.jdownloader.org/showthread.php?t=48894

Originally Posted by WRESTLiNGHDD

Crawlt 999 von 14504
[...]
Crawlt 1 von 14504

Fixed.
Hintergrund:
Das Plugin erwartete, dass eine 'volle Seite' immer 1000 Elemente enthält, aber das vk System kann zwischendrin auch eine andere Anzahl von Elementen zurückgeben ohne, dass das Ende erreicht ist.
Dadurch dachte das Plugin es wäre auf der letzten Seite und stoppte zu früh.
Ich bekomme beim ersten Link 14416 Elemente von angeblich 14504, beim zweiten Link 14362.
Herauszufinden ob wirklich welche fehlen und welche, überlasse ich dir (mehr dazu weiter unten).

Originally Posted by WRESTLiNGHDD

Das gleiche Profil, nur die "added" Sektion statt "uploaded":
...
Passiert nichts

Funktioniert hier einwandfrei.
Ich bekomme 1453 Ergebnisse.

Originally Posted by WRESTLiNGHDD

11 von 1004

Ich bekomme 1003 Elemente.

Warte am besten aufs nächste Update, bevor du weiteres Feedback gibst.

Bzlg potentiell fehlender Elemente habe ich nicht weiter geforscht, aber ich vermute, dass das so zustande kommt:
- Manche Videos wurden gelöscht, aber der Zähler nicht aktualisiert
- Manche Videos sind doppelt enthalten (selbe videoID), aber in JD kann jede eindeutige VideoID nur 1x hinzugefügt werden
- Oder: vk hat ein Bug im Pagination System, wodurch manche Elemente tatsächlich "verloren gehen"

Auch Betreiber großer Webseiten machen Fehler und einen solchen haben wir auch schonmal an anderer Stelle entdeckt daher prüfe es gerne. Es kann natürlich auch an einem Bug im Plugin liegen.

VK Änderungen fürs nächste Update (interner Changelog):
- fixed pagination when page size is lower than expected max such as 999 items on a page instead of 1000
- video pagination: now respects 'sleep between pages' plugin setting
- deleted a lot of old audio crawl functionality as audio streams are DRM protected and cannot be downloaded with JD anyways

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-
EDIT

Nachtrag, da du mir per PN eine Frage bzlg "CORE Updates" gestellt hattest:
In dem Fall ist die Art des Updates auch egal.
Du hast die besagten Änderungen, sobald sich die von vkontakte zusammenhängenden JDownloader Plugin-Klassen das nächste mal nach einem Update ändern.

[patriks]

Hello, how are you?

After the latest updates, the "linkgrabber" no longer works with personal album links. It used to download the first 20 images, but now it only happens with page albums. Public albums (albums-14839763) are working perfectly as far as I can tell.

Here is the log and the links I used in the test:

11.08.24 19.15.45 <--> 11.08.24 19.27.28 jdlog://2974411370661/

n work: **External links are only visible to Support Staff****External links are only visible to Support Staff**
n work: **External links are only visible to Support Staff****External links are only visible to Support Staff**
n work: **External links are only visible to Support Staff****External links are only visible to Support Staff**
n work: **External links are only visible to Support Staff****External links are only visible to Support Staff**
n work: **External links are only visible to Support Staff****External links are only visible to Support Staff**

ok: **External links are only visible to Support Staff****External links are only visible to Support Staff**
ok: **External links are only visible to Support Staff****External links are only visible to Support Staff**

:helpsmilie:

[pspzockerscene]

Originally Posted by patriks

Hello, how are you?

Good, thanks.

Originally Posted by patriks

After the latest updates, the "linkgrabber" no longer works with personal album links.

Works fine here.
Did you add your vk.com account to JDownloader?
I haven't touched any of the album crawl functionality.

Please provide a (new) debug-log.

Please post your log-ID here
If your report is about a specific website which JD supports via plugin, please also provide example URLs which can be used to reproduce the issue you are having.
If your report is related to a login specific problem with a plugin supported website, enable debug mode before creating logs, see previously linked instructions.

Bitte poste deine Log-ID hier.
Falls dein Problem ein Problem mit einer Webseite ist, die per Plugin unterstützt wird, stelle bitte zusätzlich Testlinks zur Verfügung, mit denen sich dein Problem nachstellen lässt.
Geht es um einen nicht funktionierenden Account-Login, aktiviere vor dem Erstellen deines Logs den debug Modus (siehe zuvor verlinkte Anleitung).

-psp-

[Jiaz]

@patriks: please check again with next update

[patriks]

Originally Posted by Jiaz

@patriks: please check again with next update

16.08.24 16.17.24 <--> 16.08.24 16.17.10 jdlog://7094411370661/

After many tests, I discovered that the problem might be with the login. Here's what I did:

I took files X and Y from an my backup of JD (where my login was saved).
I placed them in the newly downloaded JD, where I wasn't able to download while logged in (even the public album, which is accessible without an account). The logs are from this JD, and I also placed the files in the one where I had the issue initially.
It was able to find all the albums now.

I logged in correctly, but maybe it’s not able to detect the login or save it properly in the file in the new version?

X= org.jdownloader.settings.AccountSettings.json
Y= org.jdownloader.settings.AccountSettings.accounts.ejs

links I tested:
work: **External links are only visible to Support Staff****External links are only visible to Support Staff**
work: **External links are only visible to Support Staff****External links are only visible to Support Staff** only 20 photos
work: **External links are only visible to Support Staff****External links are only visible to Support Staff**

work: **External links are only visible to Support Staff****External links are only visible to Support Staff**
work: **External links are only visible to Support Staff****External links are only visible to Support Staff**

[patriks]

Sorry for the inconvenience, but I would really like to ask for one thing, an option that can be activated: that it downloads the original image and not the image converted by the server, I don't know why VK does this, but it converts all the images and often ends up being twice the size. Example:

converted: **External links are only visible to Support Staff****External links are only visible to Support Staff**impg/UsF_Ur8tOGxv2VjdQMUTeOskJjetRc2nTq0WqQ/zN4qgC2fcQE.jpg?size=1620x2160&quality=95&sign=4772b127c48c68c9c3a4b2b37014ec1b&type=album

original: **External links are only visible to Support Staff****External links are only visible to Support Staff**

link: **External links are only visible to Support Staff****External links are only visible to Support Staff**

thx

[Jiaz]

Originally Posted by patriks

I don't know why VK does this, but it converts all the images and often ends up being twice the size.

Hmmm,not for me?!
vk version (with params) = 1.58MB
original version? (without params) = 2.32 MB
Have you mixed up your tests ?
Does this URL rewrite work always? if so, of course we could add an option for it.

[patriks]

Originally Posted by Jiaz

Hmmm,not for me?!
vk version (with params) = 1.58MB
original version? (without params) = 2.32 MB
Have you mixed up your tests ?
Does this URL rewrite work always? if so, of course we could add an option for it.

I used a bad example,, it was the first image I took from my timeline to give as an example

but I guarantee that in the vast majority of cases this is not what happens, I had already noticed this problem some time ago.
I suggested it as an "option" to avoid causing problems in links that may not have this pattern, but the pattern I have noticed so far is this.

// new links

**External links are only visible to Support Staff****External links are only visible to Support Staff**
converted: 610 KB

**External links are only visible to Support Staff****External links are only visible to Support Staff**
original: 381 KB

// old link 2014

**External links are only visible to Support Staff****External links are only visible to Support Staff**
converted: 387 KB

**External links are only visible to Support Staff****External links are only visible to Support Staff**
original: 242 KB

I believe it is always better to get the original image without unnecessary conversions. This also makes it easier to find duplicates.

[pspzockerscene]

Originally Posted by patriks

I believe it is always better to get the original image without unnecessary conversions. This also makes it easier to find duplicates.

I agree.

Jiaz' question was if you think/know that the original image is always available.

[Jiaz]

[Jiaz]

@pspzockerscene: Can you check this? https://board.jdownloader.org/showpo...postcount=3487

[patriks]

Originally Posted by pspzockerscene

I agree.

Jiaz' question was if you think/know that the original image is always available.

Yes, I made a python script with chatgpt that modifies the links inside the downloadList*.zip files.

You just need to remove "impg/" or "impf/" + everything that comes after .jpg

Just out of curiosity, here is the excerpt from the python script I used:

def modify_directlink(url):
    """Modifies the URL by removing 'impg/', 'impf/', and everything after the image file extension."""
    parsed_url = urlparse(url)
    path_parts = parsed_url.path.split('/')
    
    # Identifies the last part of the path (file with extension)
    file_name_with_ext = path_parts[-1]
    if '.' in file_name_with_ext:
        # Removes 'impg/' and 'impf/' from the path
        path_parts = [part for part in path_parts if part not in ('impg', 'impf')]
        
        # Removes everything after the file extension
        path_parts = path_parts[:path_parts.index(file_name_with_ext) + 1]
        new_path = '/'.join(path_parts)
        
        # Recreates the URL without 'impg/', 'impf/' and parameters after the file
        new_url = urlunparse(parsed_url._replace(
            path=new_path,
            query='',
            fragment=''
        ))
        return new_url
    return url

[pspzockerscene]

Added plugin setting for this (default = turned off).
Code snippet:

    private String modifyFinalDownloadurl(final String input) {
        final Regex photo_url_regex;
        if (this.getPluginConfig().getBooleanProperty(VKPHOTOS_DOWNLOAD_ORIGINAL_IMAGE_VIA_MAGIC_TRICK, default_VKPHOTOS_DOWNLOAD_ORIGINAL_IMAGE_VIA_MAGIC_TRICK) && (photo_url_regex = new Regex(input, Pattern.compile("(https?://[^/]+)/imp(?:f|g)/([^\\?]+)\\?.+", Pattern.CASE_INSENSITIVE))).patternFind()) {
            /* Modify URL according to: https://board.jdownloader.org/showpost.php?p=538448&postcount=3490 */
            final String new_url = photo_url_regex.getMatch(0) + "/" + photo_url_regex.getMatch(1);
            return new_url;
        } else {
            final Regex urlregex = new Regex(input, "(https?://cs(\\d+)\\.vk\\.me/)");
            if (!urlregex.patternFind()) {
                /* Return unmodified url */
                return input;
            }
            /* 2024-08-20: This is very old. I'm unsure whether this is still needed and working. */
            final String oldserver = urlregex.getMatch(0);
            final String serv_id = urlregex.getMatch(1);
            final String newserver = "**External links are only visible to Support Staff** + serv_id + "/";
            return input.replace(oldserver, newserver);
        }
    }

You can obtain our full source code here:
https://support.jdownloader.org/de/k...up-ide-eclipse

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-