JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 22.10.2020, 12:43
Etshy Etshy is offline
Vacuum Cleaner
 
Join Date: Oct 2019
Posts: 17
Default Hitomi.la

Seems like hitomi.la is broken.

After a little investigation, they added AVIF file format in their sources tag, but not sure if that is what cause the problem.

In their sources, the code hasn't changed, so I guess the problem comes from their AVIF sources ?
Reply With Quote
  #2  
Old 22.10.2020, 12:45
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,222
Default

Hi,

as always, please post example URLs.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3  
Old 22.10.2020, 14:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,222
Default

Found the issue and "fixed" webp handling.
I'm not really sure about this but it works for the URLs I've tested ...

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #4  
Old 22.10.2020, 18:02
damo damo is offline
Vacuum Cleaner
 
Join Date: Jan 2020
Posts: 15
Exclamation

They have slightly changed the code from the version I posted in Post https://board.jdownloader.org/showpo...8&postcount=16

IMO it still should grab the original .jpg/.png, although this could be a configurable option for the plugin, as some people might prefer the smaller webp versions. BTW they have added .avif as well as with webp there is a "hasavif: 1" entry in the <galleryid>.js file. Furthermore, I'd really appreciate a way to keep the original file names, e.g., "105_chapter10_8.jpg" instead of just "60.jpg".

Here is what has changed
Code:
 function subdomain_from_url(url, base) {
-        var retval = 'a';
+        var retval = 'b';

// ...
         var m = r.exec(url);
         if (!m) {
-                return retval;
+                return 'a';
         }
Here are the changed functions, further down is the full version.

Code:
  String subdomain_from_url(String url, String base) {
    String retval = "b";
    if (base != null) {
      retval = base;
    }

    int number_of_frontends = 3;

    Matcher m = SUBDOMAIN_FROM_URL_PATTERN.matcher(url);
    if (!m.find()) {
      return "a";
    }

    try {
      int g = Integer.parseInt(m.group(1), 16);
      if (g < 0x30) {
        number_of_frontends = 2;
      }
      if (g < 0x09) {
        g = 1;
      }
      retval = subdomain_from_galleryid(g, number_of_frontends) + retval;
    } catch (NumberFormatException ignore) {}

    return retval;
  }

  String url_from_hash(String galleryid, Map<String, String> image, String dir, String ext) {
    ext = isNotBlank(ext) ? ext : (isNotBlank(dir) ? dir :
        image.get("name").split("\\.")[1]);
    dir = isNotBlank(dir) ? dir : "images";

    return "**External links are only visible to Support Staff** + dir + '/' + full_path_from_hash(image.get("hash")) + '.' + ext;
  }
The best way to get Test Data is to open a gallery and set this breakpoint in the DevTools in common.js:65

Code:
'Assert.assertEquals("'+url_from_url(url_from_hash(galleryid, image, dir, ext), base)+'", url_from_url_from_hash("'+galleryid+'", mapOf("'+ image.hash+'", "'+ image.name+'"), '+ (dir ? '"'+dir+'"' : 'null') +', '+ (ext ? '"'+ext+'"' : 'null') +', '+ (base ? '"'+base+'"' : 'null') +'));'
This will log out assert statements that can be copy pasted into a test and then you only need this helper function to test the current code.

Code:
  Map<String, String> mapOf(String hash, String name) {
    HashMap<String, String> m = new HashMap<>();
    m.put("hash", hash);
    m.put("name", name);
    return m;
  }
Code:
    Assert.assertEquals("**External links are only visible to Support Staff**, url_from_url_from_hash("1760873", mapOf("7c3d8d36f3a73375b1194207d95127d62002f0ce94d1d9a6045c21a9bed2c78e", "105_chapter10_8.jpg"), null, null, null));
    Assert.assertEquals("**External links are only visible to Support Staff**,  url_from_url_from_hash("1760873", mapOf("7c3d8d36f3a73375b1194207d95127d62002f0ce94d1d9a6045c21a9bed2c78e", "105_chapter10_8.jpg"), "avif", null, "a"));
    Assert.assertEquals("**External links are only visible to Support Staff**,  url_from_url_from_hash("1760873", mapOf("7c3d8d36f3a73375b1194207d95127d62002f0ce94d1d9a6045c21a9bed2c78e", "105_chapter10_8.jpg"), "webp", null, "a"));
As you can see, you need to pass null for dir, ext, and base for the original image and "avif"/"webp" for dir plus "a" for base for the avif/webp versions.

This is the current working version:

Code:
  public static final Pattern SUBDOMAIN_FROM_URL_PATTERN = Pattern.compile("/[0-9a-f]/([0-9a-f]{2})/");
  public static final Pattern URL_FROM_URL_PATTERN = Pattern.compile("//..?\\.hitomi\\.la/");
  public static final Pattern FULL_PATH_FROM_HASH_PATTERN = Pattern.compile("^.*(..)(.)$");



  String subdomain_from_galleryid(int g, int number_of_frontends) {
    int o = g % number_of_frontends;

    return String.valueOf((char)(97 + o));
  }

  String subdomain_from_url(String url, String base) {
    String retval = "b";
    if (base != null) {
      retval = base;
    }

    int number_of_frontends = 3;

    Matcher m = SUBDOMAIN_FROM_URL_PATTERN.matcher(url);
    if (!m.find()) {
      return "a";
    }

    try {
      int g = Integer.parseInt(m.group(1), 16);
      if (g < 0x30) {
        number_of_frontends = 2;
      }
      if (g < 0x09) {
        g = 1;
      }
      retval = subdomain_from_galleryid(g, number_of_frontends) + retval;
    } catch (NumberFormatException ignore) {}

    return retval;
  }

  String url_from_url(String url, String base) {
    return URL_FROM_URL_PATTERN.matcher(url).replaceAll("//" + subdomain_from_url(url, base) + ".hitomi.la/");
  }


  String full_path_from_hash(String hash) {
    if (hash.length() < 3) {
      return hash;
    }
    return FULL_PATH_FROM_HASH_PATTERN.matcher(hash).replaceAll("$2/$1/" + hash);
  }

  String url_from_hash(String galleryid, Map<String, String> image, String dir, String ext) {
    ext = isNotBlank(ext) ? ext : (isNotBlank(dir) ? dir :
        image.get("name").split("\\.")[1]);
    dir = isNotBlank(dir) ? dir : "images";

    return "**External links are only visible to Support Staff** + dir + '/' + full_path_from_hash(image.get("hash")) + '.' + ext;
  }

  String url_from_url_from_hash(String galleryid, Map<String, String> image, String dir, String ext, String base) {
    return url_from_url(url_from_hash(galleryid, image, dir, ext), base);
  }

  boolean isNotBlank(String str) {
    return str != null && !str.isEmpty();
  }
Reply With Quote
  #5  
Old 23.10.2020, 15:08
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,222
Default

Sorry I'm still not so much into js stuff.
I've tested your code and it doesn't seem to work.
Here is the random URL I've used for testing:
hitomi.la/cg/宇崎ちゃんと遊びたい!-日本語-1761639.html
The .jpg URLs which your code has returned were invalid, resulting in 403 errors instead of files.

Thanks for working on this - I for sure would not have found the time to properly fix this!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #6  
Old 23.10.2020, 23:11
damo damo is offline
Vacuum Cleaner
 
Join Date: Jan 2020
Posts: 15
Default

I've checked the urls they are correct, I assume that you have to set the correct referrer when requesting the image. When I opened the calculated urls directly it also gave me 403, but then I opened it via the gallery it worked and it was the same url.

If you open the gallery you can switch to the original by pressing "w" that should toggle the file type between jpg/png and webp.
Reply With Quote
  #7  
Old 24.10.2020, 20:15
Etshy Etshy is offline
Vacuum Cleaner
 
Join Date: Oct 2019
Posts: 17
Default

Sorry I was'nt available these days and didn't had time to look more into their codes etc.

When I posted the first post, links that had the new AVIF Source (in html) didn't worked.

Tried today, The comics are now downloaded (with or without the new AVIF extension) in WebP.

not really a problem for me as I installed the webP codec but might cause problem for others (I don't know if WebP can be opened without the webp codec and without using a browser)

Edit :
Seems like old links without webp or avif doesn't work
example : **External links are only visible to Support Staff****External links are only visible to Support Staff**
Jdownloader try to download the jpg/png and fail with "File not found" message
maybe the referrer error damo talked about.

little question, is there a way to see the exact download url jdownloader detect ?
When I try to right click on an image > open in browser (or show download URL) it returns the comic URL not the image URL.

Edit2 : I updated my js tampermonkey script to download hitomi images.
It works with all links I tested (without webp AND avif, with webp AND without avif, with both webp AND avif) and it download in png

I still don't know a thing about JAVA so I can't translate this to JAVA, but here is a link for my script **External links are only visible to Support Staff****External links are only visible to Support Staff**.

I use an img tag to download the image but I guess JAVA have many tools to do better than that.
The img tag crossOrigin="Anonymous" parameter is mandatory to download.


Hope this can help.

Last edited by Etshy; 24.10.2020 at 22:07.
Reply With Quote
  #8  
Old 26.10.2020, 16:30
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,222
Default

@damo
Are you sure they're checking the referer?
Why would they do this for .jpg images but not for .webp?

@Etshy
That doesn't help me.
To answer your question though:
Quote:
Originally Posted by Etshy View Post
little question, is there a way to see the exact download url jdownloader detect ?
Settings -> User Interface -> Downoadlink address display -> Move "Data" to the top or deactivate all other options

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #9  
Old 01.11.2020, 19:26
damo damo is offline
Vacuum Cleaner
 
Join Date: Jan 2020
Posts: 15
Default

@pspzockerscene
I just verified it you need to include the Referer header, actually for both jpg/png and webp. My guess is that if you tried it from the browser, you might have had the webp already in the cache.
As for which url to use as referer, afaics it should be
Code:
**External links are only visible to Support Staff**
So with the updated algorithm, it should work again for jpg/png.

Any thoughts on my other requests in regards to using original names and choosing which image version to download?
Reply With Quote
  #10  
Old 02.11.2020, 17:52
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,222
Default

Fixed - the issue was not caused by the missing referer.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #11  
Old 02.11.2020, 18:22
damo damo is offline
Vacuum Cleaner
 
Join Date: Jan 2020
Posts: 15
Default

Nice, but could you elaborate a bit more?
Reply With Quote
  #12  
Old 02.11.2020, 18:42
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,279
Default

small misstake, he added protocol twice
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 02.11.2020, 18:44
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,222
Default

Quote:
Originally Posted by damo View Post
Nice, but could you elaborate a bit more?
I was about to check it - I've just made it work with your changes before although I knew something else had caused the issue.

Thanks to Jiaz for re-checking it.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 09:15.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.