JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 15.01.2020, 00:12
Etshy Etshy is offline
DSL Light User
 
Join Date: Oct 2019
Posts: 34
Default Hitomi.la

Jdownloader doesn't grab images from comics' page anymore.
Reply With Quote
  #2  
Old 18.01.2020, 18:12
BJN01 BJN01 is offline
JD Adviser
 
Join Date: Jan 2020
Posts: 113
Default

es link

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**


the name of the files has become something like:
"38e4562c59213f809fd766b31c4bbcde165a08691f395df9a706ab5f1a704b3e.webp"

Last edited by BJN01; 18.01.2020 at 18:15.
Reply With Quote
  #3  
Old 19.01.2020, 01:50
Etshy Etshy is offline
DSL Light User
 
Join Date: Oct 2019
Posts: 34
Default

Can't say if the webP usage is new or not.
but the hash part is the same in the thumbnail and in the original image, so the only difficult thing is the subdomain part, I guess.
I made a tapermonkey (dirty) script to try some things and download comics.

I dunno how Jdownloader works, but if it can load/execute javascript it's quite easy to get the subdomain (there is function in the common.js from hitomi.la for that.
if it can't , converting these functions should lead to a fix I guess.

Edit: Btw I don't know how hitomi itself download their own comics, but after few tests my dirty script seems to be about 60% more efficient ...

Last edited by Etshy; 19.01.2020 at 01:54.
Reply With Quote
  #4  
Old 19.01.2020, 05:38
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,507
Default

Maybe you can share your script.
Or try eclipse for java and update the plugin.
__________________
FAQ: How to upload a Log
Reply With Quote
  #5  
Old 19.01.2020, 20:34
Etshy Etshy is offline
DSL Light User
 
Join Date: Oct 2019
Posts: 34
Default

Quote:
Originally Posted by tony2long View Post
Maybe you can share your script.
Or try eclipse for java and update the plugin.
Pasted the userscript in this pastebin
**External links are only visible to Support Staff****External links are only visible to Support Staff**

I used their owns functions to get the subdomains so I don't know it will be useful.

I don't really know how to JAVA so can't really help for that ^^"
But the functions are all in the common.js (**External links are only visible to Support Staff****External links are only visible to Support Staff**) I guess a real JAVA dev will be able to convert the code faster than me.
Reply With Quote
  #6  
Old 20.01.2020, 20:21
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Sorry I'am not really into js.
Here a ticket:

I failed to understand where certain parts of their final URLs come from e.g.
ca.hitomi.la/webp/c/1a/a6fc7ba071a0efd3970e90209c2dae1ba2e7374d966b194a8749a88cd31cd1ac.webp

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #7  
Old 21.01.2020, 00:20
Etshy Etshy is offline
DSL Light User
 
Join Date: Oct 2019
Posts: 34
Default

From what I see the "c/1a" in you example is the same as the thumbnails (I guess you should have access to them when we copy the comics URL).
If you have an example where it's not the case, let me know, I'll look if it's generated elsewhere.

For the subdomain part that seems to change often, even in the same comic, I'll try to dig a bit.

First you need their comicId (they name it galleryId in their files)
You can get it via the "Read Online" button, that's the number part, just before ".html"

Then you need to get their "json" (it's a js array/object but that's basically a json)
in this URL : "//ltn.hitomi.la/galleries/" + galleryid + ".js"
Example : //ltn.hitomi.la/galleries/1553604.js
By deleting "var galleryinfo = " you can get a JSON.

Then you can loop this json and call
url_from_url_from_hash(galleryid, our_galleryinfo[i], 'webp');
Here is a pastebin copy of their common.js with url related functions
**External links are only visible to Support Staff****External links are only visible to Support Staff**
You can "copy" and "convert" the code (it's mostly a bit of regex and String.replace things)

If there is specific part in the JS where you need help, let me know.



edit: btw, it seems my call would work only if the comic have webP (as I pass "webp" as argument)
(seems all news comics have webP but not sure).
You can have the information if the image have a webP in the JSON linked, so a condition on that would be needed to pass webP if true and null if false (I guess, not tested)

Last edited by Etshy; 21.01.2020 at 00:25.
Reply With Quote
  #8  
Old 21.01.2020, 00:46
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

We are already using that .js site to get the Array.

I was unable to find the full list of thumbnail URLs anywhere.
These would already be enough to "convert" them to the URLs leading to the full images.
Our code of this plugin is a bit messy but here you go:
Spoiler:
Code:
//jDownloader - Downloadmanager
//Copyright (C) 2009  JD-Team support@jdownloader.org
//
//This program is free software: you can redistribute it and/or modify
//it under the terms of the GNU General Public License as published by
//the Free Software Foundation, either version 3 of the License, or
//(at your option) any later version.
//
//This program is distributed in the hope that it will be useful,
//but WITHOUT ANY WARRANTY; without even the implied warranty of
//MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
//GNU General Public License for more details.
//
//You should have received a copy of the GNU General Public License
//along with this program.  If not, see <**External links are only visible to Support Staff**
package jd.plugins.decrypter;

import java.text.DecimalFormat;
import java.util.ArrayList;

import org.appwork.utils.StringUtils;
import org.jdownloader.plugins.components.antiDDoSForDecrypt;

import jd.PluginWrapper;
import jd.controlling.ProgressController;
import jd.http.Browser;
import jd.http.URLConnectionAdapter;
import jd.http.requests.HeadRequest;
import jd.nutils.encoding.Encoding;
import jd.parser.Regex;
import jd.plugins.CryptedLink;
import jd.plugins.DecrypterException;
import jd.plugins.DecrypterPlugin;
import jd.plugins.DownloadLink;
import jd.plugins.FilePackage;
import jd.plugins.LinkStatus;
import jd.plugins.PluginException;

/**
 *
 * @author raztoki
 *
 */
@DecrypterPlugin(revision = "$Revision: 41693 $", interfaceVersion = 3, names = { "hitomi.la" }, urls = { "https?://(www\\.)?hitomi\\.la/(?:galleries/\\d+\\.html|reader/\\d+\\.html|[^/]+/.*?-\\d+\\.html)" })
public class HitomiLa extends antiDDoSForDecrypt {
    public HitomiLa(PluginWrapper wrapper) {
        super(wrapper);
    }

    public ArrayList<DownloadLink> decryptIt(CryptedLink param, ProgressController progress) throws Exception {
        final ArrayList<DownloadLink> decryptedLinks = new ArrayList<DownloadLink>();
        final String parameter = param.toString();
        String guid = new Regex(parameter, "/(?:galleries|reader)/(\\d+)").getMatch(0);
        if (guid == null) {
            guid = new Regex(parameter, "/[^/]+/.*?-(\\d+)\\.html").getMatch(0);
        }
        if (guid == null) {
            throw new PluginException(LinkStatus.ERROR_PLUGIN_DEFECT);
        }
        br.setFollowRedirects(true);
        final String fpName = br.getRegex("<title>([^<>\"]*?) \\| Hitomi\\.la</title>").getMatch(0);
        int i = 0;
        String imghost = getImageHost(guid) + "a";
        int numberOfPages;
        final boolean use_Thumbnails = false;
        if (use_Thumbnails) {
            this.getPage(parameter);
            if (br.getHttpConnection().getResponseCode() == 404) {
                decryptedLinks.add(createOfflinelink(parameter));
                return decryptedLinks;
            }
            final String[] thumbnails = br.getRegex("<img src=\"//tn\\.hitomi\\.la/bigtn/(./\\d+/[a-f0-9]+)\\.jpg\">").getColumn(0);
            numberOfPages = thumbnails.length;
            final DecimalFormat df = numberOfPages > 999 ? new DecimalFormat("0000") : numberOfPages > 99 ? new DecimalFormat("000") : new DecimalFormat("00");
            for (final String thumbnail : thumbnails) {
                final DownloadLink dl = createDownloadlink("direct**External links are only visible to Support Staff** + imghost + ".hitomi.la/webp/" + thumbnail + ".webp");
                dl.setProperty("Referer", br.getURL());
                dl.setProperty("requestType", "GET");
                dl.setAvailable(true);
                dl.setFinalFileName(df.format(i) + ".webp");
                decryptedLinks.add(dl);
            }
        } else {
            /* Avoid https, prefer http */
            getPage("**External links are only visible to Support Staff** + guid + ".html");
            if (br.getHttpConnection().getResponseCode() == 404) {
                decryptedLinks.add(createOfflinelink(parameter));
                return decryptedLinks;
            }
            // get the image host.
            // retval = subdomain_from_galleryid(g) + retval;
            String[] links = br.getRegex("(/" + guid + "/(?:[^<>\"]*?\\.[a-z]+)+)").getColumn(0);
            if (links == null || links.length == 0) {
                final String js = br.getRegex("src\\s*=\\s*\"([^\"]+" + guid + "\\.js)\"").getMatch(0);
                if (js != null) {
                    final Browser brc = br.cloneBrowser();
                    getPage(brc, js);
                    final String names[] = brc.getRegex("\"name\"\\s*:\\s*\"(.*?)\"").getColumn(0);
                    if (names != null && names.length > 0) {
                        final ArrayList<String> urls = new ArrayList<String>();
                        for (String name : names) {
                            urls.add("/" + guid + "/" + name);
                        }
                        links = urls.toArray(new String[0]);
                    }
                }
            }
            if (links == null || links.length == 0) {
                logger.warning("Decrypter broken for link: " + parameter);
                return null;
            }
            numberOfPages = links.length;
            final DecimalFormat df = numberOfPages > 999 ? new DecimalFormat("0000") : numberOfPages > 99 ? new DecimalFormat("000") : new DecimalFormat("00");
            boolean checked = false;
            for (final String singleLink : links) {
                ++i;
                if (!checked) {
                    HeadRequest head = br.createHeadRequest("**External links are only visible to Support Staff** + imghost + ".hitomi.la/galleries" + singleLink);
                    URLConnectionAdapter con = br.cloneBrowser().openRequestConnection(head);
                    try {
                        if (con.isOK() && StringUtils.containsIgnoreCase(con.getContentType(), "image")) {
                            checked = true;
                        } else {
                            con.disconnect();
                            head = br.createHeadRequest("**External links are only visible to Support Staff** + singleLink);
                            con = br.cloneBrowser().openRequestConnection(head);
                            if (con.isOK() && StringUtils.containsIgnoreCase(con.getContentType(), "image")) {
                                checked = true;
                                imghost = "0a";
                            } else {
                                throw new PluginException(LinkStatus.ERROR_PLUGIN_DEFECT);
                            }
                        }
                    } finally {
                        con.disconnect();
                    }
                }
                final DownloadLink dl = createDownloadlink("direct**External links are only visible to Support Staff** + imghost + ".hitomi.la/galleries" + singleLink);
                dl.setProperty("Referer", br.getURL());
                dl.setProperty("requestType", "GET");
                dl.setAvailable(true);
                dl.setFinalFileName(df.format(i) + getFileNameExtensionFromString(singleLink, ".jpg"));
                decryptedLinks.add(dl);
            }
        }
        if (fpName != null) {
            final FilePackage fp = FilePackage.getInstance();
            fp.setName(Encoding.htmlDecode(fpName.trim()));
            fp.addLinks(decryptedLinks);
        }
        return decryptedLinks;
    }

    /**
     * they do some javascript trickery (check reader.js,common.js). rewritten in java.
     *
     * @param guid
     * @return
     * @throws DecrypterException
     */
    private String getImageHost(final String guid) throws DecrypterException {
        // number of subdmains, var number_of_frontends = 3;
        final int i = 3;
        // guid is always present, so not sure why they have failover. That said you don't need subdomain either base domain works also!
        String g = new Regex(guid, "^\\d*(\\d)$").getMatch(0);
        if (false && "1".equals(g)) {
            g = "0";
        }
        final String subdomain = Character.toString((char) (97 + (Integer.parseInt(g) % i)));
        return subdomain;
    }
}


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #9  
Old 21.01.2020, 02:35
Etshy Etshy is offline
DSL Light User
 
Join Date: Oct 2019
Posts: 34
Default

Ok.

Actually there is no need to have the full list of thumbnails.
convert thei common.js into a JAVA class (or just functions in the plugin) seems to "find" the full url from the image hash, present in the JSON array related to the comic.

From what I found the part you didn't know is the 3 last digit of the hash.
the very last one is the first part and the 2 last before the last are the second part, after "/"
see an example of an hash and the regex : **External links are only visible to Support Staff****External links are only visible to Support Staff**
That's done in the "full_path_from_hash" function from common.js


I tried to make some quick addition (I don't know JAVA so I just added the JS function, tried to explain/convert it to JAVA as I could and added how I think you could call the added functions).

I did that quite "quickly" (lost my time to search some java things ^^") and it's late here so maybe I forgot some thinsg (I hope not though)
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Hope that can help, and still available to help if needed after that.

If pastebined the plugin isn't good, let me know, it's saved on my pastebin account so I can delete it.
Reply With Quote
  #10  
Old 21.01.2020, 04:10
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Fixed!

Thanks for helping out!

You are always welcome to help with such stuff

We have a whole collection of "challenging" tickets on hold because of js stuff.
If you find something interesting, PN me;
https://svn.jdownloader.org/issues/86377

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
EDIT

Maybe you can check the part with the host again but all in all it appears to work now ...
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 21.01.2020 at 04:23.
Reply With Quote
  #11  
Old 03.02.2020, 16:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Etshy are you still around?
You might be able to help with THIS

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 01:55.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.