JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 18.12.2021, 02:10
damo damo is offline
DSL Light User
 
Join Date: Jan 2020
Posts: 34
Exclamation Hitomi.la plugin broken

Jdownloader reports plugin defekt for hitomi.la galleries since yesterday.

Last edited by damo; 01.01.2022 at 13:26.
Reply With Quote
  #2  
Old 21.12.2021, 14:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

wait for next plugin update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 01.01.2022, 03:21
damo damo is offline
DSL Light User
 
Join Date: Jan 2020
Posts: 34
Default

Unfortunately, it is broken again.
Reply With Quote
  #4  
Old 04.01.2022, 12:11
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Either cat/mouse or random change. Wait for next plugin update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 07.01.2022, 15:34
Etshy Etshy is offline
DSL Light User
 
Join Date: Oct 2019
Posts: 34
Default

Not here to complain, just to say I can confirm issue since today for me (last time I downloaded from hitomi was about one week ago)
As this thread is marked as Solved, should I create a new one ?
Reply With Quote
  #6  
Old 07.01.2022, 15:57
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@Etshy: wait for another update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 07.01.2022, 16:07
Etshy Etshy is offline
DSL Light User
 
Join Date: Oct 2019
Posts: 34
Default

Sure
Reply With Quote
  #8  
Old 28.01.2022, 17:28
damo damo is offline
DSL Light User
 
Join Date: Jan 2020
Posts: 34
Default

Broken again :(
Reply With Quote
  #9  
Old 10.02.2022, 17:10
Cesra250 Cesra250 is offline
Modem User
 
Join Date: Feb 2022
Posts: 1
Default

Plugin still broken.

Last edited by Cesra250; 11.02.2022 at 01:08.
Reply With Quote
  #10  
Old 15.02.2022, 14:57
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

See the prefix of this thread.
You're also free to fix it yourself - we're open source.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #11  
Old 14.04.2022, 11:12
Etshy Etshy is offline
DSL Light User
 
Join Date: Oct 2019
Posts: 34
Default

Hitomi stopped working since a few days (I didn't try to download for a few days and today every link are "offline" instead of detecting the pages.).

I'll try to see why, but I don't know sh*t about java, so I won't fix it, just try to help find the problem.

EDIT :

Here are my notes from quick look

ProgressController argument used is deprecated (just saying, I don't thing that's the problem)

I guess the "Offline Files" in Hitomi plugin comes from these lines
Code:
decryptedLinks.add(createOfflinelink(parameter));
return decryptedLinks;
If that's the case, one possible reason could be these lines
Code:
final String js = br.getRegex("src\\s*=\\s*\"([^\"]+" + gallery_id + "\\.js)\"").getMatch(0);
if (js == null) {
     logger.info("Seems like this is no downloadable/supported content");
     decryptedLinks.add(createOfflinelink(parameter));
     return decryptedLinks;
}
Not sure but I guess this code will search for a html tag with the "*gallery_id*.js" as a src attribute ?
From what I see in galleries and reader page this script tag doesn't exist.
On the Reader page, they load "*gallerie_id*.js" from jquery at this url :
Code:
**External links are only visible to Support Staff**
the "ltn" part is fixed for prod server
Code:
var domain = (/^dev\./.test(document.location.hostname.toString()) ? 'dev' : 'ltn')+'.hitomi.la';
and the rest of the URL too
Code:
var result = window.location.href.match(/\/reader\/([0-9]+)\.html/);
var galleryid = result[1];
$.getScript('//'+domain+'/galleries/'+galleryid+'.js' ...

example what I mean with "reader page" : **External links are only visible to Support Staff****External links are only visible to Support Staff**
example what I mean with "galleries page" : **External links are only visible to Support Staff****External links are only visible to Support Staff**

There are also some changes in method "copied" from their js

Code:
private String url_from_url_from_hash(String galleryid, Map<String, String> image, String dir, String ext, String base) throws Exception {
    //NEW (JS)
    if ('tn' === base) {
        return url_from_url('**External links are only visible to Support Staff**+dir+'/'+real_full_path_from_hash(image.hash)+'.'+ext, base);
    }
    //ENDNEW
    return url_from_url(url_from_hash(galleryid, image, dir, ext), base);
}
New method "real_full_path_from_hash"
Code:
function real_full_path_from_hash(hash) {
        return hash.replace(/^.*(..)(.)$/, '$2/$1/'+hash);
}
Code:
private String url_from_hash(String galleryid, Map<String, String> image, String dir, String ext) throws Exception {
	//CHANGE JS - not sure the "split" part is equivalent, guess the java part shoud be just "split(".")" too ?
	ext = ext || dir || image.name.split('.').pop();
	//CHANGE JAVA (existing)
    ext = isNotBlank(ext) ? ext : (isNotBlank(dir) ? dir : last(image.get("name").split("\\.")));
    //ENDCHANGE
    dir = isNotBlank(dir) ? dir : "images";
    return "**External links are only visible to Support Staff** + dir + '/' + full_path_from_hash(image.get("hash")) + '.' + ext;
}
The whole subdomain_from_url function is too different from the JS one in hitomi sources, so I can't say for sure there is no changes

Also it seems the "base" variable is always "a" and "dir" variable is either "webp" or "avif" now, but seems "webp" will always work.
(from what I can see in JDownloader sources, it was null before)
(see in reader.js, search for "url_from_url_from_hash(", the only calls without "a" as base variable



That was a really quick look though, I hope I wasn't too confusing and this will help.

Last edited by Etshy; 14.04.2022 at 12:42.
Reply With Quote
  #12  
Old 14.04.2022, 13:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@Etshy:They have changed again, wait for next plugin update. Can't promise jpeg support as they move to webp and avif image format.
Please check again with next update. I'm sorry but they stopped jpg and only have avif and webp. in case browser doesn't support webp, they use javascript webp decoder
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 14.04.2022 at 13:29.
Reply With Quote
  #13  
Old 14.04.2022, 14:16
Etshy Etshy is offline
DSL Light User
 
Join Date: Oct 2019
Posts: 34
Default

Yeah we already saw that they stopped jpeg support a few months ago and was transitionning to webp.
from their JS I guess they completely change from jpeg to webp as almost everything seems to use webp.

Thanks a lot for the quick update
Reply With Quote
  #14  
Old 14.04.2022, 14:23
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Quote:
Originally Posted by Etshy View Post
Thanks a lot for the quick update
Wait for update and thank me once it's working for you I've updated plugin to prefer webp over avif, okay?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 14.04.2022, 14:57
Etshy Etshy is offline
DSL Light User
 
Join Date: Oct 2019
Posts: 34
Default

Okay I'll wait to really thank then ^^
For me webp is fine, no problem.
Reply With Quote
  #16  
Old 14.04.2022, 17:07
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@Etshy: Update will be available in about 5 mins
__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 14.04.2022, 17:46
Etshy Etshy is offline
DSL Light User
 
Join Date: Oct 2019
Posts: 34
Default

Seems they changed something else about the url generation
All files appears as not Found and have 403 when i go to them
https://i.imgur.com/JeBkhY5.png
Example url generated by jdownloader : **External links are only visible to Support Staff****External links are only visible to Support Staff**

Even on their reader page ( **External links are only visible to Support Staff****External links are only visible to Support Staff** ) the webp seems to be in 403, so that's a bit weird (their json for this manga show this manga have webp though.)
**External links are only visible to Support Staff****External links are only visible to Support Staff**

The subdomain seems to be wrongly guessed too

I'll make some more research tonight.

Edit :

webp url from Jdownloader (not working) : **External links are only visible to Support Staff****External links are only visible to Support Staff**
webp url from hitomi donwloader : **External links are only visible to Support Staff****External links are only visible to Support Staff**


I found something about the "gg" variable, a part of the process that find the subdomain and first path part.
there is a file "gg.js" that doesn't seem to be loaded in Jdownloader.
It's just an object with with 3 properties/methods (it should be simple to remake them I think)

Some process about finding a number and then transforming this number into a letter to generate the subdomain

That should fix the problem with the subdomain

And then there is another problem with the first part of the path (after the '/webp')

in js
Code:
function full_path_from_hash(hash) {
        return gg.b+gg.s(hash)+'/'+hash;
}

//gg methods
s: function(h) { var m = /(..)(.)$/.exec(h); return parseInt(m[2]+m[1], 16).toString(10); },
b: '1649948402/'
both errors seems to be related to that "gg.js".
I guess this is not loaded, or it's loading some old script from their server
The actual gg.js is here : **External links are only visible to Support Staff****External links are only visible to Support Staff**

I guess the "engine" in HitomiPlugin is executing the js from the page it loaded ?
so just loading the correct gg.js should be enough I think.

Last edited by Etshy; 14.04.2022 at 18:42. Reason: More information
Reply With Quote
  #18  
Old 14.04.2022, 20:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@Etshy: I will have to check again. My example links did work fine. I'm sure something I just have overseen
__________________
JD-Dev & Server-Admin
Reply With Quote
  #19  
Old 14.04.2022, 20:11
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Quote:
Originally Posted by Etshy View Post
I guess the "engine" in HitomiPlugin is executing the js from the page it loaded ?
so just loading the correct gg.js should be enough I think.
Plugin is already using the same one
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 14.04.2022, 22:26
Etshy Etshy is offline
DSL Light User
 
Join Date: Oct 2019
Posts: 34
Default

I really don't know what's wrong with the URL generation whthin JDownloader.
But if you can exec the js script, why not call their "url_from_url_from_hash" directly ?

Code:
if (haswebp == 1) {
                    url = url_from_url_from_hash(gallery_id, picInfo, "webp", null, "a");
                } else if (hasavif == 1) {
                    url = url_from_url_from_hash(gallery_id, picInfo, "avif", null, "a");
                } else {
                    url = url_from_url_from_hash(gallery_id, picInfo, null, null, null);
                }
Here you could just exec their function, no ?

Not sure if the following is good I wrote that from example in the source

Code:
//create a variable containing the object image in js
engine.eval("var image=JSON.parse(" + JavaObjectToJSON(picInfo)+ ");");
if (haswebp == 1) {
    engine.eval("var url = url_from_url_from_hash(" + gallery_id + ",image, 'webp', null, 'a');");
} else if (hasavif == 1) {
    engine.eval("var url = url_from_url_from_hash(" + gallery_id + ",image, 'avif', null, 'a');");
} else {
    engine.eval("var url = url_from_url_from_hash(" + gallery_id + ",image);");
}

final String result = engine.get("url").toString();
Honestly I have no idea if it would work. But if the js is loaded in the "engine" it should work I guess.
Reply With Quote
  #21  
Old 16.04.2022, 14:23
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@Etshy: check again with next plugin update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #22  
Old 16.04.2022, 15:19
Etshy Etshy is offline
DSL Light User
 
Join Date: Oct 2019
Posts: 34
Default

It seems to be working now.
I'll try with more links

Thanks a lot.
Reply With Quote
  #23  
Old 16.04.2022, 18:49
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@Etshy: Thanks for the feedback and you're welcome!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #24  
Old 16.04.2022, 22:31
Etshy Etshy is offline
DSL Light User
 
Join Date: Oct 2019
Posts: 34
Default

after trying a lot of links, I git a few "File not found" issue.
but only a few pages per manga, not the whole chapter.
I guess there is still somethong wrong, I'll keep the manga link that have issues and search why some pages doesn't have the good url

Edit : I'm starting to think they have some kind of hidden/backend security.
I re-added manga that had errors, and now it works fine (some "temporary unavailable" show but retrying a few times, it work)

Got some weird page names too : https://i.imgur.com/GkaKRZn.png

Werid names come after a "Temporary unavailable". it loose the page number somehow.

Edit : found on HDoujinDownloader that Hitomi started rate-limiting sinces a few months.
That should be the cause of the "Temporary unavailable" that occasionnaly lead to a "File Not Found" after some time.

Edit : Oh and another thing, the name of the package is not quite right, it always containe "manga" or "doujinshi" at the start.
If that part could be removed, it'd be great.

Last edited by Etshy; 17.04.2022 at 11:56.
Reply With Quote
  #25  
Old 18.04.2022, 14:14
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Quote:
Originally Posted by Etshy View Post
Edit : Oh and another thing, the name of the package is not quite right, it always containe "manga" or "doujinshi" at the start.
If that part could be removed, it'd be great.
will be fixed with next update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #26  
Old 18.04.2022, 14:29
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Quote:
Originally Posted by Etshy View Post
Edit : found on HDoujinDownloader that Hitomi started rate-limiting sinces a few months.
That should be the cause of the "Temporary unavailable" that occasionnaly lead to a "File Not Found" after some time.
Rate-limit in meaning of requests/time or overall number of requests? I will add rate limit this week and waiting for new feedback from you then, okay?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #27  
Old 22.04.2022, 13:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

With next plugin update, new added links will have a request limit of 2 request/sec
__________________
JD-Dev & Server-Admin
Reply With Quote
  #28  
Old 19.05.2022, 14:55
damo damo is offline
DSL Light User
 
Join Date: Jan 2020
Posts: 34
Default

We seem to be missing something else. It works for freshly imported gallery, but after a while I they start to all fail with 403.
There seems to be some time component, that doesn't affect the download url itself.
Maybe a dynamic cookie or something?

This could probably be defeated, by storing the gallery url with the package and retrieve the gallery.js if the imported timestamp is older than x-minutes.
Reply With Quote
  #29  
Old 19.05.2022, 14:57
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@damo: the links are only valid for specific time, right now you should not collect and download later.
to support this I need to add more plugin code to refresh expired links
I will check and rewrite to support refresh
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 15:18.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.