#1
|
|||
|
|||
Exhentai/E-hentai 5000 images ghost limit
So I've looked a bit into this issue in the forums and it seems to be somewhat persistent and has already been reported multiple times. However I decided to make a new thread to avoid potentially sending notifications to other users who might be still be subscribed to that thread just to be good mannered and not necro the thread again.
To keep things short, it seems that the plugin still works fine with e-hentai links, however it runs into the ghost image limit issue whenever an exhentai link is started after it reaches the default 5000 limit (i have a 25000 limit on my account and JD recognizes that just fine). I also ran a couple of tests by stopping the downloads and switching the package order around (exhentai pack first followed by e-hentai package and viceversa) to confirm that it was just the exhentai package that didn't want to cooperate. here's the log 06.12.22 22.11.36 <--> 06.12.22 22.25.13 jdlog://8361311370661/ I'm willing to share my account details if necessary again (think I've done so in the past) also not related to the limit but still relevant to the plugin, it seems like the "image size too small" error pops up randomly on some downloads for some reason even though the error doesn't really apply to the image that it reports as such. currently i've only managed to reproduce that behavior while downloading and connected through a vpn (proton), but the images seem to download just fine when not connected via vpn. the behavior should also be reflected in that log also |
#2
|
||||
|
||||
Quote:
__________________
JD-Dev & Server-Admin |
#3
|
||||
|
||||
The server doesn't report content lenght and thus the *not a picture* detection fails to handle it. We have to check if this is new sort of *ghost image* or just missing header and workaround required.
__________________
JD-Dev & Server-Admin |
#4
|
||||
|
||||
Maybe the limit also somehow connected to IP or it's explicit VPN blocking/rate limit try? Maybe with VPN connection, your account won't be used /server side and it's acting like a normal(no/limited account) user? What happens when it starts to happen in JDownloader when you test in browser with same IP?
__________________
JD-Dev & Server-Admin |
#5
|
|||
|
|||
Quote:
Quote:
Quote:
either way i'd like to clarify that the VPN/"image too small" is a separate/minor issue that i just felt like pointing out as well. the main issue at hand is the 5000 limit bug that doesn't seem to be completely ironed out yet, and sorry if made it sound too complicated by mentioning that minor one as well, since they're both unrelated anyways. |
#6
|
||||
|
||||
Quote:
Context: https://board.jdownloader.org/showth...456#post477456 I've never found a solution for this issue although I always said "it can't be that hard". It is still to this day unclear, whether the problem happened due to serverside detection/block/limit of e(x)hentai or is a "bug/small oversight in our plugin" though I'm voting for the latter. Quote:
Quote:
My guess is that it fails due to a small oversight of us like one missing request or a request which is currently done differently in our plugin than in browser resulting in that problem. To work on this, we will definitely need your ehentai login credentials. Please send them to: support@jdownloader.org and please don't have high hopes for this.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 08.12.2022 at 01:27. Reason: Fixed spelling mistakes |
#7
|
|||
|
|||
Quote:
Still, appreciate the support given either way, at the least it has cleared a bunch of uncertainties I had. I've also just sent my account details. |
#8
|
||||
|
||||
@serveral: I can try to dig deeper and maybe be able to find the cause. Am I allowed to use account to use up 5k and more so I try to reach the limit in JDownloader?
__________________
JD-Dev & Server-Admin |
#9
|
|||
|
|||
Quote:
Edit: I just checked my mail. The easiest way to reach the limit would be by checking the "prefer original quality" option in the e-hentai plugin's settings, and then just grab a random gallery with a large amount of pictures (1k plus pictures should reach the 5k limit quite easily) Last edited by serveral; 08.12.2022 at 17:22. Reason: Additional info |
#10
|
||||
|
||||
@serveral: just to make sure, does visiting the /home.php do any reset or is there some sort of button to click? I'm not familiar with that site
__________________
JD-Dev & Server-Admin |
#11
|
|||
|
|||
Quote:
The image limits are IP based either way, so a simple IP change should reset the limit too. |
#12
|
||||
|
||||
Quote:
__________________
JD-Dev & Server-Admin |
#13
|
||||
|
||||
@serveral: just to make sure to look for the right issue
JDownloader fails after about 5000 images? whereas the browser works fine? also check the "image too small" issue, right?
__________________
JD-Dev & Server-Admin |
#14
|
|||
|
|||
Quote:
The "image too small" issue might be a little harder to reproduce since I only got it to happen when downloading via VPN (proton i was using then) so I can't really provide anymore info beyond that. Could just be a simple connection issue on my end either way. |
#15
|
||||
|
||||
Quote:
Just reactivate your VPN, reproduce the error and upload a fresh log for us so in the best case we can fix it without needing to be able to reproduce it locally. No.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#16
|
|||
|
|||
08.12.22 12.22.42 <--> 08.12.22 12.25.15 jdlog://1761311370661/ here, I managed to reproduce the small file error in this log. The file that JD flagged is shown to be "316.20 KiB" of size, which shouldn't trigger that error from my experience.
|
#17
|
||||
|
||||
This is the http response of the image download attempt:
Code:
----------------Response------------------------ HTTP/1.1 200 OK Server: Genetic Lifeform and Distributed Open Server 1.6.1 Date: Thu, 08 Dec 2022 16:24:33 GMT Content-Type: image/jpeg Cache-Control: public, max-age=31536000 Transfer-Encoding: chunked Connection: close ------------------------------------------------
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#18
|
|||
|
|||
If I recall correctly (I've deleted the package already) it should be this picture **External links are only visible to Support Staff****External links are only visible to Support Staff** (or it's e-hentai mirror here **External links are only visible to Support Staff****External links are only visible to Support Staff**).
|
#19
|
||||
|
||||
I know. Finding that in your log was not a problem.
We'll look into it. Thanks for the log.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#20
|
|||
|
|||
Just wanted to mention that the "image too small error" doesn't seem to be VPN dependent as I had originally thought since I just replicated it when I had it turned off. I did manage to download the image in the end by changing my IP through with the VPN, and then, just as a test, I deleted the image with the reset option in JD, disconnected from the VPN and then tried downloading it again and it went just fine...
10.12.22 22.36.39 <--> 11.12.22 01.01.11 jdlog://9271311370661/ here's the log for that session. I do want to add something else to this issue though, which is that I've had something somewhat similar happen in the past, where a few images at random intervals just failed to download and just gave a "host offline" error and remained like that for some good 15 to 30 minutes unless I changed my IP with a VPN or went into my router's settings to get a new IP (dynamic). I always blamed it on my ISP, as I had packet loss issues on my ADSL line for a good while then (due to poor maintenance and crappy infrastructure where I live), and I sort of guessed that it should've been that, since I also had issues while browsing galleries through the browser where pictures just wouldn't load sometimes (unless I clicked on the "click here if image fails loading" button they have there), which mirrored the issues on JD with those particular pictures that wouldn't download. However I've started thinking it might actually just be an issue related to their hentai@home service failing to provide the requested images due to it's reliance on third party users to help out with their bandwidth issues. That's just a theory though, as the issue is now somewhat different and the behavior isn't really mirrored anymore between the browser and JD, though I guess that could be just a user agent/cookie thing too. Last edited by serveral; 11.12.2022 at 07:37. Reason: syntax |
#21
|
||||
|
||||
Our ehentai can't really automatically use that alternative/fallback image at this moment though that shouldn't matter as it at least shouldn't ever download corrupt/dummy images.
Regarding the "image too small" problem: The changes we did haven't been released yet. Bitte auf das nächste CORE-Update warten! Please wait for the next CORE-Update! Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#22
|
|||
|
|||
Has there been any progress on this ghost limit issue?
|
#23
|
||||
|
||||
No.
EDIT I've changed the thread title back to "See Bugtracker" as no one is actively working on this. We're open source so anyone is free to investigate this further, see: https://support.jdownloader.org/Know...up-ide-eclipse
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#24
|
||||
|
||||
If someone is willing to provide an account for testing, I'm willing to give this one more shot.
Send logins to: psp@jdownloader.org
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#25
|
|||
|
|||
I've just sent the account details, however, there have been some changes to their site since this issue was originally posted that may impact some of the tests. I'm going to quote some of relevant details from **External links are only visible to Support Staff**this post in the site's forum.
First off, they introduced a "peak hours" system that completely ignores the "image limits" system during those specified hours, instead using one of their in-site currencies called "GP" during those hours to download the original images from a gallery (which, in my experience, is the fastest way to reach the image limit quota). Quote:
Quote:
To summarize: whoever is going to do the testing, just keep in mind not do it during the peak hours periods. And to also only use newer galleries for the tests, since galleries that are more than 1 year old only use that GP download system as well. |
#26
|
||||
|
||||
Not sure how I could test it now as your account only has the default/5000 limit now.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#27
|
|||
|
|||
That's odd. My account settings show that I have a 50k limit
|
#28
|
||||
|
||||
I see.
I got that old "bug" in browser where it is 5000 until you visit "/hathperks.php" once.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#29
|
|||
|
|||
I don't know how helpful this might be but this **External links are only visible to Support Staff**script and gallery-dl's exhentai **External links are only visible to Support Staff**extractor have workarounds to guess an user's account actual image limits.
I'm aware they're both written in different languages to JDownloader's, and that they may also include libraries or dependencies that aren't available in JD's code base, but still thought they may provide some insight in solving this issue. |
#30
|
||||
|
||||
I've now used more than 10 000 GP and I can still download via JD.
I'm stopping now so I'm not wasting all of your credits. Could it be that this issue fixed itself? Also I've implemented some measurements to try to avoid wasting GP when downloading original images though it's probably something which only people with a bad internet connection will notice (re-use directurls on resume).
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#31
|
|||
|
|||
I'll try downloading some galleries and get back to you in a bit.
|
#32
|
|||
|
|||
Can confirm that the issue is no longer present. Gone past the 5k image limit mark and downloads are not being stopped.
Edit: Nevermind, I seem to have hit the limit at 10k now, here's the log 14.08.23 15.56.26 <--> 14.08.23 17.40.31 jdlog://7907311370661/ Last edited by serveral; 15.08.2023 at 00:51. |
#33
|
||||
|
||||
Your log looks good:
Code:
[jd.plugins.hoster.EHentaiOrg(getNormalmageDownloadurl)] -> Failed to get around limit - limit is definitely reached! --ID:2750TS:1692048559736-8/14/23 5:29:19 PM - [] -> Exception thrown at jd.controlling.downloadcontroller.SingleDownloadController.download(SingleDownloadController.java:441): jd.plugins.AccountUnavailableException: Downloadlimit reached->ERROR_PREMIUM|Value:0 Once that time is over, JD will check it again. If 0 GP is available, the account will stay active but with 0 bytes of traffic-left. --> Accounts with zero traffic won't be used for downloading. If more than 0 GP is available again, the account will be used again until one download is running into that limit-message again. While the account can't be used, JD will continue in free mode which will allow the download of non-original images. If you wish to avoid this completely, you might want to create an account usage rule for e-hentai.org and disable free mode there so whenever account downloads for original images are not possible, all items will get a wait status until the account can be used again. EDIT I've updated the plugin so all GP related limit error-messages will temporary deactivate your account (yellow status). I think this is the way to go. EDIT2 If you can't reproduce the "Ghost limit bug", I will close all related tickets.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 15.08.2023 at 13:24. |
#34
|
|||
|
|||
The peak hours period is in place as of right now so I'll get back to you in some hours after it's lifted. I'll update this post to inform you whether I can still replicate the ghost limit or not.
Edit: Seems like an IP ban (white page) was triggered a couple of minutes after I began my test. Log: 15.08.23 16.22.52 <--> 15.08.23 16.44.40 jdlog://2117311370661/ my guess is that their system noticed some sort of bot-like behavior coming from newly implemented GP/credit checks. Last edited by serveral; 15.08.2023 at 23:50. |
#35
|
||||
|
||||
White page where?
Single image pages, account overview or even main page of e-hentai? Also, what am I supposed to do then? Quote:
No: 1. No additional checks have been implemented. 2. Upon refactoring the plugin, I even removed some unnecessary checks so the plugin is doing less http requests. EDIT So if you want me to somehow detect this block - yes I could try to do this. Your log contains a lot of download activity, also of other filehosts. Please provide a fresh log - in the ideal case it is only from one session and contains only e-hentai activity. Keep in mind that if a website really wants to block public tools like JDownloader, they can do this at any point of time.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 16.08.2023 at 12:27. |
#36
|
|||
|
|||
Quote:
This attempt to circumvent their bandwidth usage limit is making things way worse. Please don't mess with GP conversions. Last edited by simon_7617118; 16.08.2023 at 16:53. |
#37
|
||||
|
||||
@simon_7617118
To make this absolutely clear:
Quote:
If you mean the log-quote I posted in post #33: That was a badly worded log output in a function which simply checked if the limit was really reached after the first time it looked like itwas reached. You can verify that information by looking into our source code: Code reference: github.com/mirror/jdownloader/blob/5f50ef81a535c64d31713ed043587eac7a3da057/src/jd/plugins/hoster/EHentaiOrg.java#L493C14-L493C14 In the meanwhile the wording of that log-output was changed as well.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 16.08.2023 at 17:18. Reason: Fixed typo and updated wording |
#38
|
|||
|
|||
So is this change coming from EH itself? Because right now I can't download any image at original resolution, even though my image points usage is at 0. It's counting towards GP, and I've run out quickly because I don't play that game.
Sorry if that's the case. I'm not a coder, I can't review anything. Is there any chance that a recent change in the plugin may have triggered this behavior? It has never used GP before, and I can't imagine how EH is able to discern traffic from JD. Previously, the only way they could detect it was if the simultaneous download was set too high. |
#39
|
|||
|
|||
Quote:
Quote:
Quote:
Edit No bans triggered so far when downloading resized pictures (original quality setting unchecked) but I seem to have hit the image limit near 5k. Kinda odd given the last time I ran into the limit was at 10k instead (the one I reported in one of the posts above), but that was with the original quality settings ticked. Here's the log 16.08.23 12.24.44 <--> 16.08.23 14.40.02 jdlog://9217311370661/ There are probably some errors/download failures scattered across the whole log but those are most likely local network issues, the important stuff should be near the end of the log session. Edit 2 Seems like the white page/ban thing was a one time only thing, maybe something I did on my end but so far I haven't managed to replicate it. However I did try once more to download some more galleries, this time with the original image quality setting ticked on to see if the image limit bug was still there, and this time I got to around 8k before JD started saying that I had reached the limit. Log of this session: 16.08.23 20.31.43 <--> 16.08.23 20.42.52 jdlog://4317311370661/ . This last log is not as clean as the previous one because my system crashed, (unrelated to JD) but prior to the crash I had already passed the 5k image limit mark and downloads kept going (until the crash of course). After the crash, I was able to download about a dozen more pictures before it finally stopped at about a little over 8k. Last edited by serveral; 17.08.2023 at 04:21. Reason: Update |
#40
|
|||
|
|||
Okay, that must be it, although I've only started running into this problem this week. Thanks for the explanation.
|
Thread Tools | |
Display Modes | |
|
|