#81
|
||||
|
||||
Quote:
2x from south america/argentina 1x from europe/spain 1x from south america/bogota 1x from south america/santiago 2x from south america/lima 1x from asia/india
__________________
irc.libera.chat #jDownloader web.libera.chat/#jDownloader |
#82
|
|||
|
|||
Europe/Romania blocked.
I think mediafire deploys its cloudflare protection gradually that's why for now it works for some and not for others but sooner or later it won't work for anybody. |
#83
|
|||
|
|||
USA and Spectrum here and blocked by cloudflare protection
|
#84
|
|||
|
|||
Looks like the real download link can't resloved.
Coming from a filecrypt container like this: **External links are only visible to Support Staff****External links are only visible to Support Staff** and clicking on Click'n Load, the added files could not be downloaded. Resolving the link in the browser by clicking the "Herunterladen" button at filecrypt and then adding the links via the "Download" button from mediafire, the added links will be downloaded just fine. Its a workaround, but it adds manual effort... |
#85
|
||||
|
||||
That will always work because download servers rarly/never are protected by cloudflare as the cloudflare traffic costs would explode then
__________________
JD-Dev & Server-Admin |
#86
|
||||
|
||||
Quote:
On what OS are you?
__________________
JD-Dev & Server-Admin |
#87
|
|||
|
|||
Last time I downloaded from mediafire was like two weeks ago. And they were working fine, but now they're acting up from the 1st link. I'm on Win10.
After some testing, I think I see where the issue __might be__. I've used wget (for convenience) to replicate what was written in the logs. When requesting /file why do you add as referer api/1.5/file/get_info.php...? When I've tried the exact header structure as in the logs, I too have gotten a 403 response in wget, but removing just that referer header the page downloaded properly. Also the page seems to load fine without the need of any cookies... both in wget and in the browser (Edge, completely blocked cookies for mediafire), it loads fine. Also, seems strange that Cloudflare protection fires up in JD2, but not in the browser. I do encounter plenty of sites behind cloudflare and their lil' captcha but not on mediafire when browsing... like never. Buzzheavier, sure, but not mediafire. Last edited by Emm; 07.10.2024 at 07:59. |
#88
|
||||
|
||||
Quote:
Quote:
Quote:
Please test with next update Update: that change is live
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 07.10.2024 at 17:41. |
#89
|
|||
|
|||
I don't know what VPN are you using, but I could share with you the one I 'm using,
so you try by yourself and see if that way you are able to reproduce the blocking. |
#90
|
||||
|
||||
Sure, name the VPN you are using.
Please still keep in mind that most likely we will not be able to do anything to fix this (see Cloudflare FAQ). Did any of you guys already contact the mediafire.com support and ask them what's going on? After all, we are using their APIs which are specifically ment to be used by tools like JDownloader and those are being blocked by Cloudflare: mediafire.com/developers/ This means that this problem will also affect other projects which rely on the mediafire.com API.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#91
|
|||
|
|||
I'm using "ExpressVPN".
This is not a big deal anyway, but there should be a way of discovering the mystery behind how they are blocking specifically the "JD2" tool for some users, while leaving others still use it normally. By the way, I just tried with "MiPony" download tool, links are resolved normally and files get downloaded correctly. So, this kind of confirms "Cloudflare" is blocking specifically the "JD2" tool only, at least for some users. Mediafire support team was already informed of the issue; still waiting for an answer from them. |
#92
|
||||
|
||||
@ElCho: have you installed latest JDownloader updates that adressed the findings by https://board.jdownloader.org/showpo...7&postcount=87 ?
@ElCho: Can you provide endpoints/geo locations that fail for you? and does it work for you with vpn or not?
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 07.10.2024 at 23:31. |
#93
|
|||
|
|||
Yes, I already tried with the "48254" update, and still the same.
Respect your second question: https://board.jdownloader.org/showpo...6&postcount=72 It seems they are detecting "JD2" use, besides some other unknown parameter, and the proceeding to block the access, disregarding the IP locations and cookies. When I have some time, I'm gonna try with different workstations, another OS and even a different ISP, to see what happens. |
#94
|
|||
|
|||
Quote:
It could very well be finger printing, but that also means it's bypassable. If nothing shows up in the browser, then nothing should show up in JD2's internal calls. Unless it's something deeper... I know the browser uses HTTP/3, Edge shows that, while JD2 uses v1.1, that means it's not a cloudflare issue that can't be solved, but it does point to fingerprinting. It also means a host/cdn can simply "disable" JD2's use if they don't like the excess traffic it causes, in mass and erratically so you can't pinpoint the issue. At least with buzz it's a captcha and front check, but here it's a sleazy block. Is it possible to upgrade JD2 http engine to use v3 and a newer browser agent? Tested with update 48254, still gets blocked. Quote:
But where does JD2 use the API do get the content link? From what I can see in the logs, it doesn't. The only API call is to get metadata and that one works without a problem, it outputs a pretty json in the logs every time, hence it shows up nicely until the download has to start. To start the download, JD2 accesses the same page we all do in the browser, mediafire.com/file/MediaID/file ... with or without the file name parameter. I mean, I suppose it follows what the API call returns but that's the same link that was added to JD2 in the first place... links normal_download has as value same initial link, not a direct download / content link. The content link obtained manually from the page works in JD2. I made myself a Tampermonkey script that gets those direct links from problematic hosts like buzz and now mediafire, it's just cumbersome to use when the download is split in 30 links, since JD2 has a clipboard scanning lag, I have to open one at a time every 3-4 seconds. But it works, so content links get accessed and downloaded with JD2 directly without a problem. The cloudflare problem is the page. I don't use VPN, don't have too, so I got to try VPNjantit for OpenVPN. I've tired Canada Quebec, Germany Berlin and Germany Frankfurt. As a free user don't get a lot of access: Canada is blocked, Saint Jean Quebec/OVH SAS Germany is blocked, Berlin/1&1 Internet AG Germany is blocked, Frankfurt/OVH SAS Otherwise, in all vpn's the content link works in JD2, and browser page loads without any issue. Normal links don't work. Last edited by Emm; 08.10.2024 at 04:48. |
#95
|
||||
|
||||
Quote:
Cloudflare or mediafire.com? Quote:
It has nothing to do with what you would call a "browser engine". In the Cloudflare thread, some open source solutions are linked which basically consist of setting up a special proxy which does some special requests using a "real browser" in order to get through Cloudflare. Search in this thread: https://board.jdownloader.org/showthread.php?t=83712 The JDownloader project is open source so feel free to do local modifications and test them. Source code: https://support.jdownloader.org/know...up-ide-eclipse Everything except free downloads is done via API in JD2. Quote:
After all I still can't reproduce it so I could only make assumptions. Neither do I. I've done some tests with Mullvad VPN (only german IPs) and none were blocked.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#96
|
|||
|
|||
Mediafire.
By the way, they sent me what, at first, I thought it was a "regular" reply from a representative, which has a series of steps I should follow, in order to help them to figure out which could be the problem and the possible solutions. But after reading all the instructions, they tell me I could try with another download manager, at least temporarily, but, to my surprise, they mention a nowadays "abandonware" program, whose not even the developer's website is alive anymore: Speedbit's "Download Accelerator Plus". That little detail suggests me they just delivered me a generic reply, whose template isn't even updated (last DAP's version dates from 2013). Alright, after reading here and there about this problem, I finally was able to make "JD2" works as expected again in my system. Basically, someone mentioned he found a "portableapps" version which was working correctly with "Mediafire", I tested it and it was true. I then thought about what could be the reason, and I found it: JAVA JRE. The thing is, that specific portable was using Oracle's JRE "v1.8.0_202 (64bit/X86)"; while my current (non-working) installation has Oracle's last version: "1.8.0_421 (64bit/X86)". So, I downloaded "202" version from Oracle's website, replaced it from my installation, and that was the solution. I also tried with the JRE v21.0.2 (64 bit) from "Eclipse Adoptium" company, and it was working rightly wit hit too, but I don't like that JRE's version, since it alters the JD2 interface I'm used to. After testing the "202" build, I also tested going back to the original version I had installed "421", and the downloads kept working. So I will keep using it, provided it remains working fine. So, the mystery is finally revealed, well at least partially, since it remains to know why the Oracle's JRE "421" build ceased to work all of a sudden. And why doing the "421>202>421" exchange makes the last Oracle's JRE build works again. |
#97
|
||||
|
||||
@ElCho: To make it short but you can dig into the web for more detailed articles. Cloudflare does deep analyze the hole request (UA, headers (size, number, order, content) and https settings (protocol, offered ciphers, selected cipher) and many many more) and does fingerprint those. Now they have a good *sight* at fingerprints and can easily tell that browser A has 99% fingerprint A but 1% fingerprint B, so requests from B are *strange* and might (depending on cloudflare site settings) require javascript challenge that now checks the browser environment and fingerprints that as well. and when that doesn't help either to clearify internal stats, then turnstile will be requested too. That's why a simple change of UA might help, or changing https settings. Now with different java version, the default protocol might have changed and/or the default offered/selected cipher and many more.
I tried some stuff and was able to get blocked by cloudflare as well but it was totally random and could not find a *muster*. working fine for several minutes, blocked again...changing java...blocked...working fine again....changing java...doesn't help..changing ciphers...doesn't help.....working fine again....blocked again... To me it looks like they might be under attack and it's more random (but also depending on IP range) if cloudflare is protecting or not Short: enjoy while it lasts but doesn't mean that it has to last long
__________________
JD-Dev & Server-Admin |
#98
|
|||
|
|||
Build 421 ceased to work again after a while, so I switched back to build "202", let's see how much it lasts.
|
#99
|
|||
|
|||
At this point I'm pretty sure it's Cloudflare, after all that's what they're getting paid for, protection, so that mediafire can be lazy. And it pretty much points to fingerprinting, most likely of communication initiation in TLS.
Quote:
I've tried build 202 of JRE as well and yes, it works. The only difference in the logs are the SSL/TSL handshakes, everything else is the same. Build 202 uses SSLCipher: AutoSwitch|JVM|Protocol:TLSv1.2|CipherSuite:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA Build 322 (the one I've had in JD2's folder since installation) uses SSLCipher: JVM|Protocol:TLSv1.2|CipherSuite:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 Wget doesn't have ssl/tls debug capability so I can only assume it negotiates something Cloudflare likes or doesn't find suspicious enough to get triggered. However, Curl, which can show headers, fails just like JD2 with 322 does. SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305 HTTP/2 403 That's as much as I can see from the logs and debug output... probably there's more to it, but yes, fingerprinting it is. Which might be caused by JD2's popularity and use on certain Cloudflare nodes, or IDK. After searching a bit, I ended up at a post from two years ago stating exactly that, TLS fingerprinting. where-does-cloudflare-detect-web-and-terminal-requests-on-equal-terms **External links are only visible to Support Staff****External links are only visible to Support Staff** Searching further, seems curl is Persona non grata for Cloudflare. Ended up at curl-impersonate. **External links are only visible to Support Staff**curl-impersonate Which looks like a mixed bag, on its own it gets blocked but with with JD2's header structure (without cookies) seems to work just like wget or build 202, i.e. fine. You gotta love this answer... No, this type of issues looks more like a youtube vs adblockers back and forth fight. Have fun. Last edited by Emm; 11.10.2024 at 09:35. |
#100
|
||||
|
||||
Quote:
In the end it is still the website owners (mediafire) who configure Cloudflare. I know I'm repeating myself but has anyone asked mediafire.com about the situation yet? Quote:
We've already stated that we are not participating in any "Cloudflare vs bots war", see: https://board.jdownloader.org/showthread.php?t=83712 Read the whole FAQ. We are not planning to put any time into this. Use the tools that are already available and linked there and or dig deeper into existing github "Cloudflare proxy" projects. Cloudflare has been around for a long time and if ppl really wanted it to block tools like JD, they've had their chance all of the time. You can see this in numerous threads in our forums, just search for "Cloudflare" in titles-only. I'm sure the people who are working on Cf solutions and see it as a challenge will have fun doing so.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 11.10.2024 at 13:46. Reason: Fixed typo |
#101
|
||||
|
||||
Quote:
I think it's more likely that sites will stop using them as soon as us laws request cloudflare to hold/verify personal details about site owners
__________________
JD-Dev & Server-Admin |
#102
|
|||
|
|||
don't know exactly how Jdownloader download the files, but using browser is working fine, incognito working fine, just with JD i get blocked by cloudflare...don't even download 1 file... and just message "blocked by cloudflare" just wondering if JD can simulate the download just like the browser do... also could JD simulate incognito mode like the browser do it ??
i haven't change my IP and is not needed because with browser and incognito is working, so i should say JD needs and could be improved to bypass these "checks" Last edited by Rickz; 14.10.2024 at 06:40. |
#103
|
||||
|
||||
@Rickz: read https://board.jdownloader.org/showpo...4&postcount=97
Quote:
Quote:
__________________
JD-Dev & Server-Admin |
#104
|
|||
|
|||
@Jiaz if mipony can i'm sure JD can, JD BEST download manager so far!!
there are some work around that i believe can be automated.. check: **External links are only visible to Support Staff****External links are only visible to Support Staff** Last edited by Rickz; 15.10.2024 at 05:04. |
#105
|
||||
|
||||
@Rickz
Read the Cloudflare FAQ again closely! Short version: By copying the direct URLs from your browser, your browser already did all of the Cloudflare verify java script stuff before (see also the Cloudflare cookies that are present). This way you effectively download manually (or in a "semi automatic" way). About "Mipony": I've also read on other places that this is supposed to work. Did you test that? I cannot test it as I'am not blocked by Cloudflare so JD is still working for MF for me.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#106
|
|||
|
|||
Quote:
i only get blocked by cloudflare with JD, same IP and mipony is working... |
#107
|
||||
|
||||
Quote:
We can try to work on this but our main problem is that we have hard time to get blocked at all and thus cannot work on the problem when not getting blocked.
__________________
JD-Dev & Server-Admin |
#108
|
|||
|
|||
I just tested it with recent 431 JRE's build, and JD2 is still blocked with it; went back to the still working fine "202" build.
"MyPony" works correctly, but maybe because it (still) wasn't fingerprinted. Since you aren't being blocked, do you think a portable version of mine, among the same VPN and Windows' VM would be enough to get it triggered? |
#109
|
|||
|
|||
Quote:
|
#110
|
|||
|
|||
I already tried with different OSES and still the same; could try with a different ISP, workstations, etc., but have not the time right now. Anyway, as long as it keeps working, I will stay with build "202".
|
#111
|
|||
|
|||
Where do I get build "202"?
|
#112
|
|||
|
|||
https://www.oracle.com/java/technolo...downloads.html
https(://)sites.google.com/view/java-se-download-url-converter I use this package: jre-8u202-windows-x64.tar.gz Last edited by ElCho; 25.10.2024 at 06:37. |
#113
|
|||
|
|||
Thanks, that worked for me also.
|
Thread Tools | |
Display Modes | |
|
|