|
#1
|
|||
|
|||
Archive.org Problems
hello everyone, i have two problems with jdownloader when i try to download from archive.org. The first is that when I try to download this **External links are only visible to Support Staff****External links are only visible to Support Staff** jdownloader seems to find nothing, while the second is that I would like to include the .xml and .sqlite files automatically, from what I have seen it is possible to download them only if I directly copy the links of these files. Do you have solutions?
|
#2
|
||||
|
||||
Hi,
there was a small bug in our crawler plugin which I just fixed. It only occured for archive.org content containing a huge number of files. Bitte auf das nächste CORE-Update warten! Please wait for the next CORE-Update! Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#3
|
|||
|
|||
Thank you very much
|
#4
|
||||
|
||||
Publicly answering your private message here as my reply might also be useful for other users:
You've tested my changes in our dev build and told me that some .xml and .sqlite files are still missing. Indeed this was not a "bug" as that crawled has always skipped metadata files as most of all users do not want to download them. After some refactoring I've added another setting to our archive.org plugin which, when enabled, lets you crawl/download those files too (default = disabled). Thanks for your feedback! Bitte auf das nächste CORE-Update warten! Please wait for the next CORE-Update! Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#5
|
|||
|
|||
Works as it should, thank you for your commitment, i love this new feature!
|
#6
|
||||
|
||||
Thanks for your feedback and thanks for the early testing!
I've added some other improvements to this plugin but it's stuff that you won't necessarily "see"^^
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#7
|
||||
|
||||
Xatone has reported another archive.org plugin issue via PN: The download of .html files failed with error "File not found".
I've fixed this one too. Bitte auf das nächste CORE-Update warten! Please wait for the next CORE-Update! Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#8
|
|||
|
|||
link collector breaks beyond 10k links
Try parsing this:
**External links are only visible to Support Staff****External links are only visible to Support Staff** He will eventually get all 14k files, but you cannot start downloading anything. If you try to parse this one: **External links are only visible to Support Staff****External links are only visible to Support Staff** He won't even be able to parse a single link |
#9
|
||||
|
||||
Merged archive.org threads.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#10
|
||||
|
||||
CORE-Updates have been released!
All announced bugfixes and features are live! Please update your JDownloader and report any issues you find asap. If this thread gets marked as "[Solved]" by our forum staff you can still post in it and we will read- and reply to it! CORE-Updates wurden released! Alle angekündigten Bugfixes/Features sind nun verfügbar! Bitte JDownloader updaten und eventuelle Bugs schnellstmöglich an uns melden. Falls dieser Thread vom Team als "[Erledigt]" markiert wird, kannst du weiterhin darin antworten und wir lesen/beantworten auch solche Threads! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#11
|
|||
|
|||
Link grabber not working for archive.org
Please fix this... it used to work with missing files if we wanted to create a DLC. Now it can't add if the links are offline.
And the online links don't seem to work, too. That dawn update broke JDW for good or this server is broken |
#12
|
|||
|
|||
@Perene: Please provide example links that fail for you
|
#13
|
|||
|
|||
|
#14
|
||||
|
||||
No need to double post.
Please provide the missing information such as example URLs and logs. Please post your log-ID here | bitte poste deine Log-ID hier. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#15
|
|||
|
|||
Quote:
- Link grabber is working, so if the link really exists in "archive" servers, it will be detected after a copy-paste. - But it's not working anymore if the link DOES NOT EXIST. What? Before this update, I used to add missing links to create a DLC before sending the file itself. Now, unless the link really exists, you can't do that anymore. So no results for the linkgrabber. It will "scan" for the link and cease trying to find anything. So no results of "this link is offline" in that DLC. (note: haven't checked old DLCs with missing links, I am talking about new stuff, created now). If you used "archive" before, you know this DLC with links that still don't exist but you plan to add there is useful, because the "DERIVE" step of the process can take hours if not days. So now we are forced to wait untill everything is ONLINE to use linkgrabber and create this new DLC. Example of what I am talking about: PLATO-DIALOGUES.part1.7z PLATO-DIALOGUES.part2.7z PLATO-DIALOGUES.part3.7z PLATO-DIALOGUES.part4.7z (still not online, link does not exist) JDW will scan parts 1-3 and will completely ignore part 4. Before: part 4 listed, but with a red flag and saying OFFLINE. ******** Second change I noticed: - Before: Let's say you created a series of 7z files with this name: PLATO-DIALOGUES.part1.7z PLATO-DIALOGUES.part2.7z And so on. Posted on a page called OPENLIBRARY. OPENLIBRARY (page name, with all the files)/PLATO-DIALOGUES.part1.7z Now when we add to Linkgrabber, it will list all these parts like this: - OPENLIBRARY (15 links) Before: PLATO-DIALOGUES (15 links) Now the container name is based on the server page, not the most common filename of that list. I have all these enabled, in this order: https://support.jdownloader.org/Know...age-feature/12 https://support.jdownloader.org/Know...lder-structure Last edited by Perene; 28.03.2023 at 19:01. |
#16
|
||||
|
||||
Quote:
Please post your log-ID here | bitte poste deine Log-ID hier. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#17
|
|||
|
|||
@Perene: Please provide example links that contain *not exist/offline* links. I think I understand what you're trying to say but we need example links for testing/fixing. Also please provide example links for the file package naming
Edit: I think I found cause of your issue, will be changed with next update.But would be nice you still could provide example links with offline links Last edited by notice; 28.03.2023 at 20:03. |
#18
|
|||
|
|||
For the 1st complaint:
**External links are only visible to Support Staff****External links are only visible to Support Staff** This link does not exist, so it can't be recognized by JDW. (only the server page is valid, not the file) About the 2nd issue mentioned: **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** When you add all 4, the package should be named 9X_Files and not WULOSTE... as JDW is now showing. The correct name was based on all these 4 parts. If JDW behaves differently, then it also complicates things. (Unless, of course, we can switch between the two methods) Last edited by Perene; 29.03.2023 at 01:58. |
#19
|
||||
|
||||
1. This regression happened due to my refactoring of this crawler.
Indeed this is not the intended behavior. We'll work on this soon. 2. Technically this is not a problem in this case. Explanation: Archive.org is treated with the default behavior for cloud folders. Explanation here: https://support.jdownloader.org/Know...lder-structure The adopted folder structure of "/archive.org/download/rootFoldername/filename.ext" is "/rootFoldername". This is exactly why "/rootFoldername" is set as packagename. This same logic is used for all other cloud folder crawlers too. JDownloader cannot know in beforehand that you'Re adding those "single files of a folder" independent from the folder thus it will set a "meaningful folder structure" and packagename (= same as folder structure). If you're doing exactly this more frequently, I suggest creating a LinkCrawler rule. To "solve" this for you via a setting and only for archive.org, a setting like "Use auto folder handling for loose files" could be an idea. I don't know how exactly the archive.org plugin was behaving before the recent big update but if it behaved differently than now, that was not the intended behavior.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 29.03.2023 at 14:50. Reason: Better formatting |
#20
|
|||
|
|||
@Perene: I've updated plugin to behave different. when adding /download/ link to a specific file and no folder (like your examples) then no package will be set and JDownloader will auto package it.
please try again with next core update and provide feedback Last edited by notice; 29.03.2023 at 19:37. |
#21
|
||||
|
||||
@Perene
That means both of your points have been addressed. Please read: Bitte auf das nächste CORE-Update warten! Please wait for the next CORE-Update! Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#22
|
|||
|
|||
I noticed after the I.A. verdict that download speeds are now limited to 1 MB/s or half that, no matter how many times I tried with JDW or changed my dynamic IP. Also seems to make no difference if I add a free account there or not.
I need to ask here if this may be related to the recent changes you made for this server. Because if you are getting more than 1 MB/s (download), then the answer is no. If possible, please try to download anything from there and tell me how fast it is. Example: **External links are only visible to Support Staff****External links are only visible to Support Staff** And even if you are getting 1 MB/s, this may not be related at all with the JDW changes, perhaps it's then again capping the bandwidth or experiencing technical issues. Also probably due to this recent and worldwide exposure. This sort of thing has been complained with I.A. ad nauseam. Oddly before this verdict I was getting much better DL speeds, and was able to download dozens of GB in the same day. Now when I try this, it never surpasses 1 MB/s. It's also possible my ISP may be having issues where I live or did this to them (unlikely). So there are lots of combined factors that may account for this, and it may be temporary or permanent. P.S. Speeds are now 1-2 MB/s. That is, if I try to download as much different links as possible. Combined, I can get 1-2 MB/s DL, no more than that. Last edited by Perene; 01.04.2023 at 02:01. |
#23
|
||||
|
||||
Quote:
Are you getting better speeds via browser and/or when using a VPN? Please go through this troubleshooting guide: https://support.jdownloader.org/Know...-downloadspeed Quote:
That is very slow. I got a 1GiB/s connection here. I guess the file is on a slow server or there is a routing issue.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#24
|
|||
|
|||
If you are also getting bad speeds, then we don't need to look further. It's I.A. again having server problems, they are infamous for capping all their speeds (as the tweet below indicates)
**External links are only visible to Support Staff****External links are only visible to Support Staff** What changed now is that downloading is a LOT worse. So bad that I had to ask here. I have a 600/300 Mbps internet and never managed to get more than 6, 7 MB/s in terms of uploading (even so, this is quite good, Bandwidth Monitor is the app I use to measure this), but these days the DL is reduced to 1-2 MB/s tops. For example, right now 10 links are being downloaded and the speeds combined are 0.7 MB/s. The best I was able to achieve was 2 MB/s. And I try downloading all day. What I used to get in hours now takes days, if each link is from a bigger file. 1 MB/s is very slow compared to what it was before this verdict, it was probably 5-10 times faster. A VPN is not necessary to test this, because if you are having the same problems and we are from totally different countries, this means they are not capping based on IP/range, which is another possibility I suspected. I hope in the future this can be fixed, if it's temporary only time will tell... |
#25
|
||||
|
||||
Quote:
Quote:
If it is e.g. a serverside global limit by IP, you could try to use multiple proxies/VPNs to get around it. Quote:
I remember getting at least 30MiB/s. Quote:
Did you also contact their support or do you rely on other users doing this?
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#26
|
||||
|
||||
CORE-Updates have been released!
All announced bugfixes and features are live! Please update your JDownloader and report any issues you find asap. If this thread gets marked as "[Solved]" by our forum staff you can still post in it and we will read- and reply to it! CORE-Updates wurden released! Alle angekündigten Bugfixes/Features sind nun verfügbar! Bitte JDownloader updaten und eventuelle Bugs schnellstmöglich an uns melden. Falls dieser Thread vom Team als "[Erledigt]" markiert wird, kannst du weiterhin darin antworten und wir lesen/beantworten auch solche Threads! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#27
|
|||
|
|||
A guy that works there replied to my complaint the DL speeds are down to 300 KB/s now. He said:
"It's not the server. In the power event a week and a half ago, our main router took extraordinary damage and we've ordered a replacement for it that will arrive and will fix the issue." This was an odd coincidence of events, and the speeds are really pathetic right now, I was thinking it could be many things, even that I had to add a free account into JDW. I am glad this has been clarified, and JDW fixed the other problems. \o/ |
#28
|
|||
|
|||
@Perene: Thank you for sharing this information
|
#29
|
|||
|
|||
Can you explain if there's some logic on IA download speeds with 11 links at the same time reaching 6, 7 MB/s total, and then dropping to 300 KB/s (total) when only 4 are being download also at the same time? I noticed that drop imediatelly after.
Is this a server limitation from them or some sort of configuration from my ISP or JDW that accounts for that sudden drop? I also had ISP issues this week, but today they seem to have been fixed (and were affecting DL speeds). |
#30
|
||||
|
||||
JD does not even have the ability to limit single downloads atm.
There are all kinds of possible reasons: - You're downloading each file with multiple chunks and once the majority of them disconnects, speed drops. At this moment JD won't re-order chunk-sizes dynamically so if this was the reason of your problems, there would be no solution. - Archive.org is purposely recognizing- and limiting JDownloader users (chances are low I think) - Routing issues (using a VPN could solve such problems) Basically everything stated in this article.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#31
|
|||
|
|||
Looks like IA fixed both DL and UL speeds.
But the 2nd issue mentioned in post #18 is not 100% fixed. I'll explain: When you add several links, a few still use the page URL as a package name. I'll give you an example to show how this is not working as it should: 172 links were added using the filename as a future new folder. OK and expected (it was like this before) 16 were not. (no reason) Look at the screenshot: **External links are only visible to Support Staff****External links are only visible to Support Staff** The correct behavior would be: ALL of them with that 1st name as folder, not the 2nd one which is the page-URL. You can check the DLC for yourself here, instead of doing a copy-paste of 188 links. **External links are only visible to Support Staff****External links are only visible to Support Staff** After downloading, simply add to JDW and then you'll probably get the same results. Last edited by Perene; 15.05.2023 at 02:16. |
#32
|
||||
|
||||
@Perene
Thanks for reporting. Due to a bug in an encoding library, the "single files handling" fails for items containing one or multiple apostrophe characters. We're working on it...
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#33
|
|||
|
|||
Looks like a new issue is there: when I only add a specific file (URL), JDW is only checking the entire page.
Example: I asked for this file: **External links are only visible to Support Staff****External links are only visible to Support Staff** But JDW only shows this: https://i.postimg.cc/g28CR6wm/JDW.png In other words, all dozens of files from the page (not what I requested): **External links are only visible to Support Staff****External links are only visible to Support Staff** |
#34
|
||||
|
||||
Fixed.
Regarding that other encoding issue reported before post #32: I'm unable to reproduce this and I'll mark this thread as solved for now. It has possibly been fixed when we switched over to a different "check for single file". If you can still reproduce this, please provide further logs. Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#35
|
|||
|
|||
New archive.org problem
Hi, I think there are new archive.org problems.
In jdownloader 2 build date "Tue Aug 29 19:09:55 CEST 2023", I could paste a archive.org link into linkgrabber, and it would crawl and enumerate all the files in the archive. As of jdownloader 2 build date "Wed Aug 30 16:16:27 CEST 2023", when I paste an archive.org link into linkgrabber, it only shows the torrent of the archive. I have an archive.org login, and I have tried all of the archive.org plugin settings, with the best i could do was also to see the sqlite and xml files in addition to the torrent. Here are support logs- Working: 01.09.23 10.26.35 <--> 30.08.23 14.33.14 jdlog://3547311370661/ Not working: 01.09.23 10.26.35 <--> 01.09.23 10.46.55 jdlog://2547311370661/ Thanks! |
#36
|
||||
|
||||
Yes there was a regression - sorry for that!
It will be fixed soon and announced here: https://board.jdownloader.org/showthread.php?t=94212
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#37
|
|||
|
|||
Quote:
**External links are only visible to Support Staff****External links are only visible to Support Staff** But, if we visit the ARCHIVE page, and copy from the server, it will recognize fully: **External links are only visible to Support Staff****External links are only visible to Support Staff** What's JDW problem? Looks like it's not recognizing filenames with this symbol: # Note: the 1st URL from pastes.io is recognized like the 2nd, if JDW is working fine. But it's not, simply because that symbol is there. But if IA's server allows the # symbol, why does JDW refuses in the 1st example (by saying it's offline)? |
#38
|
|||
|
|||
All fixed, updated to sept 4 build and it works again. Thank you!
|
#39
|
||||
|
||||
Quote:
When I open that link in browser, all I get is a 404 error page of archive.org. Your 2nd link also fails. Quote:
I do not see a JD problem here. Both links fail via browser too.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#40
|
|||
|
|||
I thought you knew about this, so I didn't give valid URLs. Since you don't, I have uploaded two files, so now you can check the problem...
First link: **External links are only visible to Support Staff****External links are only visible to Support Staff** Second: **External links are only visible to Support Staff****External links are only visible to Support Staff** The 1st can't be detected by JDW (says OFFLINE) - but it's online, too. In case you want to check their location in the IA server: **External links are only visible to Support Staff****External links are only visible to Support Staff** And the full URL address from both, with spaces (this way works for the linkcollector, too): **External links are only visible to Support Staff****External links are only visible to Support Staff** This is the 1st time I noticed this, and it's always due to this symbol: # Last edited by Perene; 22.11.2023 at 20:46. |
Thread Tools | |
Display Modes | |
|
|