[Xatone]

hello everyone, i have two problems with jdownloader when i try to download from archive.org. The first is that when I try to download this **External links are only visible to Support Staff****External links are only visible to Support Staff** jdownloader seems to find nothing, while the second is that I would like to include the .xml and .sqlite files automatically, from what I have seen it is possible to download them only if I directly copy the links of these files. Do you have solutions?

[pspzockerscene]

Hi,
there was a small bug in our crawler plugin which I just fixed.
It only occured for archive.org content containing a huge number of files.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Xatone]

Thank you very much

[pspzockerscene]

Publicly answering your private message here as my reply might also be useful for other users:
You've tested my changes in our dev build and told me that some .xml and .sqlite files are still missing.
Indeed this was not a "bug" as that crawled has always skipped metadata files as most of all users do not want to download them.

After some refactoring I've added another setting to our archive.org plugin which, when enabled, lets you crawl/download those files too (default = disabled).

Thanks for your feedback!

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Xatone]

Works as it should, thank you for your commitment, i love this new feature!

[pspzockerscene]

Thanks for your feedback and thanks for the early testing!

I've added some other improvements to this plugin but it's stuff that you won't necessarily "see"^^

[pspzockerscene]

Xatone has reported another archive.org plugin issue via PN: The download of .html files failed with error "File not found".

I've fixed this one too.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[widardd]

Try parsing this:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

He will eventually get all 14k files, but you cannot start downloading anything.

If you try to parse this one:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

He won't even be able to parse a single link

[pspzockerscene]

Merged archive.org threads.

[pspzockerscene]

CORE-Updates have been released!
All announced bugfixes and features are live!
Please update your JDownloader and report any issues you find asap.
If this thread gets marked as "[Solved]" by our forum staff you can still post in it and we will read- and reply to it!

CORE-Updates wurden released!
Alle angekündigten Bugfixes/Features sind nun verfügbar!
Bitte JDownloader updaten und eventuelle Bugs schnellstmöglich an uns melden.
Falls dieser Thread vom Team als "[Erledigt]" markiert wird, kannst du weiterhin darin antworten und wir lesen/beantworten auch solche Threads!

-psp-

[Perene]

Please fix this... it used to work with missing files if we wanted to create a DLC. Now it can't add if the links are offline.

And the online links don't seem to work, too.

That dawn update broke JDW for good or this server is broken

[Perene]

It's not working.

https://board.jdownloader.org/showthread.php?t=93075

[notice]

@Perene: Please provide example links that fail for you

[pspzockerscene]

No need to double post.
Please provide the missing information such as example URLs and logs.

Please post your log-ID here | bitte poste deine Log-ID hier.

-psp-

[Perene]

Originally Posted by pspzockerscene

No need to double post.
Please provide the missing information such as example URLs and logs.

Please post **External links are only visible to Support Staff**... | bitte poste **External links are only visible to Support Staff**....

-psp-

I did a few more tests and this is what I found out:

- Link grabber is working, so if the link really exists in "archive" servers, it will be detected after a copy-paste.

- But it's not working anymore if the link DOES NOT EXIST. What? Before this update, I used to add missing links to create a DLC before sending the file itself. Now, unless the link really exists, you can't do that anymore. So no results for the linkgrabber. It will "scan" for the link and cease trying to find anything.

So no results of "this link is offline" in that DLC. (note: haven't checked old DLCs with missing links, I am talking about new stuff, created now).

If you used "archive" before, you know this DLC with links that still don't exist but you plan to add there is useful, because the "DERIVE" step of the process can take hours if not days. So now we are forced to wait untill everything is ONLINE to use linkgrabber and create this new DLC.

Example of what I am talking about:

PLATO-DIALOGUES.part1.7z

PLATO-DIALOGUES.part2.7z

PLATO-DIALOGUES.part3.7z

PLATO-DIALOGUES.part4.7z (still not online, link does not exist)

JDW will scan parts 1-3 and will completely ignore part 4.

Before: part 4 listed, but with a red flag and saying OFFLINE.

********
Second change I noticed:

- Before:

Let's say you created a series of 7z files with this name:

PLATO-DIALOGUES.part1.7z

PLATO-DIALOGUES.part2.7z

And so on. Posted on a page called OPENLIBRARY.

OPENLIBRARY (page name, with all the files)/PLATO-DIALOGUES.part1.7z

Now when we add to Linkgrabber, it will list all these parts like this:

- OPENLIBRARY (15 links)

Before:

PLATO-DIALOGUES (15 links)

Now the container name is based on the server page, not the most common filename of that list.

I have all these enabled, in this order:

https://support.jdownloader.org/Know...age-feature/12

https://support.jdownloader.org/Know...lder-structure

[pspzockerscene]

Originally Posted by Perene

But it's not working anymore if the link DOES NOT EXIST. What? Before this update, I used to add missing links to create a DLC before sending the file itself. Now, unless the link really exists, you can't do that anymore. So no results for the linkgrabber.

Please provide example URLs and a log.

Please post your log-ID here | bitte poste deine Log-ID hier.

-psp-

[notice]

@Perene: Please provide example links that contain *not exist/offline* links. I think I understand what you're trying to say but we need example links for testing/fixing. Also please provide example links for the file package naming

Edit: I think I found cause of your issue, will be changed with next update.But would be nice you still could provide example links with offline links

[Perene]

For the 1st complaint:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

This link does not exist, so it can't be recognized by JDW. (only the server page is valid, not the file)

About the 2nd issue mentioned:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

When you add all 4, the package should be named 9X_Files and not WULOSTE... as JDW is now showing.

The correct name was based on all these 4 parts. If JDW behaves differently, then it also complicates things.

(Unless, of course, we can switch between the two methods)

[pspzockerscene]

1. This regression happened due to my refactoring of this crawler.
Indeed this is not the intended behavior.
We'll work on this soon.

2. Technically this is not a problem in this case.
Explanation:
Archive.org is treated with the default behavior for cloud folders. Explanation here:
https://support.jdownloader.org/Know...lder-structure
The adopted folder structure of "/archive.org/download/rootFoldername/filename.ext" is "/rootFoldername".
This is exactly why "/rootFoldername" is set as packagename.

This same logic is used for all other cloud folder crawlers too.
JDownloader cannot know in beforehand that you'Re adding those "single files of a folder" independent from the folder thus it will set a "meaningful folder structure" and packagename (= same as folder structure).

If you're doing exactly this more frequently, I suggest creating a LinkCrawler rule.

To "solve" this for you via a setting and only for archive.org, a setting like "Use auto folder handling for loose files" could be an idea.
I don't know how exactly the archive.org plugin was behaving before the recent big update but if it behaved differently than now, that was not the intended behavior.

[notice]

@Perene: I've updated plugin to behave different. when adding /download/ link to a specific file and no folder (like your examples) then no package will be set and JDownloader will auto package it.

please try again with next core update and provide feedback