JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 28.01.2023, 18:33
Xatone Xatone is offline
Modem User
 
Join Date: Jan 2023
Posts: 3
Default Archive.org Problems

hello everyone, i have two problems with jdownloader when i try to download from archive.org. The first is that when I try to download this **External links are only visible to Support Staff****External links are only visible to Support Staff** jdownloader seems to find nothing, while the second is that I would like to include the .xml and .sqlite files automatically, from what I have seen it is possible to download them only if I directly copy the links of these files. Do you have solutions?
Reply With Quote
  #2  
Old 30.01.2023, 14:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,796
Default

Hi,
there was a small bug in our crawler plugin which I just fixed.
It only occured for archive.org content containing a huge number of files.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #3  
Old 30.01.2023, 20:34
Xatone Xatone is offline
Modem User
 
Join Date: Jan 2023
Posts: 3
Default

Thank you very much
Reply With Quote
  #4  
Old 02.02.2023, 03:42
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,796
Default

Publicly answering your private message here as my reply might also be useful for other users:
You've tested my changes in our dev build and told me that some .xml and .sqlite files are still missing.
Indeed this was not a "bug" as that crawled has always skipped metadata files as most of all users do not want to download them.

After some refactoring I've added another setting to our archive.org plugin which, when enabled, lets you crawl/download those files too (default = disabled).

Thanks for your feedback!

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #5  
Old 02.02.2023, 12:15
Xatone Xatone is offline
Modem User
 
Join Date: Jan 2023
Posts: 3
Default

Works as it should, thank you for your commitment, i love this new feature!
Reply With Quote
  #6  
Old 02.02.2023, 14:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,796
Default

Thanks for your feedback and thanks for the early testing!

I've added some other improvements to this plugin but it's stuff that you won't necessarily "see"^^
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #7  
Old 13.02.2023, 13:53
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,796
Default

Xatone has reported another archive.org plugin issue via PN: The download of .html files failed with error "File not found".

I've fixed this one too.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #8  
Old 05.03.2023, 01:40
widardd widardd is offline
Modem User
 
Join Date: Mar 2020
Posts: 3
Default link collector breaks beyond 10k links

Try parsing this:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

He will eventually get all 14k files, but you cannot start downloading anything.

If you try to parse this one:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

He won't even be able to parse a single link
Reply With Quote
  #9  
Old 14.03.2023, 19:03
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,796
Default

Merged archive.org threads.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #10  
Old 24.03.2023, 15:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,796
Default

CORE-Updates have been released!
All announced bugfixes and features are live!
Please update your JDownloader and report any issues you find asap.
If this thread gets marked as "[Solved]" by our forum staff you can still post in it and we will read- and reply to it!

CORE-Updates wurden released!
Alle angekündigten Bugfixes/Features sind nun verfügbar!
Bitte JDownloader updaten und eventuelle Bugs schnellstmöglich an uns melden.
Falls dieser Thread vom Team als "[Erledigt]" markiert wird, kannst du weiterhin darin antworten und wir lesen/beantworten auch solche Threads!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #11  
Old 28.03.2023, 09:57
Perene Perene is offline
JD Adviser
 
Join Date: Jun 2019
Posts: 113
Default Link grabber not working for archive.org

Please fix this... it used to work with missing files if we wanted to create a DLC. Now it can't add if the links are offline.

And the online links don't seem to work, too.

That dawn update broke JDW for good or this server is broken
Reply With Quote
  #12  
Old 28.03.2023, 09:59
Perene Perene is offline
JD Adviser
 
Join Date: Jun 2019
Posts: 113
Default

It's not working.

https://board.jdownloader.org/showthread.php?t=93075
Reply With Quote
  #13  
Old 28.03.2023, 11:22
notice notice is offline
JD Supporter
 
Join Date: Mar 2023
Posts: 506
Default

@Perene: Please provide example links that fail for you
Reply With Quote
  #14  
Old 28.03.2023, 13:41
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,796
Default

No need to double post.
Please provide the missing information such as example URLs and logs.

Please post your log-ID here | bitte poste deine Log-ID hier.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #15  
Old 28.03.2023, 18:56
Perene Perene is offline
JD Adviser
 
Join Date: Jun 2019
Posts: 113
Default

Quote:
Originally Posted by pspzockerscene View Post
No need to double post.
Please provide the missing information such as example URLs and logs.

Please post **External links are only visible to Support Staff**... | bitte poste **External links are only visible to Support Staff**....

-psp-
I did a few more tests and this is what I found out:

- Link grabber is working, so if the link really exists in "archive" servers, it will be detected after a copy-paste.

- But it's not working anymore if the link DOES NOT EXIST. What? Before this update, I used to add missing links to create a DLC before sending the file itself. Now, unless the link really exists, you can't do that anymore. So no results for the linkgrabber. It will "scan" for the link and cease trying to find anything.

So no results of "this link is offline" in that DLC. (note: haven't checked old DLCs with missing links, I am talking about new stuff, created now).

If you used "archive" before, you know this DLC with links that still don't exist but you plan to add there is useful, because the "DERIVE" step of the process can take hours if not days. So now we are forced to wait untill everything is ONLINE to use linkgrabber and create this new DLC.

Example of what I am talking about:

PLATO-DIALOGUES.part1.7z

PLATO-DIALOGUES.part2.7z

PLATO-DIALOGUES.part3.7z

PLATO-DIALOGUES.part4.7z (still not online, link does not exist)

JDW will scan parts 1-3 and will completely ignore part 4.

Before: part 4 listed, but with a red flag and saying OFFLINE.

********
Second change I noticed:

- Before:

Let's say you created a series of 7z files with this name:

PLATO-DIALOGUES.part1.7z

PLATO-DIALOGUES.part2.7z

And so on. Posted on a page called OPENLIBRARY.

OPENLIBRARY (page name, with all the files)/PLATO-DIALOGUES.part1.7z

Now when we add to Linkgrabber, it will list all these parts like this:

- OPENLIBRARY (15 links)

Before:

PLATO-DIALOGUES (15 links)

Now the container name is based on the server page, not the most common filename of that list.

I have all these enabled, in this order:

https://support.jdownloader.org/Know...age-feature/12

https://support.jdownloader.org/Know...lder-structure

Last edited by Perene; 28.03.2023 at 19:01.
Reply With Quote
  #16  
Old 28.03.2023, 19:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,796
Default

Quote:
Originally Posted by Perene View Post
But it's not working anymore if the link DOES NOT EXIST. What? Before this update, I used to add missing links to create a DLC before sending the file itself. Now, unless the link really exists, you can't do that anymore. So no results for the linkgrabber.
Please provide example URLs and a log.

Please post your log-ID here | bitte poste deine Log-ID hier.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #17  
Old 28.03.2023, 19:57
notice notice is offline
JD Supporter
 
Join Date: Mar 2023
Posts: 506
Default

@Perene: Please provide example links that contain *not exist/offline* links. I think I understand what you're trying to say but we need example links for testing/fixing. Also please provide example links for the file package naming

Edit: I think I found cause of your issue, will be changed with next update.But would be nice you still could provide example links with offline links

Last edited by notice; 28.03.2023 at 20:03.
Reply With Quote
  #18  
Old 29.03.2023, 01:55
Perene Perene is offline
JD Adviser
 
Join Date: Jun 2019
Posts: 113
Default

For the 1st complaint:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

This link does not exist, so it can't be recognized by JDW. (only the server page is valid, not the file)

About the 2nd issue mentioned:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

When you add all 4, the package should be named 9X_Files and not WULOSTE... as JDW is now showing.

The correct name was based on all these 4 parts. If JDW behaves differently, then it also complicates things.

(Unless, of course, we can switch between the two methods)

Last edited by Perene; 29.03.2023 at 01:58.
Reply With Quote
  #19  
Old 29.03.2023, 14:49
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,796
Default

1. This regression happened due to my refactoring of this crawler.
Indeed this is not the intended behavior.
We'll work on this soon.

2. Technically this is not a problem in this case.
Explanation:
Archive.org is treated with the default behavior for cloud folders. Explanation here:
https://support.jdownloader.org/Know...lder-structure
The adopted folder structure of "/archive.org/download/rootFoldername/filename.ext" is "/rootFoldername".
This is exactly why "/rootFoldername" is set as packagename.

This same logic is used for all other cloud folder crawlers too.
JDownloader cannot know in beforehand that you'Re adding those "single files of a folder" independent from the folder thus it will set a "meaningful folder structure" and packagename (= same as folder structure).

If you're doing exactly this more frequently, I suggest creating a LinkCrawler rule.

To "solve" this for you via a setting and only for archive.org, a setting like "Use auto folder handling for loose files" could be an idea.
I don't know how exactly the archive.org plugin was behaving before the recent big update but if it behaved differently than now, that was not the intended behavior.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 29.03.2023 at 14:50. Reason: Better formatting
Reply With Quote
  #20  
Old 29.03.2023, 19:08
notice notice is offline
JD Supporter
 
Join Date: Mar 2023
Posts: 506
Default

@Perene: I've updated plugin to behave different. when adding /download/ link to a specific file and no folder (like your examples) then no package will be set and JDownloader will auto package it.

please try again with next core update and provide feedback

Last edited by notice; 29.03.2023 at 19:37.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 19:23.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.