[pspzockerscene]

Both links are working fine here.
Screenshot:

Spoiler:

Please provide a debug-log:

Please post your log-ID here
If your bugreport is about a specific website which JD supports via plugin, please also provide example URLs which can be used to reproduce the issue you are having.

Bitte poste deine Log-ID hier.
Falls dein gemeldetes Problem ein Problem mit einer Webseite ist, die per Plugin unterstützt wird, stelle bitte zusätzlich Testlinks zur Verfügung, mit denen sich dein Problem nachstellen lässt.

-psp-

[Perene]

You are correct, both are saying online.

But if you try with spaces, they are not working, only the one without the # symbol.

So, if you try the links from my previous posts, 2/2 online.

Try again with spaces, from here:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Just do a control + C and V from pastes.io, and you'll notice the issue.

[pspzockerscene]

Again:
Your "First link" in that pastebin fails in JD & browser.
Your "Second" link in that pastebin works in JD & browser.

I still can't see any JDownloader bug here.
Also you didn't answer my question:

Originally Posted by pspzockerscene

Where did you get the non-url-encoded link from?

So again:
Where did you get the non-url-encoded links / links with spaces from?

[Perene]

OK, sorry for not explaining.

Since I own the file, if I were to post something like this to you:

The%20Woody%20Woodpecker%20Show%20%281957-1977%29%20-%20Epis%C3%B3dio%202x23%20%28%2349%29%20-%20%C3%81udio%20Original%20-%20Legendado%20%28Web-DL%29.mp4

You would have a problem reading that.

So, it would be better if I opted to write like this:

URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (#49) - Áudio Original - Legendado (Web-DL).mp4

The thing is, JDW recognizes anything from IA if I write like in the 2nd method. Either way, it says ONLINE. Regardless of how you type.

But if you try to insert that symbol (non-URL encoded link), JDW refuses to tell it's online. But it is.

That's why I had to rename that same file to:

URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (-49-) - Áudio Original - Legendado (Web-DL).mp4

Bottom line:

JDW recognizes something like this:

The%20Woody%20Woodpecker%20Show%20%281957-1977%29%20-%20Epis%C3%B3dio%202x23%20%28%2349%29%20-%20%C3%81udio%20Original%20-%20Legendado%20%28Web-DL%29.mp4

But not like this:

URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (#49) - Áudio Original - Legendado (Web-DL).mp4

And the only reason is the # symbol.

[pspzockerscene]

I still don't understand.

Here is my point of view:
For some reason you removed the URL-encoding of this URL.
This broke the URL in JDownloader in broweser but you expect JDownloader to recognize it anyways.

The correct solution would be to either fix the URL yourself or not break it in the first place.
See e.g.:
urlencoder.org/

[Perene]

My point is:

******************************

When you click with your right button after visiting the Internet Archive server, and try to copy the URL from a file stored there, it will be showed like this:

The%20Woody%20Woodpecker%20Show%20%281957-1977%29%20-%20Epis%C3%B3dio%202x23%20%28%2349%29%20-%20%C3%81udio%20Original%20-%20Legendado%20%28Web-DL%29.mp4

The problem with that idea is if you are trying to display that URL in a blog or to a specific person, it will look like garbage (as if you are typing nonsense: KDdgjadgjadfgaskjfg927o98427942jbewfdsf5546bsfLSDASLDHA )

I can't tell from just looking at that the * real * filename.

So, I need to show it like this:

URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (-49-) - Áudio Original - Legendado (Web-DL).mp4

What's the difference between providing JDOWNLOADER's Link Collector with the two examples?

None whatsoever.

I can say the same about Internet Download Manager. Or any other program. They simply recognize (after checking IA's server) that URL is valid, regardless of you putting something between spaces or not.

Informing the filename with spaces (if they really exist, of course) or not, makes no difference. ZERO.

******************************
Right. Now I can proceed with the problem we are having here...

******************************

Since this is only a choice, nothing to do, right? Just do a control + C and V, so JDW will say the two methods are online.

Wrong.

If the filename contains the # symbol, this is what I discovered:

This will work:

The%20Woody%20Woodpecker%20Show%20%281957-1977%29%20-%20Epis%C3%B3dio%202x23%20%28%2349%29%20-%20%C3%81udio%20Original%20-%20Legendado%20%28Web-DL%29.mp4

This will not:

URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (#49) - Áudio Original - Legendado (Web-DL).mp4

Why? It's the same valid URL!

It is. The difference is that JDW is not seeing as it should.

What did I do to "fix" this issue in the meantime?

I had to rename all the files affected, and use -49- (or anything similar) instead of #49, because your link collector is not working in this scenario. The # symbol breaks the recognition. If, and only if, the provided URL has spaces.

[pspzockerscene]

I do not see any JD issue here.
Please manually fix such URLs if you keep breaking them yourself by re-applying the URL-encoding to the part of said URLs which contains the filenames.

[Perene]

Originally Posted by pspzockerscene

I do not see any JD issue here.

Then please check this log:

27.11.23 13.46.14 <--> 27.11.23 13.49.14 jdlog://4929311370661/

Also, check these screenshots, they help my argument more than what I wrote so far:

#1 - JDW makes a distinction because of the # symbol:

https://i.postimg.cc/0x4b5qmg/JDW-1.jpg

************************************

#2 - This is how JDW shows the URL with spaces + the # symbol: OFFLINE:

https://i.postimg.cc/dwTDLSmh/JDW-2.jpg

************************************

#3 - Now, look how JDW behaves if you try to copy-paste the same URL, without inserting spaces into the full link:

https://i.postimg.cc/vGFT9L33/JDW-3.jpg

************************************

What behavior I was expecting?

Picture #2 not saying it is offline (because it actually isn't). Somehow the # symbol makes JDW have a problem recognizing that same URL.

Why is that happening, I don't know. And why am I insisting on this? Because even if I tell JDW the provided link has spaces, your LINK-COLLECTOR automatically transforms them from this:

URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (#49) - Áudio Original - Legendado (Web-DL).mp4

Into this:

The%20Woody%20Woodpecker%20Show%20%281957-1977%29%20-%20Epis%C3%B3dio%202x23%20%28%2349%29%20-%20%C3%81udio%20Original%20-%20Legendado%20%28Web-DL%29.mp4

You are telling me to only use the latter, but the catch is: we don't need to be so precise. We never did.

We do now, because JDW is breaking the "transformation" due to the # symbol.

That's all.

If I were to guess, the # is some sort of forbidden character, which breaks the conversion since it's close to the ( ).

[pspzockerscene]

Originally Posted by Perene

If I were to guess, the # is some sort of forbidden character, which breaks the conversion since it's close to the ( ).

The # symbol is the anchor element which can be located at the end of a URL.
If you want it to be part in a filename or in the path of the URL it needs to be URL-encoded.
In your case it's not URL encoded so that URL is invalid.
Explanation:
developer.mozilla.org/en-US/docs/Learn/Common_questions/Web_mechanics/What_is_a_URL

I will not repeat this anymore. If you want to have support for broken URLs in JDownloader, feel free to implement it yourself as JDownloader is open source:
https://support.jdownloader.org/Know...up-ide-eclipse

This is my last comment on this topic.

[Perene]

No, it's not invalid. The URL is 100% fine either way, it's just a problem from your side, not ours.

If you don't want to implement fixes then this forum is pointless, If I could do your job I wouldn't be here asking for help. This forum is meant to exist for posting feedbacks on issues found on your app.

And the issue here is this (if you want me to be more precise):

JDownloader appears to have difficulty recognizing URLs that include spaces and an anchor (#) symbol, particularly when the URLs are not percent-encoded. The issue seems to affect the accurate parsing of URLs with these specific characteristics.

I suggested enhancing JDownloader's URL parsing logic to better handle URLs with spaces and the # symbol, even when these characters are not percent-encoded. This improvement could significantly benefit users who prefer providing URLs with spaces for better readability.

It would be great if JDownloader could automatically handle or convert spaces and # symbols in URLs to their proper percent-encoded equivalents to ensure compatibility and accurate parsing.

In URLs, the # symbol is often used to indicate a fragment identifier. The fragment identifier is not typically sent to the server but is processed by the client (browser or, in this case, JDownloader) to navigate within the document.

In some cases, special characters like # might interfere with how URLs are interpreted or parsed by certain software.

A few things to consider:

- Encoding the Fragment Identifier: While spaces are generally encoded as %20 in URLs, special characters like # should also be properly encoded, usually as %23. Ensure that if your URL includes a fragment identifier, it is correctly encoded.

Examples:

Original URL which JDW does not recognize:

URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (#49) - Áudio Original - Legendado (Web-DL).mp4

With the fixing applied:

URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (%2349) - Áudio Original - Legendado (Web-DL).mp4

Sure there are other priorities, but I believe this enhancement could improve the overall user experience and make JDownloader even more user-friendly.

P.S. You didn't mention if there are internal options related to URL encoding or handling special characters. I am going to assume there are none that impact how URLs are processed.

[pspzockerscene]

Soon, the "heart" of our archive.org crawler will be swappe for a new one which is using this endpoint: archive.org/metadata/<identifier>

This may or may not solve your encoding problem.
Please provide feedback once the update is live.
The update will also provide the following improvements:
- Faster crawling
- Crawling of items which failed before
- Better Handling for playlists
- Video auto handling: if a video cannot be downloaded but only streamed, the video-playlist will be returned no matter which plugin setting you are using.

[pspzockerscene]

Said update has been released.
Additional changes:

• New plugin setting: File crawler: Crawl thumbnails?
• New plugin setting: File crawler: Crawl restricted [=un-downloadable] items?
• Improved book crawl logic, fixed missing elements for multi chapter books
• Improved stream crawling: force-return video stream if original files are restricted/not-downloadable
• New plugin properties: artist, genre, is_restricted, filetype
• Book crawler: Only return selected chapter if one is contained in the added URL e.g. archive.org/details/<identifier>/<chapterID>
• Book crawler: Return results asynchronously as crawling a lot of chapters can be extremely time intensive
• Setting "File crawler: Include archive view?" has no effect for this moment since I wasn't able to find any advantages of this feature. If anyone still needs this, please let me know.

Although I've been doing extensive tests, bugs may occur so if they do, please let me know and always include example URLs in your reports.

[StormFoo]

Some URLs that previously worked for me now appear as offline. For example, any link in the following folder of this collection:

Spoiler:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Examples that don't work:

Spoiler:

* **External links are only visible to Support Staff****External links are only visible to Support Staff**
* **External links are only visible to Support Staff****External links are only visible to Support Staff**

However, I am able to download from the root folder or another subfolder in the same collection.

Examples that work:

Spoiler:

* **External links are only visible to Support Staff****External links are only visible to Support Staff**
* **External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Fixed.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[StormFoo]

Thanks, links are working again, much appreciated. I did have to update some Packagizer rules that were looking for the URL to contain specific URLs to instead look at the package name, but otherwise behaving as before.

[Jiaz]

@StormFoo: Thanks for the feedback. Could you please elaborate the part with packagizer rules to be updates? maybe some sort of regression we should look into

[Perene]

Please look into archive.org again, since it's adding all files/URLs from a page, not just those you selected it.

Example:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

I asked for the PDF only; JDW is now adding the entire page, located here:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

As you can see from here:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

The PDF is just one of the multiple files available.

Before the last update, it was adding the desired URLs, as a package with the name of the IA's page. This update broke things...

[pspzockerscene]

Thanks for your report.
Yes even after thorough testing, updating the core crawl handling of archive.org has indeed broken some stuff but in the long term this should make it work better than before.
Also the handling is not as easy as you might think since we are not(!) accessing the URL which you add but instead we're going via "/metadata/<identifier>" and then looking for the single file/folders, if one is available in the path of the added URL e.g.:
archive.org/download/warbetweenperu00mark/warbetweenperu00mark_abbyy.gz
-> Plugin goes to archive.org/metadata/warbetweenperu00mark
--> Looks for "warbetweenperu00mark_abbyy.gz" and returns only that if it exists. Same for subpaths with files in them.

I've fixed that issue.

For internal reasons, I've also updated the "single desired files" handling again:
If you add an invalid path/file with a valid identifier, all items will be returned e.g.:
archive.org/download/warbetweenperu00mark/invalidFilename.example
-> Returns all items and not a single offline item.

If you do not agree with this change, I'am willing to add a setting for this behavior.

As for the possible encoding problem you've reported a while back:
Please test if you can still reproduce that problem.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[Perene]

@pspzockerscene the plugin broke again... I ask for specific links, it's returning me the thumbnails and all the rest from the page... didn't you fix in your last post?

If you did, it looks like what I said in post #57 has returned...

P.S. I discovered more broken stuff:

I compiled a list of 22 files meant to be uploaded one by one.

17 were showed as "online", since I completed the upload.

The other 5 were not posted yet. So it should have appeared anyway, as "offline".

Problem is, they are not appearing anywhere in the linkgrabber! Not even as offline. And like I said, it's adding stuff not asked, such as the IA's torrent-link (or thumbnails).

In case you are wondering why I need to also list offline links, it's simple:

I create the DLC before the files are even uploaded to IA's server. That's because I also compile a full TXT list of them.

So this is another issue that needs fixing...

+++++++++++
If you need a test page...

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Try this one.

For example, if I wanted the MKV and nothing more:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Look what happens:

https://i.postimg.cc/wgJtWLCF/imagem...-010817423.png

Now, let's try with a link that doesn't even exist:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Did you notice it's not even added as an offline link? I don't need to post another screenshot, because it's that same from PostImg.

Expected result: show 2.1.mkv as "offline", even in link grabber.

++++++++++++++

Another odd behavior:

- If you try to add MP4 or any other file extension, it's added alone:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Proof:

https://i.postimg.cc/pRz1ZcqT/imagem...-011542571.png

Now, if I add any MKV, it's adding all the rest.

Consider that MKVs cannot be added anymore like the other filetypes.