JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #41  
Old 23.11.2023, 11:37
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

Both links are working fine here.
Screenshot:
Spoiler:

Please provide a debug-log:

Please post your log-ID here
If your bugreport is about a specific website which JD supports via plugin, please also provide example URLs which can be used to reproduce the issue you are having.

Bitte poste deine Log-ID hier.
Falls dein gemeldetes Problem ein Problem mit einer Webseite ist, die per Plugin unterstützt wird, stelle bitte zusätzlich Testlinks zur Verfügung, mit denen sich dein Problem nachstellen lässt.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #42  
Old 23.11.2023, 17:03
Perene Perene is offline
Bandwidth Beast
 
Join Date: Jun 2019
Posts: 132
Default

You are correct, both are saying online.

But if you try with spaces, they are not working, only the one without the # symbol.

So, if you try the links from my previous posts, 2/2 online.

Try again with spaces, from here:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Just do a control + C and V from pastes.io, and you'll notice the issue.
Reply With Quote
  #43  
Old 23.11.2023, 17:05
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

Again:
Your "First link" in that pastebin fails in JD & browser.
Your "Second" link in that pastebin works in JD & browser.

I still can't see any JDownloader bug here.
Also you didn't answer my question:
Quote:
Originally Posted by pspzockerscene View Post
Where did you get the non-url-encoded link from?
So again:
Where did you get the non-url-encoded links / links with spaces from?
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #44  
Old 23.11.2023, 19:19
Perene Perene is offline
Bandwidth Beast
 
Join Date: Jun 2019
Posts: 132
Default

OK, sorry for not explaining.

Since I own the file, if I were to post something like this to you:

The%20Woody%20Woodpecker%20Show%20%281957-1977%29%20-%20Epis%C3%B3dio%202x23%20%28%2349%29%20-%20%C3%81udio%20Original%20-%20Legendado%20%28Web-DL%29.mp4

You would have a problem reading that.

So, it would be better if I opted to write like this:

URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (#49) - Áudio Original - Legendado (Web-DL).mp4

The thing is, JDW recognizes anything from IA if I write like in the 2nd method. Either way, it says ONLINE. Regardless of how you type.

But if you try to insert that symbol (non-URL encoded link), JDW refuses to tell it's online. But it is.

That's why I had to rename that same file to:

URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (-49-) - Áudio Original - Legendado (Web-DL).mp4

Bottom line:

JDW recognizes something like this:

The%20Woody%20Woodpecker%20Show%20%281957-1977%29%20-%20Epis%C3%B3dio%202x23%20%28%2349%29%20-%20%C3%81udio%20Original%20-%20Legendado%20%28Web-DL%29.mp4

But not like this:

URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (#49) - Áudio Original - Legendado (Web-DL).mp4

And the only reason is the # symbol.

Last edited by Perene; 23.11.2023 at 19:22.
Reply With Quote
  #45  
Old 24.11.2023, 12:03
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

I still don't understand.

Here is my point of view:
For some reason you removed the URL-encoding of this URL.
This broke the URL in JDownloader in broweser but you expect JDownloader to recognize it anyways.

The correct solution would be to either fix the URL yourself or not break it in the first place.
See e.g.:
urlencoder.org/
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #46  
Old 24.11.2023, 21:51
Perene Perene is offline
Bandwidth Beast
 
Join Date: Jun 2019
Posts: 132
Default

My point is:

******************************

When you click with your right button after visiting the Internet Archive server, and try to copy the URL from a file stored there, it will be showed like this:

The%20Woody%20Woodpecker%20Show%20%281957-1977%29%20-%20Epis%C3%B3dio%202x23%20%28%2349%29%20-%20%C3%81udio%20Original%20-%20Legendado%20%28Web-DL%29.mp4

The problem with that idea is if you are trying to display that URL in a blog or to a specific person, it will look like garbage (as if you are typing nonsense: KDdgjadgjadfgaskjfg927o98427942jbewfdsf5546bsfLSDASLDHA )

I can't tell from just looking at that the * real * filename.

So, I need to show it like this:

URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (-49-) - Áudio Original - Legendado (Web-DL).mp4

What's the difference between providing JDOWNLOADER's Link Collector with the two examples?

None whatsoever.

I can say the same about Internet Download Manager. Or any other program. They simply recognize (after checking IA's server) that URL is valid, regardless of you putting something between spaces or not.

Informing the filename with spaces (if they really exist, of course) or not, makes no difference. ZERO.

******************************
Right. Now I can proceed with the problem we are having here...

******************************

Since this is only a choice, nothing to do, right? Just do a control + C and V, so JDW will say the two methods are online.

Wrong.

If the filename contains the # symbol, this is what I discovered:

This will work:

The%20Woody%20Woodpecker%20Show%20%281957-1977%29%20-%20Epis%C3%B3dio%202x23%20%28%2349%29%20-%20%C3%81udio%20Original%20-%20Legendado%20%28Web-DL%29.mp4

This will not:

URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (#49) - Áudio Original - Legendado (Web-DL).mp4

Why? It's the same valid URL!

It is. The difference is that JDW is not seeing as it should.

What did I do to "fix" this issue in the meantime?

I had to rename all the files affected, and use -49- (or anything similar) instead of #49, because your link collector is not working in this scenario. The # symbol breaks the recognition. If, and only if, the provided URL has spaces.

Last edited by Perene; 24.11.2023 at 21:57.
Reply With Quote
  #47  
Old 27.11.2023, 13:18
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

I do not see any JD issue here.
Please manually fix such URLs if you keep breaking them yourself by re-applying the URL-encoding to the part of said URLs which contains the filenames.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #48  
Old 27.11.2023, 19:04
Perene Perene is offline
Bandwidth Beast
 
Join Date: Jun 2019
Posts: 132
Default

Quote:
Originally Posted by pspzockerscene View Post
I do not see any JD issue here.
Then please check this log:

27.11.23 13.46.14 <--> 27.11.23 13.49.14 jdlog://4929311370661/

Also, check these screenshots, they help my argument more than what I wrote so far:

#1 - JDW makes a distinction because of the # symbol:

https://i.postimg.cc/0x4b5qmg/JDW-1.jpg

************************************

#2 - This is how JDW shows the URL with spaces + the # symbol: OFFLINE:

https://i.postimg.cc/dwTDLSmh/JDW-2.jpg

************************************

#3 - Now, look how JDW behaves if you try to copy-paste the same URL, without inserting spaces into the full link:

https://i.postimg.cc/vGFT9L33/JDW-3.jpg

************************************

What behavior I was expecting?

Picture #2 not saying it is offline (because it actually isn't). Somehow the # symbol makes JDW have a problem recognizing that same URL.

Why is that happening, I don't know. And why am I insisting on this? Because even if I tell JDW the provided link has spaces, your LINK-COLLECTOR automatically transforms them from this:

URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (#49) - Áudio Original - Legendado (Web-DL).mp4

Into this:

The%20Woody%20Woodpecker%20Show%20%281957-1977%29%20-%20Epis%C3%B3dio%202x23%20%28%2349%29%20-%20%C3%81udio%20Original%20-%20Legendado%20%28Web-DL%29.mp4

You are telling me to only use the latter, but the catch is: we don't need to be so precise. We never did.

We do now, because JDW is breaking the "transformation" due to the # symbol.

That's all.

If I were to guess, the # is some sort of forbidden character, which breaks the conversion since it's close to the ( ).

Last edited by Perene; 27.11.2023 at 19:07.
Reply With Quote
  #49  
Old 27.11.2023, 19:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

Quote:
Originally Posted by Perene View Post
If I were to guess, the # is some sort of forbidden character, which breaks the conversion since it's close to the ( ).
The # symbol is the anchor element which can be located at the end of a URL.
If you want it to be part in a filename or in the path of the URL it needs to be URL-encoded.
In your case it's not URL encoded so that URL is invalid.
Explanation:
developer.mozilla.org/en-US/docs/Learn/Common_questions/Web_mechanics/What_is_a_URL

I will not repeat this anymore. If you want to have support for broken URLs in JDownloader, feel free to implement it yourself as JDownloader is open source:
https://support.jdownloader.org/Know...up-ide-eclipse

This is my last comment on this topic.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #50  
Old 28.11.2023, 01:39
Perene Perene is offline
Bandwidth Beast
 
Join Date: Jun 2019
Posts: 132
Default

No, it's not invalid. The URL is 100% fine either way, it's just a problem from your side, not ours.

If you don't want to implement fixes then this forum is pointless, If I could do your job I wouldn't be here asking for help. This forum is meant to exist for posting feedbacks on issues found on your app.

And the issue here is this (if you want me to be more precise):

JDownloader appears to have difficulty recognizing URLs that include spaces and an anchor (#) symbol, particularly when the URLs are not percent-encoded. The issue seems to affect the accurate parsing of URLs with these specific characteristics.

I suggested enhancing JDownloader's URL parsing logic to better handle URLs with spaces and the # symbol, even when these characters are not percent-encoded. This improvement could significantly benefit users who prefer providing URLs with spaces for better readability.

It would be great if JDownloader could automatically handle or convert spaces and # symbols in URLs to their proper percent-encoded equivalents to ensure compatibility and accurate parsing.

In URLs, the # symbol is often used to indicate a fragment identifier. The fragment identifier is not typically sent to the server but is processed by the client (browser or, in this case, JDownloader) to navigate within the document.

In some cases, special characters like # might interfere with how URLs are interpreted or parsed by certain software.

A few things to consider:

- Encoding the Fragment Identifier: While spaces are generally encoded as %20 in URLs, special characters like # should also be properly encoded, usually as %23. Ensure that if your URL includes a fragment identifier, it is correctly encoded.

Examples:

Original URL which JDW does not recognize:

Code:
URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (#49) - Áudio Original - Legendado (Web-DL).mp4
With the fixing applied:

Code:
URL from the server/The Woody Woodpecker Show (1957-1977) - Episódio 2x23 (%2349) - Áudio Original - Legendado (Web-DL).mp4
Sure there are other priorities, but I believe this enhancement could improve the overall user experience and make JDownloader even more user-friendly.

P.S. You didn't mention if there are internal options related to URL encoding or handling special characters. I am going to assume there are none that impact how URLs are processed.
Reply With Quote
  #51  
Old 20.03.2024, 14:07
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

Soon, the "heart" of our archive.org crawler will be swapped for a new one which is using this endpoint: archive.org/metadata/<identifier>

This may or may not solve your encoding problem.
Please provide feedback once the update is live.
The update will also provide the following improvements:
- Faster crawling
- Crawling of items which failed before
- Better Handling for playlists
- Video auto handling: if a video cannot be downloaded but only streamed, the video-playlist will be returned no matter which plugin setting you are using.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 18.04.2024 at 20:29. Reason: Fixed typo
Reply With Quote
  #52  
Old 28.03.2024, 17:32
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

Said update has been released.
Additional changes:
  • New plugin setting: File crawler: Crawl thumbnails?
  • New plugin setting: File crawler: Crawl restricted [=un-downloadable] items?
  • Improved book crawl logic, fixed missing elements for multi chapter books
  • Improved stream crawling: force-return video stream if original files are restricted/not-downloadable
  • New plugin properties: artist, genre, is_restricted, filetype
  • Book crawler: Only return selected chapter if one is contained in the added URL e.g. archive.org/details/<identifier>/<chapterID>
  • Book crawler: Return results asynchronously as crawling a lot of chapters can be extremely time intensive
  • Setting "File crawler: Include archive view?" has no effect for this moment since I wasn't able to find any advantages of this feature. If anyone still needs this, please let me know.

Although I've been doing extensive tests, bugs may occur so if they do, please let me know and always include example URLs in your reports.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #53  
Old 03.04.2024, 19:48
StormFoo StormFoo is offline
Baby Loader
 
Join Date: Apr 2024
Location: United Kingdom
Posts: 6
Default

Some URLs that previously worked for me now appear as offline. For example, any link in the following folder of this collection:

Spoiler:

**External links are only visible to Support Staff****External links are only visible to Support Staff**


Examples that don't work:
Spoiler:

* **External links are only visible to Support Staff****External links are only visible to Support Staff**
* **External links are only visible to Support Staff****External links are only visible to Support Staff**


However, I am able to download from the root folder or another subfolder in the same collection.

Examples that work:

Spoiler:

* **External links are only visible to Support Staff****External links are only visible to Support Staff**
* **External links are only visible to Support Staff****External links are only visible to Support Staff**

Last edited by StormFoo; 03.04.2024 at 19:52. Reason: Made working example explanation less link-specific for those that can't see the links.
Reply With Quote
  #54  
Old 04.04.2024, 13:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

Fixed.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #55  
Old 06.04.2024, 02:10
StormFoo StormFoo is offline
Baby Loader
 
Join Date: Apr 2024
Location: United Kingdom
Posts: 6
Default

Thanks, links are working again, much appreciated. I did have to update some Packagizer rules that were looking for the URL to contain specific URLs to instead look at the package name, but otherwise behaving as before.
Reply With Quote
  #56  
Old 06.04.2024, 14:17
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,055
Default

@StormFoo: Thanks for the feedback. Could you please elaborate the part with packagizer rules to be updates? maybe some sort of regression we should look into
__________________
JD-Dev & Server-Admin
Reply With Quote
  #57  
Old 08.04.2024, 23:50
Perene Perene is offline
Bandwidth Beast
 
Join Date: Jun 2019
Posts: 132
Default

Please look into archive.org again, since it's adding all files/URLs from a page, not just those you selected it.

Example:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

I asked for the PDF only; JDW is now adding the entire page, located here:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

As you can see from here:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

The PDF is just one of the multiple files available.

Before the last update, it was adding the desired URLs, as a package with the name of the IA's page. This update broke things...
Reply With Quote
  #58  
Old 09.04.2024, 13:17
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

Thanks for your report.
Yes even after thorough testing, updating the core crawl handling of archive.org has indeed broken some stuff but in the long term this should make it work better than before.
Also the handling is not as easy as you might think since we are not(!) accessing the URL which you add but instead we're going via "/metadata/<identifier>" and then looking for the single file/folders, if one is available in the path of the added URL e.g.:
archive.org/download/warbetweenperu00mark/warbetweenperu00mark_abbyy.gz
-> Plugin goes to archive.org/metadata/warbetweenperu00mark
--> Looks for "warbetweenperu00mark_abbyy.gz" and returns only that if it exists. Same for subpaths with files in them.

I've fixed that issue.

For internal reasons, I've also updated the "single desired files" handling again:
If you add an invalid path/file with a valid identifier, all items will be returned e.g.:
archive.org/download/warbetweenperu00mark/invalidFilename.example
-> Returns all items and not a single offline item.

If you do not agree with this change, I'am willing to add a setting for this behavior.

As for the possible encoding problem you've reported a while back:
Please test if you can still reproduce that problem.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #59  
Old 18.04.2024, 06:56
Perene Perene is offline
Bandwidth Beast
 
Join Date: Jun 2019
Posts: 132
Default

@pspzockerscene the plugin broke again... I ask for specific links, it's returning me the thumbnails and all the rest from the page... didn't you fix in your last post?

If you did, it looks like what I said in post #57 has returned...

P.S. I discovered more broken stuff:

I compiled a list of 22 files meant to be uploaded one by one.

17 were showed as "online", since I completed the upload.

The other 5 were not posted yet. So it should have appeared anyway, as "offline".

Problem is, they are not appearing anywhere in the linkgrabber! Not even as offline. And like I said, it's adding stuff not asked, such as the IA's torrent-link (or thumbnails).

In case you are wondering why I need to also list offline links, it's simple:

I create the DLC before the files are even uploaded to IA's server. That's because I also compile a full TXT list of them.

So this is another issue that needs fixing...

+++++++++++
If you need a test page...

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Try this one.

For example, if I wanted the MKV and nothing more:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Look what happens:

https://i.postimg.cc/wgJtWLCF/imagem...-010817423.png

Now, let's try with a link that doesn't even exist:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Did you notice it's not even added as an offline link? I don't need to post another screenshot, because it's that same from PostImg.

Expected result: show 2.1.mkv as "offline", even in link grabber.

++++++++++++++

Another odd behavior:

- If you try to add MP4 or any other file extension, it's added alone:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Proof:

https://i.postimg.cc/pRz1ZcqT/imagem...-011542571.png

Now, if I add any MKV, it's adding all the rest.

Consider that MKVs cannot be added anymore like the other filetypes.

Last edited by Perene; 18.04.2024 at 07:16.
Reply With Quote
  #60  
Old 18.04.2024, 17:14
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

Quote:
Originally Posted by Perene View Post
@pspzockerscene the plugin broke again... I ask for specific links, it's returning me the thumbnails and all the rest from the page... didn't you fix in your last post?
In my last post I tried to explain that detecting single files is not as easy as it seems.
I fixed all problems which I was able to find by testing with the links you had provided in your reply #57.
Please stop being so ungrateful - I'm not implementing bugs on purpose!
You are interpreting too much here - please just report the problems and let me fix them.

Quote:
Originally Posted by Perene View Post
Not even as offline.
[...]
Did you notice it's not even added as an offline link? I don't need to post another screenshot, because it's that same from PostImg.
I've described the new offline handling in my last post.
Please re-read my last post and provide detailed feedback regarding the new offline handling. If you do not like the new handling, please suggest a better handling.

I'll quote myself here:
Quote:
Originally Posted by pspzockerscene View Post
For internal reasons, I've also updated the "single desired files" handling again:

If you add an invalid path/file with a valid identifier, all items will be returned e.g.:
archive.org/download/warbetweenperu00mark/invalidFilename.example
-> Returns all items and not a single offline item.


If you do not agree with this change, I'am willing to add a setting for this behavior.
Moving forward:
Quote:
Originally Posted by Perene View Post
Consider that MKVs cannot be added anymore like the other filetypes.
Fixed - the problem was not related only to .mkv files; again you're just interpreting too much.

Changelog for the next update:
ArchiveOrgCrawler:

- playlist setting auto mode: do not add playlist items if user added '/download/' URL
- single files/paths: fixed detection of single files/paths failing when name of file and name of identifier are the same RE forum 92666
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #61  
Old 18.04.2024, 20:06
Perene Perene is offline
Bandwidth Beast
 
Join Date: Jun 2019
Posts: 132
Default

I didn't say you were adding bugs here. Only that IA's behavior changed, so things may have been broken by the server, not you. And I assumed whenever you updated JDW, you could have forgot to check (?) or add what was working before.

Adding all files contained within a page is a bad idea, if you only ask for a single link such as a MKV file.

If I want a single link to be added, then it's IA's specific page/download/file.mkv, for example.

If I want the entire page and its items, then it's IA's specific page, perhaps don't need to include "download/" in the path, or if you do, leave any specific file/link out of it.

The new "offline handling" is bad;

If I add a Google Drive offline link (or any other server), it will not disappear from link grabber.

The old behavior was better, not just because it allows me to add not yet uploaded files into a DLC beforehand, it also allows me to determine that JDW is working properly.

If I keep adding a link that is supposed to be online (and I don't know it isn't), and JDW is not showing anything (visual confirmation this thing is not there in the server), that is a bad design from your app.

Why? Because it was never supposed to work like this, except for duplicates, that you explained in the past were omitted for obvious reasons - the hash/digital fingerprint was the same, so no use for me to download twice:
https://board.jdownloader.org/showthread.php?t=93792

Or at least allow the user to tick this new method of handling offline links, on/off.

Last edited by Perene; 18.04.2024 at 20:14.
Reply With Quote
  #62  
Old 18.04.2024, 20:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

Quote:
Originally Posted by Perene View Post
so things may have been broken by the server, not you
Nope, all recent reports were bugs which had been caused by me/my changes which I did announce here:
https://board.jdownloader.org/showpo...4&postcount=52
Archive.org hasn't changed anything.
As already explained I did do a lot of tests, unfortunately, not enough.

Quote:
Originally Posted by Perene View Post
If I want a single link to be added, then it's IA's specific page/download/file.mkv, for example.
I know - I understood you all of the time.
As explained, the issue you've experienced might seemed like I just didn't fix it last time but I did and the problem was only happening with the new links you've provided and not with the ones of post #57 anymore.

Quote:
Originally Posted by Perene View Post
The new "offline handling" is bad;
In my reply #58 I did explain why it works the way it does now.

Quote:
Originally Posted by Perene View Post
If I keep adding a link that is supposed to be online (and I don't know it isn't), and JDW is not showing anything (visual confirmation this thing is not there in the server), that is a bad design from your app.
Again it seems like you did not understand the technical background.
Either way - I understand your opinion and I will add an option which will let you get the "old behavior" back.

Also please stop comparing this to Google Drive - the way the archive.org plugin works is totally different.
Please please please at least try to understand what I mean - see post #58.
If you want I can try to explain this to you again but you might just want to use the new setting once it is released and be happy with it.

Quote:
Originally Posted by Perene View Post
Or at least allow the user to tick this new method of handling offline links, on/off.
Yes, soon you will be able to control this behavior via:
Screenshot:
Spoiler:
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 24.04.2024 at 17:01. Reason: Fixed typo
Reply With Quote
  #63  
Old 24.04.2024, 17:00
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

The update with the mentioned archive.org plugin setting has been released.
I've marked this thread as solved.
Feel free to leave feedback.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #64  
Old 18.06.2024, 14:54
StormFoo StormFoo is offline
Baby Loader
 
Join Date: Apr 2024
Location: United Kingdom
Posts: 6
Default

Quote:
Originally Posted by StormFoo View Post
Some URLs that previously worked for me now appear as offline. For example, any link in the following folder of this collection:

Spoiler:

**External links are only visible to Support Staff****External links are only visible to Support Staff**


Examples that don't work:
Spoiler:

* **External links are only visible to Support Staff****External links are only visible to Support Staff**
* **External links are only visible to Support Staff****External links are only visible to Support Staff**


However, I am able to download from the root folder or another subfolder in the same collection.

Examples that work:

Spoiler:

* **External links are only visible to Support Staff****External links are only visible to Support Staff**
* **External links are only visible to Support Staff****External links are only visible to Support Staff**
Sorry to bother you again @pspzockerscene, but I've found that these URLs are misbehaving again - same scenarios as quoted in my previous post.

My guess is that the special characters in the folder names of those that are being marked as offline aren't being matched in the metadata file (see attached image, 12 is in the working examples in the quoted post, 93 is in the offline examples).
Attached Thumbnails
metadata.jpg  

Last edited by StormFoo; 18.06.2024 at 14:54. Reason: Refer to attached thumbnail rather than embedded image that didn't embed
Reply With Quote
  #65  
Old 18.06.2024, 15:33
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

This was also a regression / logic-mistake from that refactoring:
https://board.jdownloader.org/showpo...4&postcount=52

Fixed.

This time it is not related to the encoding.

Quick note about your "Examples that don't work":
The 2nd example is offline.
If you want the plugin to return an offline-status if the single added path-item is not found, go to Settings -> Plugins -> archive.org -> File crawler: What to do when single file/folder-path is not found -> Display added URL as offline
[Otherwise / by default, all items will be added instead.]

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #66  
Old 18.06.2024, 15:42
StormFoo StormFoo is offline
Baby Loader
 
Join Date: Apr 2024
Location: United Kingdom
Posts: 6
Default

Quote:
Originally Posted by pspzockerscene View Post
This was also a regression / logic-mistake from that refactoring:
Fixed.
This time it is not related to the encoding.
Awesome, much appreciated, especially the quick response.

Quote:
Originally Posted by pspzockerscene View Post
Quick note about your "Examples that don't work":
The 2nd example is offline.
If you want the plugin to return an offline-status if the single added path-item is not found, go to Settings -> Plugins -> archive.org -> File crawler: What to do when single file/folder-path is not found -> Display added URL as offline
[Otherwise / by default, all items will be added instead.]
Ah, I forgot to re-test the second example after toggling this setting, my bad. Thanks for highlighting it though, I was unaware of that option until today.
Reply With Quote
  #67  
Old 18.06.2024, 15:46
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

The option is fairly new - it had been added on user-demand.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #68  
Old 08.07.2024, 18:42
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

A bugfix for another regression will be available soon, see:
https://board.jdownloader.org/showthread.php?t=96082
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #69  
Old 09.07.2024, 12:29
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

Archive.org plugin changelog for the next CORE-update:
- re-added detection- and proper handling for list of archive files and the download of single files which are part of an archive RE forum 96082
- Setting "What to do when a single added file/folder-path is not found?" is now also applied for files inside archives
- I haven't added any path- and sophisticated package name handling for files inside archives yet so I'm waiting on user-feedback regarding this#

Reference:
https://board.jdownloader.org/showthread.php?t=96082

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #70  
Old 13.07.2024, 14:43
Dhyanananda Dhyanananda is offline
Modem User
 
Join Date: Jul 2024
Posts: 4
Default

Hey everyone! I'm a decades long user and for the first time I'm having issues that I want to adress.

Namely jDownloader gives me an "server error" whenever I try to download Books from archive.org. What I do is kind of tricksy: I borrow books that cannot be downloaded and then use jDownloader to download the single images of the PDF which I then merge again to get a useable PDF. This has worked for a while now until last week.
PDFs that are free to download from archive I can without issue download from my browser. So their servers are working fine. But my workaround for books that cannot be downloaded doesn't work anymore.

I need those PDFs for my academic work so I want to ask for help here. Is this an issue with jDownloader? Or is archive blocking my workaround somehow?
Attached Thumbnails
jDownloader server error.JPG  
Reply With Quote
  #71  
Old 13.07.2024, 22:15
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,055
Default

@Dhyanananda: Please provide a debug log, see https://support.jdownloader.org/de/k...d-session-logs
Enable Debug Mode, then restart JDownloader and then reproduce the issue and then create log and post shown logID here
__________________
JD-Dev & Server-Admin
Reply With Quote
  #72  
Old 14.07.2024, 13:04
Dhyanananda Dhyanananda is offline
Modem User
 
Join Date: Jul 2024
Posts: 4
Default

Thanks! Heres my LogID: 14.07.24 12.02.24 <--> 14.07.24 12.03.41 jdlog://9904411370661/
Reply With Quote
  #73  
Old 15.07.2024, 00:30
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,055
Default

@Dhyanananda: Thanks for the logs, we will look into it as soon as we find time
__________________
JD-Dev & Server-Admin
Reply With Quote
  #74  
Old 15.07.2024, 15:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

I'am unable to reproduce this problem but here is what I think what happened:
The book you are trying to download was borrow-able some time ago so when you added it to JD, this is what JD did automatically but now it is unavailable for borrowing which you can check yourself on the archive.org website.

If you re-add that same book to JD now, all pages which cannot be borrowed will be displayed as offline straight away.
More info about this topic:
help.archive.org/help/why-are-so-many-books-listed-as-borrow-unavailable-at-the-internet-archive/

TL;DR
Not a JDownloader bug.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #75  
Old 15.07.2024, 16:14
Dhyanananda Dhyanananda is offline
Modem User
 
Join Date: Jul 2024
Posts: 4
Default

Thanks for looking into my issue but your explanation sadly isn't correct. I had this book borrowed on my account! (I even got access to their library for print disabilities.) I just did a clean JD install but to no avail.

What youre desribing is the free preview which works for me exactly like that. I've added Screenshots for what I'm doing: when I added an unborrowed book and tried successfully downloading the preview pages.
Here is the log for that: 15.07.24 15.01.08 <--> 15.07.24 15.03.55 jdlog://6214411370661/

But when I borrow it the results are the same as before. Here also two screenshots: when I added it all seems fine and then when downloading I get the same error.
And here is the error log again: 15.07.24 15.04.13 <--> 15.07.24 15.05.49 jdlog://7214411370661/

like, I hope this isn't an archive-side problem or decision... I'd really appreciate any help. Here is the book-link: **External links are only visible to Support Staff****External links are only visible to Support Staff**. This time I returned it. Although this happens for all the books that I need to borrow.
Attached Thumbnails
Not Borrowed - Added.JPG   Not Borrowed - Working.JPG   Borrowed - Added.JPG   Borrowed - Not Working.JPG  

Last edited by Dhyanananda; 15.07.2024 at 16:16.
Reply With Quote
  #76  
Old 15.07.2024, 16:20
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

Quote:
Originally Posted by Dhyanananda View Post
Thanks for looking into my issue but your explanation sadly isn't correct. I had this book borrowed on my account! (I even got access to their library for print disabilities.)
Please send me your archive.org login credentials via PN so I can look into this problem.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #77  
Old 15.07.2024, 17:59
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

The chain of two separate bugs could have made it possible that:
- For books already borrowed manually by the user before the crawl process, that status wasn't detected correctly
- Books where borrow is needed to view all pages weren't flagged as such, thus the errorhandling which was supposed to detect an expird borrow could fail

Also I was not aware that there are book items which can only be borrowed by a certain group of users.

Wait for the next update, delete all un-downloadable book items, re-add them and try again.
If it keeps failing after said update, please post a fresh log.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #78  
Old 15.07.2024, 18:12
Dhyanananda Dhyanananda is offline
Modem User
 
Join Date: Jul 2024
Posts: 4
Default

Thank you so, so much!! It works again already - I don't know what changed. It works both when I don't manually borrow and when I do. I'm so happy. When I am done with this Thesis I'll make sure to donate to this awesome Projekt.

Last edited by Dhyanananda; 15.07.2024 at 18:27.
Reply With Quote
  #79  
Old 15.07.2024, 18:27
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,238
Default

The mentioned update has been released.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #80  
Old 29.07.2024, 20:01
StormFoo StormFoo is offline
Baby Loader
 
Join Date: Apr 2024
Location: United Kingdom
Posts: 6
Default

Got another oddity, I'm trying to add all the files from **External links are only visible to Support Staff****External links are only visible to Support Staff**, but the link collector is only grabbing Double_Fine_PsychOdyssey_1080p_Episode_01.English.vtt and Double_Fine_PsychOdyssey_1080p_Episode_01.mp4 when there are 33 episodes within the folder (more if you count derivatives such as thumbnails and .ia.mp4 versions of the video files). I can manually add the additional episodes to the package and they appear as online and they appear in the metadata, so I'm not sure why they are being skipped.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 21:46.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.