View Single Post
  #1  
Old 26.04.2017, 02:59
FlatApio FlatApio is offline
Modem User
 
Join Date: Apr 2017
Posts: 2
Default False-positive mirror downloads and Url Display Order

Hello

First of all, let me say that I've been using JDownloader 2 for several years now and I can no longer imagine not having it installed at any OS I'm running. Today however, I'd like to report - I believe - a bug I stumbled upon just recently.

All links below lead to explicitly mature content and are given only as example.

---
Gist of the problem:
1. Downloading whole gallery off from xHamster incorrectly flags some files as mirrors due to URL display order.
2. Advanced setting "GeneralSettings: Url Order" doesn't work.

---
I'm running Windows 7 Pro x64
Build date of JD2: Tue Apr 25 02:19:30 CEST 2017
Source Revisions:
Core: #36667
Launcher: #3836
AppWork Utilities: #2884
Browser: #36630
Updater: #565

The matter is about downloading whole galleries of images from xhamster.com. Sometimes JDownloader 2 wrongly marks some of the images as "mirrors" of other files and refuses to download them. Such files are marked with "mirror directhttp is loading"(sic) while downloading is pending and then "Finished(Mirror)" once the download is complete. This however leads to missing some of the files after the downloading is done since the presumably "mirror" files are actually unique and different from their respective "copies"

This happens because JDownloader 2 validates files by their filename and "Download from" string. In case of xHamster multiple files in single gallery can have identical filename. Now because JD2 uses URL of the whole gallery rather than specific image file for validation it leads to false-positive collisions.

An example, this gallery has two pages of images:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

On page one, you can find the following entry:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Image displayed under this entry is:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

On page two, you can find the following entry
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Image displayed under this entry is:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

As you can see, both files have same filename, but different direct URL. One of them will fail to download because they will be recognized as both being downloaded from "**External links are only visible to Support Staff**.

---
I've tried to add another custom Packagizer rule to rename files depending on their download url. This however didn't seem to work either due to my lack of skill or the fact that JD2 doesn't differentiate between those false "mirror" files.

Next I attempted to see if this flagging can be overridden in some way. While messing with settings, I noticed that clicking any file in Downloader/Linkgrabber lets me select:

Properties -> Show Download URLs -> Double-click on item in "Download from" column.

This displayed both "Container" link (the "bad" link to whole gallery) and "Data" link (the direct link to this particular image). Opening this also displayed friendly message that the JD2 can let me control which URL is displayed and used for copying. Hoping that this will solve my problem ("Data" type link probably would let me make valid Packagizer rule) I navigated to given option.

Settings -> Advanced -> GeneralSettings: Url Display Order

Only to find out it doesn't exist, but there is a very similar one under name "GeneralSettings: Url Order". The field by default was set to "null". The field description informs that valid values are: CONTAINER, CONTENT, CUSTOM, ORIGIN, etc. None of those values are accepted in the given field and rolls back to null value. Error message reads (after entering "CONTENT" as value):

Code:
'CONTENT' is not a valid 'UrlDisplayEntry[]'

I tried putting in the very same value embraced with single and then double quotes then braces and finally formatted as json. None of this was accepted by the field.

At this point I've hit a stopping point and I have no idea what else could I do to resolve this issue.

Thanks in advance for any support you could provide. Please let me know if I can/should provide any additional info.
Reply With Quote