JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 26.04.2017, 02:59
FlatApio
Guest
 
Posts: n/a
Default False-positive mirror downloads and Url Display Order

Hello

First of all, let me say that I've been using JDownloader 2 for several years now and I can no longer imagine not having it installed at any OS I'm running. Today however, I'd like to report - I believe - a bug I stumbled upon just recently.

All links below lead to explicitly mature content and are given only as example.

---
Gist of the problem:
1. Downloading whole gallery off from xHamster incorrectly flags some files as mirrors due to URL display order.
2. Advanced setting "GeneralSettings: Url Order" doesn't work.

---
I'm running Windows 7 Pro x64
Build date of JD2: Tue Apr 25 02:19:30 CEST 2017
Source Revisions:
Core: #36667
Launcher: #3836
AppWork Utilities: #2884
Browser: #36630
Updater: #565

The matter is about downloading whole galleries of images from xhamster.com. Sometimes JDownloader 2 wrongly marks some of the images as "mirrors" of other files and refuses to download them. Such files are marked with "mirror directhttp is loading"(sic) while downloading is pending and then "Finished(Mirror)" once the download is complete. This however leads to missing some of the files after the downloading is done since the presumably "mirror" files are actually unique and different from their respective "copies"

This happens because JDownloader 2 validates files by their filename and "Download from" string. In case of xHamster multiple files in single gallery can have identical filename. Now because JD2 uses URL of the whole gallery rather than specific image file for validation it leads to false-positive collisions.

An example, this gallery has two pages of images:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

On page one, you can find the following entry:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Image displayed under this entry is:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

On page two, you can find the following entry
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Image displayed under this entry is:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

As you can see, both files have same filename, but different direct URL. One of them will fail to download because they will be recognized as both being downloaded from "**External links are only visible to Support Staff**.

---
I've tried to add another custom Packagizer rule to rename files depending on their download url. This however didn't seem to work either due to my lack of skill or the fact that JD2 doesn't differentiate between those false "mirror" files.

Next I attempted to see if this flagging can be overridden in some way. While messing with settings, I noticed that clicking any file in Downloader/Linkgrabber lets me select:

Properties -> Show Download URLs -> Double-click on item in "Download from" column.

This displayed both "Container" link (the "bad" link to whole gallery) and "Data" link (the direct link to this particular image). Opening this also displayed friendly message that the JD2 can let me control which URL is displayed and used for copying. Hoping that this will solve my problem ("Data" type link probably would let me make valid Packagizer rule) I navigated to given option.

Settings -> Advanced -> GeneralSettings: Url Display Order

Only to find out it doesn't exist, but there is a very similar one under name "GeneralSettings: Url Order". The field by default was set to "null". The field description informs that valid values are: CONTAINER, CONTENT, CUSTOM, ORIGIN, etc. None of those values are accepted in the given field and rolls back to null value. Error message reads (after entering "CONTENT" as value):

Code:
'CONTENT' is not a valid 'UrlDisplayEntry[]'

I tried putting in the very same value embraced with single and then double quotes then braces and finally formatted as json. None of this was accepted by the field.

At this point I've hit a stopping point and I have no idea what else could I do to resolve this issue.

Thanks in advance for any support you could provide. Please let me know if I can/should provide any additional info.
Reply With Quote
  #2  
Old 26.04.2017, 11:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

URL Order is only used for GUI/UserInterface. It has no influence on Mirror Detection.
Check Settings-Advanced Settings-search for mirror
Either enable GeneralSettings.forcemirrordetectionfilesizecheck
or set GeneralSettings.mirrordetectiondecision to SAFE
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 26.04.2017, 11:11
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by FlatApio View Post
This happens because JDownloader 2 validates files by their filename and "Download from" string. )
I'm sorry but it doesn't
It uses Filename/FileSize/FileHash where available.
See my post above.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 26.04.2017, 11:15
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by FlatApio View Post
Settings -> Advanced -> GeneralSettings: Url Display Order

Only to find out it doesn't exist, but there is a very similar one under name "GeneralSettings: Url Order". The field by default was set to "null". The field description informs that valid values are: CONTAINER, CONTENT, CUSTOM, ORIGIN, etc. None of those values are accepted in the given field and rolls back to null value. Error message reads (after entering "CONTENT" as value):

Code:
'CONTENT' is not a valid 'UrlDisplayEntry[]'

Syntax is JSON and the value expected is an Array of UrlDisplayEntry Objects

Code:
[ {
  "type" : "ORIGIN",
  "enabled" : true
}, {
  "type" : "REFERRER",
  "enabled" : false
}, {
  "type" : "CONTENT",
  "enabled" : false
}, {
  "type" : "CONTAINER",
  "enabled" : false
}, {
  "type" : "CUSTOM",
  "enabled" : false
} ]
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 26.04.2017, 19:40
FlatApio
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by Jiaz View Post
URL Order is only used for GUI/UserInterface. It has no influence on Mirror Detection.
Check Settings-Advanced Settings-search for mirror
Either enable GeneralSettings.forcemirrordetectionfilesizecheck
or set GeneralSettings.mirrordetectiondecision to SAFE
Ah, I see. My mistake then. I assumed something that was clearly wrong. Thank you dearly for clearing this one up.
The advanced settings you listed work PERFECTLY for my needs and the core problem I had was basically resolved with switching either one of them. Thanks a lot again.

Quote:
Originally Posted by Jiaz View Post
I'm sorry but it doesn't
It uses Filename/FileSize/FileHash where available.
See my post above.
Understood. Again, I assumed something solely of what I was able to work with. This is valuable knowledge too. It might be useful in case I stumble upon another of one of such unusual cases.

Quote:
Originally Posted by Jiaz View Post
Syntax is JSON and the value expected is an Array of UrlDisplayEntry Objects
(..)
Ah, the example you have given works perfectly. I admit I never tried to input whole array as an option. Description of that advanced setting lead me to believe it was "single" value only field. But with this, this advanced option works just fine as well. Thank you once more.
Reply With Quote
  #6  
Old 27.04.2017, 10:32
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Thanks for the feedback Glad I could be of help to you
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 06.02.2019, 08:59
botmtl botmtl is offline
DSL User
 
Join Date: Apr 2015
Location: Montreal
Posts: 39
Default

Quote:
Originally Posted by Jiaz View Post
Syntax is JSON and the value expected is an Array of UrlDisplayEntry Objects

Code:
[ {
  "type" : "ORIGIN",
  "enabled" : true
}, {
  "type" : "REFERRER",
  "enabled" : false
}, {
  "type" : "CONTENT",
  "enabled" : false
}, {
  "type" : "CONTAINER",
  "enabled" : false
}, {
  "type" : "CUSTOM",
  "enabled" : false
} ]
To anyone having issues filling this in, copying and pasting the above won't work. The only way I have found to change this setting is to use a string containing no blanks or carriage returns whatsoever.

Like this:

Code:
[{"type":"CONTENT","enabled":true},{"type":"REFERRER","enabled":false},{"type":"ORIGIN","enabled":false},{"type":"CONTAINER","enabled":false},{"type":"CUSTOM","enabled":false}]
Reply With Quote
  #8  
Old 06.02.2019, 12:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@botmtl: Thanks for the info. Will be fixed with next update!
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 01:40.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.