[Solved] Premiumize Torrent crawler skips certain filetypes

I hope this is the right forum for this - apologizes if I got it wrong.

I discovered a very odd thing that must have started a few days ago. All of a sudden, the Link Crawler for Premiumize Torrent downloads started to skip a list of specific file types / extentions.

This includes many checksum and info types, like .ffp .md5 .log, but it also includes other file types like .ape

Any files by these extensions are simply not added to the "Link Grabber" list, although they exist and download fine if done/dragged manually.

It looks like a filter on the filenames, but I can't find anything in the prefs (neither the GUI nor the Advances settings), especially there is nothing under the "packagizer" tab. I did not change any settings at all, so whatever is different now, must have changed by itself recently.

I tried two different installs of JD2 on two different systems, both the same - so it doesn't look like a single broken install.

Could you provide me with any hint where to look for this "filter" setting?

[pspzockerscene]

Hi
We haven't changed a single line of code of that plugin for over a month.
Maybe it is a premiumize.me serverside issue - if it worked before that is probably the case.

GreeZ psp
EDIT

Please contact their support as well, here:

**External links are only visible to Support Staff**

We're also in contact with them

Thanks, I'll do.

However - is there a setting somewhere within JD2 where I can/could make it to skip certain filetypes? I discovered a strange (changed) setting under Advanced > LinkCrawler: Link Crawler Rules that I never changed myself (a complicated set of entries in a [ { ... } ] wrap. Resetting it changed it back to an empty [ ]... I saved the original entry but can't find any differences in its behaviour.

However, I have no idea who/what/when/why changed this entry, so there may be more of that...

OK, one more observation...

While (of course) from an amateur perspective and not from the developer's knowledge, I must say that I have doubts now that the problem is (entirely) within Premiumize.

When I drag the direct download link to the final file from the Premiumize detail page, the files *also* get ignored by LinkGrabber. However, the download paths are absolutely identical up until the filename and extention.

So, for example, I have two links:

**External links are only visible to Support Staff**
**External links are only visible to Support Staff**

The "some codes and paths here" bits are absolutely identical, and the files do both exist and download fine in all browsers.

However, when I drag the first one, the file info.txt gets added to the LinkGrabber. But if I drag the second link into JD2, file info.ffp just gets ignored.

And this happens with all torrents (no matter which and where) with all of the filetypes that I listed above plus some more that I discovered now (.m3u, .accurip, .st5).

I find it intesting that the majority of "banned" files seem to be checksums and kind of low-level info files. The only major exception is .ape, which are large binary files.

Second oberservation - if I use a download plug-in in Firefox that re-directs the download of info.ffp to JD2, it suddenly works! So JD2 perfectly grabs the files as soon as they won't get processed as a "premiumize torrent" thing.

The downside is, that JD2 will then not respect the folder system and downloads all files in a flat directory. So this is not really a workaround.

[Jiaz]

Please provide example links. Easiest would be to send us your premiumize logins and the torrent url, then we can check/fix it.
I've updated the plugin to also handle /dl/ urls. It only had support for /torrentdl/ maybe that caused it.
Please wait for next plugin update and provide feedback.

Thanks a lot for your reply. The short & good news is that it perfectly works now again, all of the "skipped" filetypes are loaded in LinkGrabber again.

The slightly longer & confusing news is that the reason of the repair seems to be pretty obscure. *All* of the links on the download page have the same structure, like this:

**External links are only visible to Support Staff**

It's all exactly like this, except for filename.txt, filename.log, filename.flac, filename.ape

If the /dl/ bit would make the difference (being /torrentdl/ before) I would understand that none of the files would have been recognized any more. But, in fact, up until yesterday filename.txt and filename.flac were added, but filename.log and filename.ape where skipped. This is odd.

Anyway, the problem is gone as of this minute right now, so I can't provide you a link that would show the failure.

However, I started a ticket at Premiumize as well and they told me they "will fix it"... maybe it was coinicidence and they actually fixed something in the API in the same moment when you did these edits...

I will let them know about the progress and will ask them to provide you all access if the problems should come back. After all Premiumize massively announces JDownloader2 as their sole recommended download tool and even has step-by-step guides of how to use it. So it's their turn to offer support in return, I'd say...

[Jiaz]

I've added support for /dl/. Before they were picked up by DirectHTTP Plugin and that plugin will only process known file extensions.

I'm sorry to re-open the thread, but actually this old problem got re-introduced in the latest version.

After the massive change of their API recently, the plugin now works again, but it also started to skip some of the filetypes again, like ".ape" for example. Common files like .jpg or .txt are found by the LinkCrawler, but a couple of other files get ignored.

There also is another problem - although the LinkCrawler now re-builds the folder structure, different to the old behaviour it fails to use the Torrent name (main folder) on top. So it always merges all downloaded Torrent data into the same default folder, instead of placing it in one folder (each, with the name of the Torrent) in the default folder.

Example, if there is a Torrent1 with items aaa and bbb, and a Torrent2 with items xxx and yyy, then in the old days JD2 downloaded into:

default_folder/Torrent1/aaa/
default_folder/Torrent1/bbb/
default_folder/Torrent2/xxx/
default_folder/Torrent2/yyy/

Now it downloads into:

default_folder/aaa/
default_folder/bbb/
default_folder/xxx/
default_folder/yyy/

If it's possible to fix this, thanks in advance. As for testing purposes - I think the Premiumize owners do support JD2 and will assist, since they recommend using JD2.

[Jiaz]

Plugin does not have any filtering in place, so we need example links to check those missing files. Also no real changes despite support for single file support
It's more likely that they have added new domain and because of missing support, the direct download links are processed by generic plugin that only handles known file extensions

Please send login with example torrents to support@jdownloader.org so we can check/fix it

Thanks for the reply, I have contacted their support and asked them to investigate it first on their side.

[Jiaz]

It would be much easier and faster if you would provide logins, then we can easily find out if it's caused by changes on their side or issues in plugin

[pspzockerscene]

@rips666
Your explanation is not entirely understandable to me but I guess "xxx" and "yyy" are folders as well.
Anyways - I've tested this case and was able to reproduce your issue.
It will be fixed via the next update

GreeZ psp

Originally Posted by pspzockerscene

@rips666
Your explanation is not entirely understandable to me but I guess "xxx" and "yyy" are folders as well.

Yeah... sorry. I should have said that, but I thought "default_folder/aaa/" (with the "/" ending) made it clear.

But the problem actually affected single file items as well - if you have 10 different torrents, then all of their entire contents of the first directory level (each) are all thrown and merged into one download directory - unless the contents get separated again by new folders with the 10 different torrent names, as in the old days.

Thanks for the fix, it now work (almost) perfect again...

There's only one slight glitch left. It's not essential, but if it could be improved as well then it'll be seriously perfect :-)

The current version now "reconstructs" all subfolders (including all nested ones) and their contents, but it doesn't do with single files on the first level.

I try to explain with a (hopefully better) example.

If there is a torrent named "sampletorrent" with one file plus one subfolder "aaa" with 2 more files, like this:

/file1.dat
/aaa/file2.dat
/aaa/file3.dat

Then JD2 now downloads into:

targetdirectory/file1.dat
targetdirectory/sampletorrent/aaa/file2.dat
targetdirectory/sampletorrent/aaa/file3.dat

But file1.dat should better be downloaded into:

targetdirectory/sampletorrent/file1.dat

This can be manually fixed by selecting all of those "stray" files and simply turn on "<jd:packagename>", since they all have the torrent name as their packagenames.

But it would be nice if this could be setup automatically, of course.

(additional note: this can't be achieved by adding "<jd:packagename>" to the targetdirectory in the prefs. Doing so will "fix" for the stray files, but then all of the others would be downloaded into something like "targetdirectory/<jd:packagename>/sampletorrent/aaa/file3.dat" which resolves to "targetdirectory/aaa/sampletorrent/aaa/file3.dat"! And that of course spoils it totally...)

[Jiaz]

Can you send logins again to support@jdownloader.org
And please tell me what torrent/link you are adding so I can reproduce the issue and fix it