JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 03.11.2017, 04:01
blueguy6 blueguy6 is offline
Vacuum Cleaner
 
Join Date: Nov 2017
Posts: 17
Default Folderwatch, Folder output, and Preserving Online Folder structure

I have a crawljob like this:

autoStart=FALSE
extractAfterDownload=FALSE
downloadFolder=C:\DOWNLOADFOLDER\
text=DOWNLOADURL

It works fine, but it doesn't preserve the online folder structure.

If I add overwritePackagizerEnabled=FALSE, then it downloads to the default download folder listed in general settings, even though packagizer has no rules for download directories (apart from "Create subfolders on package name" and "Adopt folder structure", the latter of which I want).

So basically if the download link leads to a folder with:

folder1\folder1a\file.ext
folder1\file2.ext
folder2\file3.ext
file4.ext

I want it to output to:

C:\DOWNLOADFOLDER\folder1\folder1a\file.ext
C:\DOWNLOADFOLDER\folder1\file2.ext
C:\DOWNLOADFOLDER\folder2\file3.ext
C:\DOWNLOADFOLDER\file4.ext

with C:\DOWNLOADFOLDER\ being specified in the crawljob file.

I tried appending <jd:subfolderbyplugin> to downloadFolder=C:\DOWNLOADFOLDER\ but jdownloader didn't parse it correctly, thinking it wanted a subfolder called <jd:subfolderbyplugin>, which is of course an invalid file name.

Also writing a documentation might be helpful. It's a little annoying trying to learn this program by googling and searching through threads. Just doing a google search for "setBeforePackagizerEnabled" comes up with 0 results and same with a forum search, and I don't know what this option does (it doesn't seem to do anything).
Reply With Quote
  #2  
Old 03.11.2017, 12:35
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Do you have example links? Because only very few plugins make use of the subfolderbyplugin feature, for example mega folders.
Please provide example links, so we can better help you.


I just added 'setBeforePackagizerEnabled' recently to help a customer with his crawljobs and packagizer rules. That's why you do not find anything about it.

I'm still working on a good solution to this
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 03.11.2017 at 15:36.
Reply With Quote
  #3  
Old 03.11.2017, 18:56
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Added support with next core update

You've got to create a PackagizerRule like this
Code:
match always or whatever you need
set downloadDirectory to <jd:append>/<jd:subfolderbyplugin>
Your crawljob must use
overwritePackagizerEnabled=false

the new variable <jd:append> from helps to append values and not replace/set them

I've just tested with mega folder links and it is working fine for me
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 03.11.2017, 18:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Quote:
Originally Posted by blueguy6 View Post
If I add overwritePackagizerEnabled=FALSE, then it downloads to the default download folder listed in general settings, even though packagizer has no rules for download directories (apart from "Create subfolders on package name" and "Adopt folder structure", the latter of which I want).
Relative Rules replace/set the download folder and because of that the *default download folder* will be used.

That's why the new variable <jd:append> got added, so you can either set or append values
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 03.11.2017, 22:30
blueguy6 blueguy6 is offline
Vacuum Cleaner
 
Join Date: Nov 2017
Posts: 17
Default

Quote:
Originally Posted by Jiaz View Post
Added support with next core update

You've got to create a PackagizerRule like this
Code:
match always or whatever you need
set downloadDirectory to <jd:append>/<jd:subfolderbyplugin>
Your crawljob must use
overwritePackagizerEnabled=false

the new variable <jd:append> from helps to append values and not replace/set them

I've just tested with mega folder links and it is working fine for me
Hi thanks, I'm not sure if the feature has been fully implemented but Jdownloader seems to recognize it.

I'm dealing with a variety of hosts (mediafire, mega, dropbox, off the top of my head) so I didn't give a link.

It seems that the implementation works but perhaps the plugins do not.

For example when using this link:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

It does not output the directory recursively, but will instead take:

subfolder\subfolder2\file1
subfolder\file2

to:

C:\download\subfolder2\file1
C:\download\subfolder\file2

Instead of the desirable:

C:\download\subfolder\subfolder2\file1
C:\download\subfolder\file2

As well this link:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

It treats each subfolder as a separate package, rather than a subfolder.

Although not related I also found an error with the mediafire plugin (at least that's where I've noticed this issue most) that if the file has an "[" (ignore quotations) in the name (off the top of my head the "[" not being closed but I'm not sure if that affects it), it will break as jdownloader will append a bunch of data relating to the download into the filename, requiring the user to manually rename the file before it can properly download.

Unfortunately I don't quite have time to pin down a link for the [ error but I'll try to get it to you later if you need it.

Also thanks for all your work on this program.
Reply With Quote
  #6  
Old 04.11.2017, 15:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

I will check those links on Monday. FolderStructure support must be implemented in Plugins. Most plugins do only provide package naming support.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 04.11.2017, 15:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

In case you've got more example links, please send them to support@jdownloader.org
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 06.11.2017, 15:57
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Quote:
Originally Posted by blueguy6 View Post
Although not related I also found an error with the mediafire plugin (at least that's where I've noticed this issue most) that if the file has an "[" (ignore quotations) in the name (off the top of my head the "[" not being closed but I'm not sure if that affects it), it will break as jdownloader will append a bunch of data relating to the download into the filename, requiring the user to manually rename the file before it can properly download.
.
Can you please provide example links?! I've tested some links with [ in filename and did not encounter any issues
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 06.11.2017, 16:34
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

I've added folder structure support to mediafire and dropbox plugins, wait for next plugin update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 06.11.2017, 20:04
blueguy6 blueguy6 is offline
Vacuum Cleaner
 
Join Date: Nov 2017
Posts: 17
Default

Quote:
Originally Posted by Jiaz View Post
Can you please provide example links?! I've tested some links with [ in filename and did not encounter any issues
Hi sorry here are some links:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #11  
Old 07.11.2017, 16:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Thanks! Will be fixed with next update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 17.11.2017, 00:48
blueguy6 blueguy6 is offline
Vacuum Cleaner
 
Join Date: Nov 2017
Posts: 17
Default

Hi thanks for everything, one more request, keeping the package name to the first folder name.

For example here:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

JDownloader does not keep the package name the same for the subfolders, it instead uses a different package name for the subfolders when specified with <jd:packagename>.

I believe this has to do with how Jdownloader discovers subfolders, and I'm not sure how easy it would be to fix, but it would be appreciated.

In short I would like it to be:

<jd:append>\HYRULE WARRIORS LINK PACK\<jd:subfolderbyplugin>

Last edited by blueguy6; 17.11.2017 at 00:51.
Reply With Quote
  #13  
Old 26.11.2017, 01:42
blueguy6 blueguy6 is offline
Vacuum Cleaner
 
Join Date: Nov 2017
Posts: 17
Default

bump
Reply With Quote
  #14  
Old 27.11.2017, 18:07
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

packagename and subfolders are different things.
Each Dropbox folder has a packagename and the subfolder just specifies the relative path of that package. The root does have correct packagename but no subfolder because it is root (start of the adding process)

I've added special support in DropBox Plugin for this. After next update , Settings-Plugins, search for dropbox. There will be two entries in list. One for host plugin, one for crawler. In Crawling plugin you can enable *Include Root Subfolder*.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 04.12.2017, 07:51
blueguy6 blueguy6 is offline
Vacuum Cleaner
 
Join Date: Nov 2017
Posts: 17
Default

Quote:
Originally Posted by Jiaz View Post
packagename and subfolders are different things.
Each Dropbox folder has a packagename and the subfolder just specifies the relative path of that package. The root does have correct packagename but no subfolder because it is root (start of the adding process)

I've added special support in DropBox Plugin for this. After next update , Settings-Plugins, search for dropbox. There will be two entries in list. One for host plugin, one for crawler. In Crawling plugin you can enable *Include Root Subfolder*.
Thanks, I'm wondering if you can add it to mediafire, mega, google drive/docs (drive.google.com and docs.google.com) and onedrive as well.

For mediafire it seems to work unless if the main folder has no files in it, for example compare this link which works in that matter:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

and this one which does not:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

From my testing with mega it seems to implement the root folder as desired regardless of whether or not there is a file in it, so I do not think any change needs to be made. (example links that work regardless, with no files in root folder: **External links are only visible to Support Staff****External links are only visible to Support Staff**, with files in root folder: **External links are only visible to Support Staff****External links are only visible to Support Staff**)

Looking at Google drive, it seems to not generate the root folder, here are links:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Onedrive seems to have a more extreme problem in which absolutely no folder structure is preserved, see:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**


All of the links provided were added into link grabber with the only packagizer rule affecting directories being an a rule affecting all links that sets the download folder to <jd:append>/<jd:subfolderbyplugin>.

Thanks again for your efforts in making this invaluable tool.
Reply With Quote
  #16  
Old 04.12.2017, 15:34
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

So mega plugin with custom setting working fine as expected now?
Or any change wishes?
I've created a ticket to implement same setting for provided example links as well
__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 04.12.2017, 22:19
blueguy6 blueguy6 is offline
Vacuum Cleaner
 
Join Date: Nov 2017
Posts: 17
Default

Quote:
Originally Posted by Jiaz View Post
So mega plugin with custom setting working fine as expected now?
Or any change wishes?
I've created a ticket to implement same setting for provided example links as well
The dropbox plugin with the added "Include Root Subfolder" option does work as expected regardless of whether or not there is a file in the root folder or only subfolders (so perfect, no change needed), the Mega plugin already works as expected with no custom settings (except packagizer set to <jd:append>/<jd:subfolderbyplugin>, so perfect, no change needed).

There are no changes I can think of except the already stated to onedrive, mediafire, and google drive.
Reply With Quote
  #18  
Old 05.12.2017, 13:09
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Thanks for the feedback. I will update the other plugins as soon as I find time
__________________
JD-Dev & Server-Admin
Reply With Quote
  #19  
Old 25.01.2019, 17:18
blueguy6 blueguy6 is offline
Vacuum Cleaner
 
Join Date: Nov 2017
Posts: 17
Default

Resubmitting this feature request with the issues noted in post 15 (the links are still valid)

https://board.jdownloader.org/showpo...3&postcount=15

1. Mediafire does not generate folder structure if there are no files in the root folder (ie the root has only subfolders), otherwise it does

2. Onedrive does not generate folder structure at all

3. Google drive does not generate the root folder name

#1 and 2 I would really like to see fixed especially.
Reply With Quote
  #20  
Old 25.01.2019, 17:36
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

2.) already fixed with latest update, please test
__________________
JD-Dev & Server-Admin
Reply With Quote
  #21  
Old 25.01.2019, 17:43
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Just to be sure what you exactly want to happen
You want JDownloader to preserve folder name/structure even when manually adding a single folder.

currently it was meant to only work for subfolders and not a folder.
adding a folder link -> adding the links within a package
/test/test.txt -> /test.txt will result

adding a folder with subfolders -> preserve the subfolder structure
/test/test.txt -> /test.txt
/test/test2/test2.txt -> /test2/text2.txt

you want
/test/test.txt -> /test/test.txt
/test/test2/test2.txt -> /test/test2/text2.txt

?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #22  
Old 25.01.2019, 18:03
blueguy6 blueguy6 is offline
Vacuum Cleaner
 
Join Date: Nov 2017
Posts: 17
Default

onedrive does indeed appear to be fixed assuming it's not like mediafire (I can't seem to find a link with only subfolders in root to test).

Quote:
Originally Posted by Jiaz View Post
Just to be sure what you exactly want to happen
You want JDownloader to preserve folder name/structure even when manually adding a single folder.

currently it was meant to only work for subfolders and not a folder.
adding a folder link -> adding the links within a package
/test/test.txt -> /test.txt will result

adding a folder with subfolders -> preserve the subfolder structure
/test/test.txt -> /test.txt
/test/test2/test2.txt -> /test2/text2.txt

you want
/test/test.txt -> /test/test.txt
/test/test2/test2.txt -> /test/test2/text2.txt

?
Correct

Edit: Looks like dropbox has a bug, see here:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

It seems to only grab the first subfolder and not the other subfolders (which can be seen at the very bottom in the browser window).

It also appears that dropbox has a specific prompt for adding subfolders that comes up after adding link, but this only sometimes comes up. An option to make grabbing subfolders default behavior (since it is default for other plugins) would be nice.

Last edited by blueguy6; 25.01.2019 at 18:13.
Reply With Quote
  #23  
Old 25.01.2019, 19:42
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Can you provide more example links please, then I will look at them one by one.

Would it be okay if we add this as optional in plugin settings as changing default behaviour would be no good.
so default:
/test/test.txt -> /test.txt
optional:
/test/test.txt -> /test/test.txt
okay?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #24  
Old 25.01.2019, 23:24
blueguy6 blueguy6 is offline
Vacuum Cleaner
 
Join Date: Nov 2017
Posts: 17
Default

Quote:
Originally Posted by Jiaz View Post
Can you provide more example links please, then I will look at them one by one.

Would it be okay if we add this as optional in plugin settings as changing default behaviour would be no good.
so default:
/test/test.txt -> /test.txt
optional:
/test/test.txt -> /test/test.txt
okay?
Referring to keeping the root folder I'm pretty sure it's the default for mega, dropbox, and mediafire plugins using <jd:append>/<jd:subfolderbyplugin>, whereas it's not for google drive and onedrive.
Switching google drive and onedrive to function like the rest of the plugins seems easier and 'better' to me, but if you want to change all of them to work something like <jd:append>/<jd:root>/<jd:subfolderbyplugin> I'm fine with that.

For the dropbox plugin only getting some of the subfolders, you can check any of these links:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

For mediafire not generating the folder structure if there is no file in root:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
vs when it works perfectly (including generating the root folder) when there is a file in root:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

For google drive not adding the root folder, any of these:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

For onedrive not adding root folder:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #25  
Old 20.08.2021, 14:52
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

I know it's over 2.5 years later but I've just gone through old tickets and closed the one related to this thread.
The mentioned extra setting for dropbox.com has been removed so now our default handling for subfolder-structures of cloud services should be just fine.
You can find more information about this HERE.
All of our crawlers are designed in a way t try to find- and use/set the current/root folder correctly so that the structure you get afterwards resembles 100% the one from that cloud service.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 13:49.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.