JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 24.03.2020, 15:03
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 188
Default Request for plugin: Google Drive Directory Index (GoIndex)

Hello,
Can you please add support for websites built using **External links are only visible to Support Staff****External links are only visible to Support Staff**
An example website is **External links are only visible to Support Staff****External links are only visible to Support Staff**
There are plenty of websites using it, and I would like to crawl all the links at once instead of manually downloading one file at a time.
One problem is that there is no common host, i.e. people are free to create their own www. So I am not sure how the linkcrawler will know to use this plugin. But, if it is possible I would love to have a plugin for this.
Reply With Quote
  #2  
Old 24.03.2020, 15:04
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,924
Default

Hi,

I don't see how/what this website does.
It is a wrapper for Google Drive content?
JDownloader already supports normal Google Drive URLs / files / folders / subfolders.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3  
Old 24.03.2020, 15:16
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 188
Default

Some websites do not give out direct gdrive links, instead they provide using some wrapper or index service.
Parsing links from here is not too difficult, just get the href, and remove ?a=view if present, and you get the download link.
But manually creating the folder structure and remove the ?a=view from the url gets annoying after a while, therefore this request for link crawler plugin.
Reply With Quote
  #4  
Old 24.03.2020, 15:19
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 188
Default

>JDownloader already supports normal Google Drive URLs / files / folders / subfolders.

The last time I tried, login failed? And I searched the forums, it said that login has bug.
Is that fixed now.
Reply With Quote
  #5  
Old 24.03.2020, 15:49
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,924
Default

Quote:
Originally Posted by plaintext View Post
Is that fixed now.
No

Please post example URLs and a guide on how to use this website.
Is there any kind of API?
Is there any way to get the real Google Drive URLs?
-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #6  
Old 24.03.2020, 16:07
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 188
Default

> Please post example URLs and a guide on how to use this website.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Folders have class=folder
Files have class=file
(This can be seen by html inspector in browsers.)

Some files (class=file) have a href ending with "?a=view"
For eg. **External links are only visible to Support Staff****External links are only visible to Support Staff**3.txt?a=view
Removing ?a=view gives - "**External links are only visible to Support Staff**3.txt". This is a downloadable url now.

If a file does not have a href ending with "?a=view", then it can be directly downloaded. (e.g. last file in this url)

Some folders can also have password/encryption: **External links are only visible to Support Staff****External links are only visible to Support Staff**

> Is there any kind of API?

AFAIK, no.

> Is there any way to get the real Google Drive URLs?

AFAIK, no

So, the algorithm to download from the page is -

1a. For all links of class File, remove ?a=view from the link
1b. Add to download

2. For all links of class Folder, fetch that link, and use these algorithms recursively.

The only issue as I said is that there is no single host, and everybody is free to use their own host. So some heuristic would be required to detect when this plugin is used (because the website is hosted on cloudflare workers, so they have url ending with worker.dev typically. not a great heuristic though). or maybe a manual option to invoke the plugin for some url.
Reply With Quote
  #7  
Old 24.03.2020, 16:15
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 188
Default

I don't know how jdownloader handles subdirectories, because I never used it to download such links till now, but if you choose to implement this plugin, please try to make it output files in proper subdirectories.
Reply With Quote
  #8  
Old 24.03.2020, 17:05
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,924
Default

Ticket:


I'll check it out once I find the time ...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #9  
Old 24.03.2020, 17:11
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 188
Default

Okay, thank you.
Reply With Quote
  #10  
Old 30.03.2020, 18:23
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,924
Default

I've added a simple crawler for this website for now.

Please provide the following types of test-URLs so that I can add more of your requested functionality:
- Folder containing subfolders AND files
- Password protected URLs with the corresponding password

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #11  
Old 14.04.2020, 13:17
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 188
Default

Sorry for late reply. I found a link containing subfolders and files - **External links are only visible to Support Staff****External links are only visible to Support Staff**
I could not find any link with password, so leave it for now. If i find it, i will reply here.
Reply With Quote
  #12  
Old 14.04.2020, 15:03
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 188
Default

I dunno why my previous post is not showing up, so I am posting again -

> Folder containing subfolders AND files

**External links are only visible to Support Staff****External links are only visible to Support Staff**[Book]Freelancing%20and%20Marketing%20for%20Web%20Developers%20+%20Web%20Designers%20(The%20Complete%20Bun dle)/

This url contains a folder, and inside that folder are subfolders and files both.

> Password protected URLs with the corresponding password

I could not find any such links. Maybe leave password-protected links for now.
Reply With Quote
  #13  
Old 15.04.2020, 18:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,924
Default

Quote:
Originally Posted by plaintext View Post
I dunno why my previous post is not showing up, so I am posting again -
Yes because the account you are using is publicly accessible, posts of that account need to be approved manually.

Your URLs are offline and none of them seems to lead to the website I previously implemented e.g. "index.gd.workers.dev".

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #14  
Old 17.04.2020, 17:38
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 188
Default

> Your URLs are offline

Use this link for now -

**External links are only visible to Support Staff****External links are only visible to Support Staff**

> none of them seems to lead to the website I previously implemented e.g. "index.gd.workers.dev"

umm.. yeah. I said in the beginning that GoIndex is a self hosted solution, so the url host might be different but the page structure is the same, so the crawler should work.

Last edited by raztoki; 18.04.2020 at 07:41.
Reply With Quote
  #15  
Old 18.04.2020, 09:59
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 188
Default

Also, here is a password protected link -
link -
**External links are only visible to Support Staff****External links are only visible to Support Staff**
password -
see edit history

Last edited by raztoki; 18.04.2020 at 12:20.
Reply With Quote
  #16  
Old 20.04.2020, 15:39
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,924
Default

I've added simple subfolder support and support for all "workers.dev" subdomains for the next update.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #17  
Old 20.04.2020, 19:06
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 188
Default

Thank you. But it does not work properly. The parsed urls are wrong because they contain an ending slash in the url so file download fails with File Not Found. This happens with every url I tried.

Also I am getting files with "unknownFilename" if i crawl this link for eg. - **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #18  
Old 21.04.2020, 14:01
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,924
Default

Yeah I see, sorry.
Only folders/subfdolders have to end with "/".
Fixed with the next update.
I've also added subfolder-support meaning you should get the same folder-structure you got on that website also after downloading via JD.
Please keep in mind that this will lead to a LOT of packages as our package system does not allow sub-packages with e.g. other download paths.
If you do not wish to have this feature, you can turn it off here:
Settings --> Packagizer --> "Adopt folder structure".

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 11:00.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.