[plaintext]

Hello,
Can you please add support for websites built using **External links are only visible to Support Staff****External links are only visible to Support Staff**
An example website is **External links are only visible to Support Staff****External links are only visible to Support Staff**
There are plenty of websites using it, and I would like to crawl all the links at once instead of manually downloading one file at a time.
One problem is that there is no common host, i.e. people are free to create their own www. So I am not sure how the linkcrawler will know to use this plugin. But, if it is possible I would love to have a plugin for this.

[pspzockerscene]

Hi,

I don't see how/what this website does.
It is a wrapper for Google Drive content?
JDownloader already supports normal Google Drive URLs / files / folders / subfolders.

-psp-

[plaintext]

Some websites do not give out direct gdrive links, instead they provide using some wrapper or index service.
Parsing links from here is not too difficult, just get the href, and remove ?a=view if present, and you get the download link.
But manually creating the folder structure and remove the ?a=view from the url gets annoying after a while, therefore this request for link crawler plugin.

[plaintext]

>JDownloader already supports normal Google Drive URLs / files / folders / subfolders.

The last time I tried, login failed? And I searched the forums, it said that login has bug.
Is that fixed now.

[pspzockerscene]

Originally Posted by plaintext

Is that fixed now.

No

Please post example URLs and a guide on how to use this website.
Is there any kind of API?
Is there any way to get the real Google Drive URLs?
-psp-

[plaintext]

> Please post example URLs and a guide on how to use this website.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Folders have class=folder
Files have class=file
(This can be seen by html inspector in browsers.)

Some files (class=file) have a href ending with "?a=view"
For eg. **External links are only visible to Support Staff****External links are only visible to Support Staff**3.txt?a=view
Removing ?a=view gives - "**External links are only visible to Support Staff**3.txt". This is a downloadable url now.

If a file does not have a href ending with "?a=view", then it can be directly downloaded. (e.g. last file in this url)

Some folders can also have password/encryption: **External links are only visible to Support Staff****External links are only visible to Support Staff**

> Is there any kind of API?

AFAIK, no.

> Is there any way to get the real Google Drive URLs?

AFAIK, no

So, the algorithm to download from the page is -

1a. For all links of class File, remove ?a=view from the link
1b. Add to download

2. For all links of class Folder, fetch that link, and use these algorithms recursively.

The only issue as I said is that there is no single host, and everybody is free to use their own host. So some heuristic would be required to detect when this plugin is used (because the website is hosted on cloudflare workers, so they have url ending with worker.dev typically. not a great heuristic though). or maybe a manual option to invoke the plugin for some url.

[plaintext]

I don't know how jdownloader handles subdirectories, because I never used it to download such links till now, but if you choose to implement this plugin, please try to make it output files in proper subdirectories.

[pspzockerscene]

Ticket:


I'll check it out once I find the time ...

-psp-

[plaintext]

Okay, thank you.

[pspzockerscene]

I've added a simple crawler for this website for now.

Please provide the following types of test-URLs so that I can add more of your requested functionality:
- Folder containing subfolders AND files
- Password protected URLs with the corresponding password

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[plaintext]

Sorry for late reply. I found a link containing subfolders and files - **External links are only visible to Support Staff****External links are only visible to Support Staff**
I could not find any link with password, so leave it for now. If i find it, i will reply here.

[plaintext]

I dunno why my previous post is not showing up, so I am posting again -

> Folder containing subfolders AND files

**External links are only visible to Support Staff****External links are only visible to Support Staff**[Book]Freelancing%20and%20Marketing%20for%20Web%20Developers%20+%20Web%20Designers%20(The%20Complete%20Bun dle)/

This url contains a folder, and inside that folder are subfolders and files both.

> Password protected URLs with the corresponding password

I could not find any such links. Maybe leave password-protected links for now.

[pspzockerscene]

Originally Posted by plaintext

I dunno why my previous post is not showing up, so I am posting again -

Yes because the account you are using is publicly accessible, posts of that account need to be approved manually.

Your URLs are offline and none of them seems to lead to the website I previously implemented e.g. "index.gd.workers.dev".

-psp-

[plaintext]

> Your URLs are offline

Use this link for now -

**External links are only visible to Support Staff****External links are only visible to Support Staff**

> none of them seems to lead to the website I previously implemented e.g. "index.gd.workers.dev"

umm.. yeah. I said in the beginning that GoIndex is a self hosted solution, so the url host might be different but the page structure is the same, so the crawler should work.

[plaintext]

Also, here is a password protected link -
link -
**External links are only visible to Support Staff****External links are only visible to Support Staff**
password -
see edit history

[pspzockerscene]

I've added simple subfolder support and support for all "workers.dev" subdomains for the next update.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[plaintext]

Thank you. But it does not work properly. The parsed urls are wrong because they contain an ending slash in the url so file download fails with File Not Found. This happens with every url I tried.

Also I am getting files with "unknownFilename" if i crawl this link for eg. - **External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Yeah I see, sorry.
Only folders/subfdolders have to end with "/".
Fixed with the next update.
I've also added subfolder-support meaning you should get the same folder-structure you got on that website also after downloading via JD.
Please keep in mind that this will lead to a LOT of packages as our package system does not allow sub-packages with e.g. other download paths.
If you do not wish to have this feature, you can turn it off here:
Settings --> Packagizer --> "Adopt folder structure".

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-