[DukeM]

Hey, @psp!

Thanks so much for working on this. Wasn't expecting it to be this quick.

I saw a related post on r/datahoarder about this update first and the poster brought up a good point about complete crawling of users and subreddits.

I don't know how much help this would be but another user suggested PushShift for pulling the content instead to avoid a potential DoS. As far as I know about PushShift, it basically copies the data/content the moment they are submitted to Reddit. Maybe that can be a good way to go about that? Here's some more info about it if you're interested: **External links are only visible to Support Staff****External links are only visible to Support Staff**

They also have something for pulling Reddit searches but I haven't tried it. Might be worth a look for the other user who requested this particular feature.

Another user also suggested **External links are only visible to Support Staff****External links are only visible to Support Staff** but it's the first time I've heard of it.

Again, thanks! And I'll be looking out for future improvements.

[pspzockerscene]

Hi again,

thanks for the suggestions but unfortunately we're not going to use external APIs for crawling.
JD can now do the basics that will work for most of all users - if wanted, you could still e.g. use external APIs to crawl all comment-URLs of one subreddit --> Add those to JD --> And yes, this would again cause a lof of reddit http requests!

Todays update includes the following changes:
JD will now try to set filenames (basically the same as the packagename) for all reddit selfhosted content.
Please keep in mind that this will not e.g. apply for imgur content as we got a separate plugin for imgur and other services --> These will try to grab the original filenames from these sources accordingly.

Now it is up to you guys to test the existing functionality and make improvements suggestions.

The reddit ticket linked on the first page of this thread will stay open as our plugin does not yet have all of the functionality I want it to have.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[verheiratet1952]

what about "external-preview.redd.it" ?

Originally Posted by pspzockerscene

For the next update:
- v.redd.it and i.redd.it content will now also get displayed as host "reddit.com"
- v.redd.it: Always only grab the BEST video quality available
- Improved offline detection

.
.
.

-psp-

[pspzockerscene]

Hi,

Please post working example URLs.
I'm off now - seeya tomorrow ...

-psp-

[verheiratet1952]

Originally Posted by pspzockerscene

Hi,

Please post working example URLs.
I'm off now - seeya tomorrow ...

-psp-

here you are

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

I'm unable to open that "external" URL.
The content of that post is hosted on imgur.com and should be downloadable via JD just fine.

Does the imgur.com URL get added as offline for you?

-psp-

[verheiratet1952]

Originally Posted by pspzockerscene

I'm unable to open that "external" URL.
The content of that post is hosted on imgur.com and should be downloadable via JD just fine.

Does the imgur.com URL get added as offline for you?

-psp-

imgur.com URLs get added as offline, correct...

is there any solution available?

[pspzockerscene]

Originally Posted by verheiratet1952

imgur.com URLs get added as offline, correct...

Seems like you did change the default setting of your imgur.com plugin because otherwise this wouldn't have happened.

See Settings -> Plugins -> imgur.com -> Ativate "Use API[...]"
Afterwards, re-add your reddit/imgur URLs.

-psp-

[rafikabir85]

How can I download whole U/ or R/ user and subreddits RIP users and subreddits using jdownloader2?? tried it few times....probably not catching all the url's and ignoring imgur files. example: r/tightdresses and user/AshleyWilsonPT/

[pspzockerscene]

@rafikabir85
I recommend reading my posts in this thread especially this one.

-psp-

[pspzockerscene]

Next reddit Update will include a crawler to crawl all saved posts of an authenticated user but all account related features will be on hold until we got a nicer way to perform oauth logins:


Again:
We're open source - you're free to check out our code/progress HERE.

-psp-

[verheiratet1952]

Originally Posted by pspzockerscene

Seems like you did change the default setting of your imgur.com plugin because otherwise this wouldn't have happened.

See Settings -> Plugins -> imgur.com -> Ativate "Use API[...]"
Afterwards, re-add your reddit/imgur URLs.

-psp-

"Use API[...]" was already activated for months...

it does crawl most imgur links as offline, but it also adds them for both offline/online without correct renaming... it adds names like 'LNg3jgC' for package name even if title had also been added...

[pspzockerscene]

Originally Posted by verheiratet1952

"Use API[...]" was already activated for months...

Would surprise me if not, it is still the default setting but I thought you might have turned that off.

Originally Posted by verheiratet1952

it does crawl most imgur links as offline, but it also adds them for both offline/online[...]

Please post a DEBUG-log and more example URLs.
EDIT1:
Maybe your IP/ISP is blocked by the imgur API.
You could try to add your imgur.com account to JD and check if it works then.

Originally Posted by verheiratet1952

[...]without correct renaming... it adds names like 'LNg3jgC' for package name even if title had also been added...

You're wrong.
The naming might seem wrong to you but technically it is absolutely correct.
Reddit.com is linking to external websites --> JD will get the name from imgur.com and if no name is set here, the image-ID will be used.

It's the same when e.g. adding uploaded.net URLs via services such as filecrypt.cc --> JD will never use the filenames shown there - it will always try to get the filenames from the service where the file is hosted.

For reddit.com selfhoste content, the title of e.g. a comment will be set as filename.

If you wish to use the "source name" as title in such a case, you will have to create a Packagizer rule that sets the title of the package as filename for all imgur.com URLs.

-psp-

[verheiratet1952]

please have a look at blurred out image problem...

example link:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Originally Posted by verheiratet1952

please have a look at blurred out image problem...

I see no issue here.

Your URL leads to a post with NSFW content.
JD downloads the full resolution image without the blurred animation which is done via their website and has nothing todo with the content behind.

Please add a meaningful problem description.

-psp-

[pspzockerscene]

Next update will include:
- Subreddit crawler *
- User crawler *
- Superfast crawling for reddit selfhosted content

*
I've limited the crawler to crawl only the first page of a subreddit for now.
As said, crawling complete subreddits will cause a lot of http requests and I don't want reddit to ban our application so I will leave this disabled until I find a solution.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[plaintext]

> I've limited the crawler to crawl only the first page of a subreddit for now.

Is there any way to manually post the second/third/... page link and make jdownloader crawl it?

Also, the plugin does not work for 'old.reddit.com'. It only crawls from 'www.reddit.com'. Please add support for old.reddit.com as well if it is not too much work.

[pspzockerscene]

Originally Posted by plaintext

> I've limited the crawler to crawl only the first page of a subreddit for now.

Is there any way to manually post the second/third/... page link and make jdownloader crawl it?

No not (yet) but you are free to test my dev-work - my code contains switches to enable crawling all items.
We're open source!

Originally Posted by plaintext

Also, the plugin does not work for 'old.reddit.com'. It only crawls from 'www.reddit.com'. Please add support for old.reddit.com as well if it is not too much work.

Sure that is possible.
Please post example-URLs of all existing types for old.reddit.com (e.g. user, subreddit, comment, users' saved posts).

-psp-

[plaintext]

Originally Posted by pspzockerscene

No not (yet) but you are free to test my dev-work - my code contains switches to enable crawling all items.
**External links are only visible to Support Staff**...
Okay, I am not a developer so I will just wait.

Sure that is possible.
Please post example-URLs of all existing types for old.reddit.com (e.g. user, subreddit, comment, users' saved posts).

-psp-

To the best of my knowledge, the links are exactly the same with the only difference being old instead of www in the beginning.

Examples of some random subreddits and users and comments etc -
subreddit -
**External links are only visible to Support Staff****External links are only visible to Support Staff**
user -
**External links are only visible to Support Staff****External links are only visible to Support Staff**
comment -
**External links are only visible to Support Staff****External links are only visible to Support Staff**
saved posts -
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Added support for old.reddit.com.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-