JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 07.04.2019, 13:40
asdpoi2k
Guest
 
Posts: n/a
Default request plugin for reddit.com (video download is missing audio)

There are a lot of subreddits and search results from reddit.com that I would like to capture, such as:

subreddit
**External links are only visible to Support Staff****External links are only visible to Support Staff**

search result
**External links are only visible to Support Staff****External links are only visible to Support Staff**

user posts
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Can you add reddit.com account support so it will allow for showing/capturing nsfw by default, and also add scrolling support? Right now, if I add a link it captures 54 images which appear to be mostly related to the website theme, and then about 5 images which are the content I want, but they show as being "offline".

Last edited by asdpoi2k; 09.04.2019 at 13:43. Reason: extra clarification
Reply With Quote
  #2  
Old 09.04.2019, 13:44
asdpoi2k
Guest
 
Posts: n/a
Default

bump, did this one get missed or did I miss something? I don't think reddit is supported anywhere yet?
Reply With Quote
  #3  
Old 10.04.2019, 19:32
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,531
Default

I can think of supporting /user/ and maybe /r/ but unlikely search urls as we don't often add support for those as those can cause high cpu load and are only good/usefull for mass crawling.
We first have to check how the site works and maybe they provide a lightweight api that could be used

__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 12.07.2020, 16:39
Keagel Keagel is offline
Baby Loader
 
Join Date: May 2020
Posts: 7
Question Is there a reason Reddit videos aren't supported?

Whether I copy a v.redd.it link or a Reddit post's link, JDownloader doesn't do anything. I found a thread from 2 years ago with the progress still at 0%, but that person at least had the video and audio files even if they weren't merged.

Am I doing something wrong? I don't even see a Reddit plugin within JDownloader.
Reply With Quote
  #5  
Old 13.07.2020, 12:02
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

@Kaegel
Merged reddit threads.

The reasons are simple:
- We cannot add support for every website of the Internet
- A reddit plugin has been requested by serveral users but so far, not yet been created

My "workaround" suggestion:
Please use free browser addons such as "Video DownloadHelper" to download reddit content - you can use this addon as a helper to find the video downloadurls and download them with JD (well, in most cases).
If not, you can still manually download via browser.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #6  
Old 15.07.2020, 20:57
DukeM's Avatar
DukeM DukeM is offline
JD Adviser
 
Join Date: Sep 2019
Posts: 113
Default

Was planning on making a thread but noticed this is becoming the thread for Reddit, so I'll just add a reply here.

Not sure if supporting Reddit has ever been in your actual backlog but maybe this could help.

A couple software that I used/use to archive Reddit /users and /r are:

RipMe: **External links are only visible to Support Staff**...
and
BDFR: **External links are only visible to Support Staff**...

Since the introduction of RedGIFs, RipMe has essentially stopped working for almost all hosts (imgur, gfycat, v.redd.it, etc) and BDFR has some troubles of its own with very poor dev support.

Maybe a Reddit plugin support can be built from these two to be utilised in JD2? Not saying exactly copying them but going back to Jiaz's comment from last year about checking how the site works, I figured these applications could help provide some insight.

Just spitballing here as I'm out of options when it comes to software for archiving Reddit content and was kind of hoping the people behind JD2 would be interested in filling this gap. Thank you.
Reply With Quote
  #7  
Old 16.07.2020, 12:30
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

Hi DukeM,
we already know these other tools

Also there are still plenty of online "reddit downloaders" up and running fine.
Also, you can easily add URLs of ANY video site to JD by using simple helper browser addons such as e.g. "Video DownloadHelper".

Regarding reddit:
Do they provide an official API?
Do they mind ppl. downloading their content or do they not care?
I've even seen public reddit downloader bots on reddit itself so it seems like they at least don't try to stop ppl. from downloading the content ... after all it's all user-generated anyways.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #8  
Old 16.07.2020, 22:43
DukeM's Avatar
DukeM DukeM is offline
JD Adviser
 
Join Date: Sep 2019
Posts: 113
Default

Hey there, psp!

Ah, I do have VDH but I don't think their format really works for archiving entire users or subreddits. Although, I'm not aware of any other proper full archiving software out there that works as efficiently as these two (or as these two did/do, lol).

The most important bit, as far as I'm concerned, is the proper filenaming system among other things and imo, BDFR making it customisable like {DATE}{USERNAME}{POSTID} etc is a pretty neat feature. If anything though, VDH is really good at fetching HLS links. Haha.

Quote:
Originally Posted by pspzockerscene View Post
Do they provide an official API?
As far as I know, they do. This docs page might be a good start: **External links are only visible to Support Staff****External links are only visible to Support Staff**

There is also a feature where you can connect an "app" to your reddit account if you go over to **External links are only visible to Support Staff****External links are only visible to Support Staff**. Not sure what other benefits it gives yet aside from maybe accessing private/restricted subreddits your account is a member in and maybe bypassing captchas(?).

Quote:
Originally Posted by pspzockerscene View Post
Do they mind ppl. downloading their content or do they not care?
I've even seen public reddit downloader bots on reddit itself so it seems like they at least don't try to stop ppl. from downloading the content ... after all it's all user-generated anyways.
I'm not that active on Reddit, especially on meta things about Reddit, but I've not been aware of any rules disallowing sniffing out content either from users or entire subreddits. I think they're more concerned on trying to monetise the site properly and keeping a distinction between regular and NSFW content.

If you go over at r/datahoarder, you can even sometimes see discussions about archiving some users or communities.

There are also entire reddit archivers that exist like ceddit.com, unreddit.com, removeddit.com, etc. that works like WayBack Machine and can even show you mod-deleted comments.

So, I guess it'd be pretty safe to assume that Reddit doesn't care about this stuff, at least for now. And even then, most of the content are uploaded on third-party hosts (imgur, gfycat, redgifs, etc.) which is out of their jurisdiction in the first place anyway.

Really hope you lot would take an interest in this.
Reply With Quote
  #9  
Old 17.07.2020, 14:22
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

Indeed you're right and they're really developer-friendly - seems like they do not even mind crawlers - I actually read their developer.

I've started working on it but please do not ask me for any ETA.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #10  
Old 17.07.2020, 21:54
DukeM's Avatar
DukeM DukeM is offline
JD Adviser
 
Join Date: Sep 2019
Posts: 113
Default

Oh wow, that's awesome to hear! Thank you so much, psp.

And don't worry, I don't feel entitled enough to pester you on this. But I'll be sure to be keeping an eye out for when it gets deployed.

Cheers!
Reply With Quote
  #11  
Old 21.07.2020, 15:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

The next update will contain the very first version of our reddit crawler

What it can do:
Crawl single reddit posts:
- External embedded (video) content such as e.g. gfycat
- External embedded URLs e.g. "embedded" imgur.com images

What it can't do (yet):
- Crawl reddit-selfhosted videos and images (Yes I know a lot of content is hosted by reddit ... will work on it once I find the time ...)

What it might never be able to do (unsure):
- Crawling complete user profiles
- Crawling complete subreddits
Reasons: Requires a LOT of http requests

What it will definitely never be able to do as we do not provide such crawlers:
- Crawling "search query URLs" e.g.:
Code:
reddit.com/search?q=nature%20images
--> Sorry @asdpoi2k

We are open source - you can contribute code at any time

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #12  
Old 22.07.2020, 16:14
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

Added simple support for downloading reddit selfhosted content.

Now it is up to you to test our "single posts crawler" and e.g. find posts for which JD does not find anything although you might expect it to.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #13  
Old 23.07.2020, 11:58
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 326
Default

how about adding TITLE from single reddit post to foldername?

that make all files much more interesting to have a name itself


Quote:
Originally Posted by pspzockerscene View Post
Added simple support for downloading reddit selfhosted content.

Now it is up to you to test our "single posts crawler" and e.g. find posts for which JD does not find anything although you might expect it to.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!


-psp-
Reply With Quote
  #14  
Old 23.07.2020, 12:01
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

Quote:
Originally Posted by verheiratet1952 View Post
how about adding TITLE from single reddit post to foldername?
There is already a comment in the current code from myself noting this

As you can imagine, what I've done until now was just "quick and dirty" - it'll get better every day

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #15  
Old 23.07.2020, 12:28
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 326
Default

"pattern" : "**External links are only visible to Support Staff**,
"rule" : "DEEPDECRYPT",
"packageNamePattern" : "<title>(.+)</title>",


Quote:
Originally Posted by pspzockerscene View Post
There is already a comment in the current code from myself noting this

As you can imagine, what I've done until now was just "quick and dirty" - it'll get better every day

-psp-
Reply With Quote
  #16  
Old 23.07.2020, 12:45
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

@verheiratet1952
Your LinkCrawler rule will not work anymore once our crawler is released which it already is ...

Also, we are not using the reddit mainpage e.g.
Code:
reddit.com/r/de/comments/hvzlf7/falschparken_in_stuttgart/
-->
Code:
reddit.com/comments/hvzlf7/.json
As said, I just didn't have the time to add the package name yesterday

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #17  
Old 23.07.2020, 14:36
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

For the next update:
- Added package name including date when the content was posted and name of the subreddit
- Added ability to parse multiple reddit hosted pictures of present
- Added parser for post text itself: If the text of post contains downloadable URLs which JD can handle, it will find them as well now

I did not yet modify the filenames for reddit selfhosted content.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #18  
Old 23.07.2020, 15:20
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 326
Default

great changes!

is there any chance to edit/change settings? to customize for its own needs?


Quote:
Originally Posted by pspzockerscene View Post
For the next update:
- Added package name including date when the content was posted and name of the subreddit
- Added ability to parse multiple reddit hosted pictures of present
- Added parser for post text itself: If the text of post contains downloadable URLs which JD can handle, it will find them as well now

I did not yet modify the filenames for reddit selfhosted content.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!


-psp-
Reply With Quote
  #19  
Old 23.07.2020, 15:33
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

Quote:
Originally Posted by verheiratet1952 View Post
great changes!
Thanks for your feedback

Quote:
Originally Posted by verheiratet1952 View Post
is there any chance to edit/change settings? to customize for its own needs?
Which ones exactly?
If you're talking about filenames/packagenames:
You can always modify those via Packagizer rules.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #20  
Old 24.07.2020, 15:43
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

For the next update:
- v.redd.it and i.redd.it content will now also get displayed as host "reddit.com"
- v.redd.it: Always only grab the BEST video quality available
- Improved offline detection

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #21  
Old 14.09.2020, 17:46
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 326
Default

what about "external-preview.redd.it" ?

Quote:
Originally Posted by pspzockerscene View Post
For the next update:
- v.redd.it and i.redd.it content will now also get displayed as host "reddit.com"
- v.redd.it: Always only grab the BEST video quality available
- Improved offline detection

.
.
.

-psp-
Reply With Quote
  #22  
Old 26.07.2020, 06:50
DukeM's Avatar
DukeM DukeM is offline
JD Adviser
 
Join Date: Sep 2019
Posts: 113
Default

Hey, @psp!

Thanks so much for working on this. Wasn't expecting it to be this quick.

I saw a related post on r/datahoarder about this update first and the poster brought up a good point about complete crawling of users and subreddits.

I don't know how much help this would be but another user suggested PushShift for pulling the content instead to avoid a potential DoS. As far as I know about PushShift, it basically copies the data/content the moment they are submitted to Reddit. Maybe that can be a good way to go about that? Here's some more info about it if you're interested: **External links are only visible to Support Staff****External links are only visible to Support Staff**

They also have something for pulling Reddit searches but I haven't tried it. Might be worth a look for the other user who requested this particular feature.

Another user also suggested **External links are only visible to Support Staff****External links are only visible to Support Staff** but it's the first time I've heard of it.

Again, thanks! And I'll be looking out for future improvements.
Reply With Quote
  #23  
Old 27.07.2020, 18:27
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

Hi again,

thanks for the suggestions but unfortunately we're not going to use external APIs for crawling.
JD can now do the basics that will work for most of all users - if wanted, you could still e.g. use external APIs to crawl all comment-URLs of one subreddit --> Add those to JD --> And yes, this would again cause a lof of reddit http requests!

Todays update includes the following changes:
JD will now try to set filenames (basically the same as the packagename) for all reddit selfhosted content.
Please keep in mind that this will not e.g. apply for imgur content as we got a separate plugin for imgur and other services --> These will try to grab the original filenames from these sources accordingly.

Now it is up to you guys to test the existing functionality and make improvements suggestions.

The reddit ticket linked on the first page of this thread will stay open as our plugin does not yet have all of the functionality I want it to have.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #24  
Old 14.09.2020, 17:48
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

Hi,

Please post working example URLs.
I'm off now - seeya tomorrow ...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #25  
Old 14.09.2020, 17:55
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 326
Default

Quote:
Originally Posted by pspzockerscene View Post
Hi,

Please post working example URLs.
I'm off now - seeya tomorrow ...

-psp-
here you are

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #26  
Old 15.09.2020, 18:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

I'm unable to open that "external" URL.
The content of that post is hosted on imgur.com and should be downloadable via JD just fine.

Does the imgur.com URL get added as offline for you?

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #27  
Old 15.09.2020, 21:04
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 326
Default

Quote:
Originally Posted by pspzockerscene View Post
I'm unable to open that "external" URL.
The content of that post is hosted on imgur.com and should be downloadable via JD just fine.

Does the imgur.com URL get added as offline for you?

-psp-
imgur.com URLs get added as offline, correct...

is there any solution available?
Reply With Quote
  #28  
Old 16.09.2020, 14:23
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

Quote:
Originally Posted by verheiratet1952 View Post
imgur.com URLs get added as offline, correct...
Seems like you did change the default setting of your imgur.com plugin because otherwise this wouldn't have happened.

See Settings -> Plugins -> imgur.com -> Ativate "Use API[...]"
Afterwards, re-add your reddit/imgur URLs.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #29  
Old 18.09.2020, 14:49
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 326
Default

Quote:
Originally Posted by pspzockerscene View Post
Seems like you did change the default setting of your imgur.com plugin because otherwise this wouldn't have happened.

See Settings -> Plugins -> imgur.com -> Ativate "Use API[...]"
Afterwards, re-add your reddit/imgur URLs.

-psp-

"Use API[...]" was already activated for months...

it does crawl most imgur links as offline, but it also adds them for both offline/online without correct renaming... it adds names like 'LNg3jgC' for package name even if title had also been added...
Reply With Quote
  #30  
Old 18.09.2020, 16:11
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

Quote:
Originally Posted by verheiratet1952 View Post
"Use API[...]" was already activated for months...
Would surprise me if not, it is still the default setting but I thought you might have turned that off.

Quote:
Originally Posted by verheiratet1952 View Post
it does crawl most imgur links as offline, but it also adds them for both offline/online[...]
Please post a DEBUG-log and more example URLs.
EDIT1:
Maybe your IP/ISP is blocked by the imgur API.
You could try to add your imgur.com account to JD and check if it works then.

Quote:
Originally Posted by verheiratet1952 View Post
[...]without correct renaming... it adds names like 'LNg3jgC' for package name even if title had also been added...
You're wrong.
The naming might seem wrong to you but technically it is absolutely correct.
Reddit.com is linking to external websites --> JD will get the name from imgur.com and if no name is set here, the image-ID will be used.

It's the same when e.g. adding uploaded.net URLs via services such as filecrypt.cc --> JD will never use the filenames shown there - it will always try to get the filenames from the service where the file is hosted.

For reddit.com selfhoste content, the title of e.g. a comment will be set as filename.

If you wish to use the "source name" as title in such a case, you will have to create a Packagizer rule that sets the title of the package as filename for all imgur.com URLs.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #31  
Old 17.09.2020, 09:44
rafikabir85 rafikabir85 is offline
Modem User
 
Join Date: May 2020
Posts: 1
Default

How can I download whole U/ or R/ user and subreddits RIP users and subreddits using jdownloader2?? tried it few times....probably not catching all the url's and ignoring imgur files. example: r/tightdresses and user/AshleyWilsonPT/

Last edited by rafikabir85; 17.09.2020 at 09:50.
Reply With Quote
  #32  
Old 17.09.2020, 15:10
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

@rafikabir85
I recommend reading my posts in this thread especially this one.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #33  
Old 17.09.2020, 16:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

Next reddit Update will include a crawler to crawl all saved posts of an authenticated user but all account related features will be on hold until we got a nicer way to perform oauth logins:


Again:
We're open source - you're free to check out our code/progress HERE.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #34  
Old 22.09.2020, 16:10
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

Next update will include:
- Subreddit crawler *
- User crawler *
- Superfast crawling for reddit selfhosted content

*
I've limited the crawler to crawl only the first page of a subreddit for now.
As said, crawling complete subreddits will cause a lot of http requests and I don't want reddit to ban our application so I will leave this disabled until I find a solution.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 22.09.2020 at 16:19.
Reply With Quote
  #35  
Old 28.09.2020, 11:49
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 241
Default

> I've limited the crawler to crawl only the first page of a subreddit for now.

Is there any way to manually post the second/third/... page link and make jdownloader crawl it?

Also, the plugin does not work for 'old.reddit.com'. It only crawls from 'www.reddit.com'. Please add support for old.reddit.com as well if it is not too much work.
Reply With Quote
  #36  
Old 29.09.2020, 17:48
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

Quote:
Originally Posted by plaintext View Post
> I've limited the crawler to crawl only the first page of a subreddit for now.

Is there any way to manually post the second/third/... page link and make jdownloader crawl it?
No not (yet) but you are free to test my dev-work - my code contains switches to enable crawling all items.
We're open source!

Quote:
Originally Posted by plaintext View Post
Also, the plugin does not work for 'old.reddit.com'. It only crawls from 'www.reddit.com'. Please add support for old.reddit.com as well if it is not too much work.
Sure that is possible.
Please post example-URLs of all existing types for old.reddit.com (e.g. user, subreddit, comment, users' saved posts).

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #37  
Old 30.09.2020, 10:01
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 241
Default

Quote:
Originally Posted by pspzockerscene View Post
No not (yet) but you are free to test my dev-work - my code contains switches to enable crawling all items.
**External links are only visible to Support Staff**...
Okay, I am not a developer so I will just wait.

Sure that is possible.
Please post example-URLs of all existing types for old.reddit.com (e.g. user, subreddit, comment, users' saved posts).

-psp-
To the best of my knowledge, the links are exactly the same with the only difference being old instead of www in the beginning.

Examples of some random subreddits and users and comments etc -
subreddit -
**External links are only visible to Support Staff****External links are only visible to Support Staff**
user -
**External links are only visible to Support Staff****External links are only visible to Support Staff**
comment -
**External links are only visible to Support Staff****External links are only visible to Support Staff**
saved posts -
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #38  
Old 30.09.2020, 13:15
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,103
Default

Added support for old.reddit.com.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #39  
Old 30.09.2020, 14:14
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 241
Default

Quote:
Originally Posted by pspzockerscene View Post
Added support for old.reddit.com.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!


-psp-
Thank you
Reply With Quote
  #40  
Old 06.10.2020, 16:43
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 326
Default

now crawling for reddit links brings IMGUR prompt to add API credentials...

I am very sorry, but I dont see and find entry fields to add both "Client ID" and "Client Secret" ... and where to activate API?

please see screenshots...
I registered for my very own imgur account and did create an App like it is necessary with steps from jd prompt...
Attached Images
File Type: jpg imgur-api-jd2beta.JPG (50.3 KB, 1 views)
File Type: jpg imgur-plugin.JPG (91.0 KB, 2 views)
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 18:24.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.