JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #21  
Old 26.07.2020, 06:50
DukeM DukeM is offline
Super Loader
 
Join Date: Sep 2019
Posts: 25
Default

Hey, @psp!

Thanks so much for working on this. Wasn't expecting it to be this quick.

I saw a related post on r/datahoarder about this update first and the poster brought up a good point about complete crawling of users and subreddits.

I don't know how much help this would be but another user suggested PushShift for pulling the content instead to avoid a potential DoS. As far as I know about PushShift, it basically copies the data/content the moment they are submitted to Reddit. Maybe that can be a good way to go about that? Here's some more info about it if you're interested: **External links are only visible to Support Staff****External links are only visible to Support Staff**

They also have something for pulling Reddit searches but I haven't tried it. Might be worth a look for the other user who requested this particular feature.

Another user also suggested **External links are only visible to Support Staff****External links are only visible to Support Staff** but it's the first time I've heard of it.

Again, thanks! And I'll be looking out for future improvements.
Reply With Quote
  #22  
Old 27.07.2020, 18:27
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,332
Default

Hi again,

thanks for the suggestions but unfortunately we're not going to use external APIs for crawling.
JD can now do the basics that will work for most of all users - if wanted, you could still e.g. use external APIs to crawl all comment-URLs of one subreddit --> Add those to JD --> And yes, this would again cause a lof of reddit http requests!

Todays update includes the following changes:
JD will now try to set filenames (basically the same as the packagename) for all reddit selfhosted content.
Please keep in mind that this will not e.g. apply for imgur content as we got a separate plugin for imgur and other services --> These will try to grab the original filenames from these sources accordingly.

Now it is up to you guys to test the existing functionality and make improvements suggestions.

The reddit ticket linked on the first page of this thread will stay open as our plugin does not yet have all of the functionality I want it to have.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #23  
Old 14.09.2020, 17:46
verheiratet1952 verheiratet1952 is offline
JD Addict
 
Join Date: Jan 2016
Posts: 166
Default

what about "external-preview.redd.it" ?

Quote:
Originally Posted by pspzockerscene View Post
For the next update:
- v.redd.it and i.redd.it content will now also get displayed as host "reddit.com"
- v.redd.it: Always only grab the BEST video quality available
- Improved offline detection

.
.
.

-psp-
Reply With Quote
  #24  
Old 14.09.2020, 17:48
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,332
Default

Hi,

Please post working example URLs.
I'm off now - seeya tomorrow ...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #25  
Old 14.09.2020, 17:55
verheiratet1952 verheiratet1952 is offline
JD Addict
 
Join Date: Jan 2016
Posts: 166
Default

Quote:
Originally Posted by pspzockerscene View Post
Hi,

Please post working example URLs.
I'm off now - seeya tomorrow ...

-psp-
here you are

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #26  
Old 15.09.2020, 18:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,332
Default

I'm unable to open that "external" URL.
The content of that post is hosted on imgur.com and should be downloadable via JD just fine.

Does the imgur.com URL get added as offline for you?

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #27  
Old 15.09.2020, 21:04
verheiratet1952 verheiratet1952 is offline
JD Addict
 
Join Date: Jan 2016
Posts: 166
Default

Quote:
Originally Posted by pspzockerscene View Post
I'm unable to open that "external" URL.
The content of that post is hosted on imgur.com and should be downloadable via JD just fine.

Does the imgur.com URL get added as offline for you?

-psp-
imgur.com URLs get added as offline, correct...

is there any solution available?
Reply With Quote
  #28  
Old 16.09.2020, 14:23
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,332
Default

Quote:
Originally Posted by verheiratet1952 View Post
imgur.com URLs get added as offline, correct...
Seems like you did change the default setting of your imgur.com plugin because otherwise this wouldn't have happened.

See Settings -> Plugins -> imgur.com -> Ativate "Use API[...]"
Afterwards, re-add your reddit/imgur URLs.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #29  
Old 17.09.2020, 09:44
rafikabir85 rafikabir85 is offline
Modem User
 
Join Date: May 2020
Posts: 1
Default

How can I download whole U/ or R/ user and subreddits RIP users and subreddits using jdownloader2?? tried it few times....probably not catching all the url's and ignoring imgur files. example: r/tightdresses and user/AshleyWilsonPT/

Last edited by rafikabir85; 17.09.2020 at 09:50.
Reply With Quote
  #30  
Old 17.09.2020, 15:10
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,332
Default

@rafikabir85
I recommend reading my posts in this thread especially this one.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #31  
Old 17.09.2020, 16:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,332
Default

Next reddit Update will include a crawler to crawl all saved posts of an authenticated user but all account related features will be on hold until we got a nicer way to perform oauth logins:


Again:
We're open source - you're free to check out our code/progress HERE.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #32  
Old 18.09.2020, 14:49
verheiratet1952 verheiratet1952 is offline
JD Addict
 
Join Date: Jan 2016
Posts: 166
Default

Quote:
Originally Posted by pspzockerscene View Post
Seems like you did change the default setting of your imgur.com plugin because otherwise this wouldn't have happened.

See Settings -> Plugins -> imgur.com -> Ativate "Use API[...]"
Afterwards, re-add your reddit/imgur URLs.

-psp-

"Use API[...]" was already activated for months...

it does crawl most imgur links as offline, but it also adds them for both offline/online without correct renaming... it adds names like 'LNg3jgC' for package name even if title had also been added...
Reply With Quote
  #33  
Old 18.09.2020, 16:11
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 54,332
Default

Quote:
Originally Posted by verheiratet1952 View Post
"Use API[...]" was already activated for months...
Would surprise me if not, it is still the default setting but I thought you might have turned that off.

Quote:
Originally Posted by verheiratet1952 View Post
it does crawl most imgur links as offline, but it also adds them for both offline/online[...]
Please post a DEBUG-log and more example URLs.
EDIT1:
Maybe your IP/ISP is blocked by the imgur API.
You could try to add your imgur.com account to JD and check if it works then.

Quote:
Originally Posted by verheiratet1952 View Post
[...]without correct renaming... it adds names like 'LNg3jgC' for package name even if title had also been added...
You're wrong.
The naming might seem wrong to you but technically it is absolutely correct.
Reddit.com is linking to external websites --> JD will get the name from imgur.com and if no name is set here, the image-ID will be used.

It's the same when e.g. adding uploaded.net URLs via services such as filecrypt.cc --> JD will never use the filenames shown there - it will always try to get the filenames from the service where the file is hosted.

For reddit.com selfhoste content, the title of e.g. a comment will be set as filename.

If you wish to use the "source name" as title in such a case, you will have to create a Packagizer rule that sets the title of the package as filename for all imgur.com URLs.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
How to create a log || Wie man einen Log erstellt
Captcha FAQ EN || Captcha FAQ DE || Erste Schritte & Tutorials
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 15:31.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.