JDownloader Community - Appwork GmbH
 

Reply
Page 3 of 3 12 3
 
Thread Tools Display Modes
  #41  
Old 30.09.2020, 15:14
plaintext plaintext is offline
BugMeNot Account
  
Join Date: Sep 2016
Posts: 241
Default
Quote:
Originally Posted by pspzockerscene View Post
Added support for old.reddit.com.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser **External links are only visible to Support Staff**...! | Please read our **External links are only visible to Support Staff**...!

-psp-
Thank you
Reply With Quote
  #42  
Old 06.10.2020, 17:43
verheiratet1952 verheiratet1952 is offline
JD VIP
  
Join Date: Jan 2016
Posts: 317
Default
now crawling for reddit links brings IMGUR prompt to add API credentials...

I am very sorry, but I dont see and find entry fields to add both "Client ID" and "Client Secret" ... and where to activate API?

please see screenshots...
I registered for my very own imgur account and did create an App like it is necessary with steps from jd prompt...
Attached Thumbnails
imgur-api-jd2beta.JPG   imgur-plugin.JPG  
Reply With Quote
  #43  
Old 06.10.2020, 18:04
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
  
Join Date: Mar 2009
Location: Deutschland
Posts: 69,629
Default
Quote:
Originally Posted by verheiratet1952 View Post
I am very sorry, but I dont see and find entry fields to add both "Client ID" and "Client Secret" ... and where to activate API?
Sorry, I've accidently released an unfinished version of the imgur.com plugin with enabled functionality that was not yet supposed to be visible yet.

This has been fixed and the current revision will work just fine without the need of API login credentials.
You can leave your own created imgur app active in your imgur account - it might be required some time in the future

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Reply With Quote
  #44  
Old 11.11.2020, 15:04
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
  
Join Date: Mar 2009
Location: Deutschland
Posts: 69,629
Default
Added support for new linktype:
reddit.com/gallery/XXX

This also includes support for reddit galleries inside single posts which the crawler wasn't able to handle so far.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Reply With Quote
  #45  
Old 29.07.2021, 17:35
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
  
Join Date: Mar 2009
Location: Deutschland
Posts: 69,629
Default
I've closed our reddit.com SVN ticket now.

It's working as it should - only reddit selfhosted content cannot be downloaded with sound which is another issue not related to our reddit crawler plugin.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Reply With Quote
  #46  
Old 10.03.2022, 12:09
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
  
Join Date: Mar 2009
Location: Deutschland
Posts: 69,629
Default
I got a PM with the question "is it possible to crawl comnplete subreddits".
The answer is: No, not yet.
At this moment, JD can only crawl the first page of an added subreddit-URL.
I've created a ticket for this feature request:


JD can however, crawl single reddit posts so if you want to crawl all of one subreddit you could do it like this:
1. Get a browser addon like "Link Gopher".
2. Scroll all the way down to the end/beginning of the subreddit (this can take a while).
3. Use the mentioned browser-addon to extract all links to the individual posts and add them to JD.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 10.03.2022 at 12:12.
Reply With Quote
  #47  
Old 10.03.2022, 15:34
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
  
Join Date: Mar 2009
Location: Deutschland
Posts: 69,629
Default
I've worked on it a bit but it doesn't yet work as it should:
- By default, JD will only crawl the first "page" (first 100 items) of a user profile/subreddit
- I've added plugin settings (Settings -> Plugins -> reddit.com) to allow it to crawl complete subreddits/profiles
- It doesn't yet work as expected as I was only aber to get max. the last ~1000 items but I've added my changes nevertheless

Again: This feature is not finished yet, see progress marked in the ticket I've linked in my last reply!



Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Reply With Quote
  #48  
Old 28.03.2022, 16:11
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
  
Join Date: Mar 2009
Location: Deutschland
Posts: 69,629
Default
New feature for the next update:
Text download of added posts (default = 'Always'):
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Reply With Quote
  #49  
Old 14.10.2022, 23:07
TomArrow TomArrow is offline
Super Loader
  
Join Date: Sep 2017
Posts: 27
Default
Hi, this appears to work well in general, however it seems that on repeated crawls of the same subreddit, nothing is crawled anymore. The Linkgrabber shows a progress for a short time, but nothing is actually added to the LinkGrabber list. I tried this on the subreddit "lordoftherings". The first time a few days ago it worked fine and got a few hundred files, but now it doesn't find anything anymore from the same subreddit.

On another note, the download of texts of posts doesn't work properly because every one of the text files has the same name and jDownloader2 thinks they are mirrors of the same file. Would be best to add some kind of filename-ized version of the title. Or maybe the post id. Or both.
Reply With Quote
  #50  
Old 17.10.2022, 13:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
  
Join Date: Mar 2009
Location: Germany
Posts: 79,084
Default
Quote:
Originally Posted by TomArrow View Post
On another note, the download of texts of posts doesn't work properly because every one of the text files has the same name and jDownloader2 thinks they are mirrors of the same file. Would be best to add some kind of filename-ized version of the title. Or maybe the post id. Or both.
Can you provide example links? What filename scheme in Settings->Plugins->reddit.com have you setup? maybe screenshot?
Edit: I can reproduce the issue
__________________
JD-Dev & Server-Admin
Reply With Quote
  #51  
Old 17.10.2022, 13:20
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
  
Join Date: Mar 2009
Location: Germany
Posts: 79,084
Default
Quote:
Originally Posted by TomArrow View Post
Hi, this appears to work well in general, however it seems that on repeated crawls of the same subreddit, nothing is crawled anymore. The Linkgrabber shows a progress for a short time, but nothing is actually added to the LinkGrabber list.
Most likely because JDownloader currently only supports the first page, see
https://board.jdownloader.org/showpo...7&postcount=47
https://board.jdownloader.org/showpo...8&postcount=46
__________________
JD-Dev & Server-Admin
Reply With Quote
  #52  
Old 17.10.2022, 18:14
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
  
Join Date: Mar 2009
Location: Deutschland
Posts: 69,629
Default
Quote:
Originally Posted by Jiaz View Post
Most likely because JDownloader currently only supports the first page, see
https://board.jdownloader.org/showpo...7&postcount=47
https://board.jdownloader.org/showpo...8&postcount=46
That information is partially outdated.

Crawling complete subreddits is deactivated by default because those can result in a lot of results.
You can enable it under Settings -> Plugins -> reddit.com -> Crawl complete subreddits? -> Enable this setting

About the other reported problem:
I'm working on it.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Reply With Quote
  #53  
Old 17.10.2022, 19:08
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
  
Join Date: Mar 2009
Location: Deutschland
Posts: 69,629
Default
Fixed bad filenames for .txt downloads when they were part of a post containing only text and no media content.

@TomArrow
You can either wait for the next CORE-Updates or work around it right away by using a Packagizer rule to define the filenames for reddit content which will override the scheme set in the plugin setting which can then be used to work around that bug.

For all code changes mentioned in this post, the following information applies:

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Reply With Quote
  #54  
Old 18.10.2022, 19:22
TomArrow TomArrow is offline
Super Loader
  
Join Date: Sep 2017
Posts: 27
Default
Cool, thanks for solving the filename issue!

About the "crawl complete subreddit" feature, I had forgotten to mention it, but I did have that activated actually, so consider my entire post to be in the context of having "crawl complete subreddit" active. So I think that problem still remains. Or did you fix that as well?

An example link of where I had said issue:
Quote:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
(it makes no difference if I actually leave the old.reddit.com domain or use **External links are only visible to Support Staff**www.reddit.com or just reddit.com)
Reply With Quote
  #55  
Old 18.10.2022, 19:57
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
  
Join Date: Mar 2009
Location: Deutschland
Posts: 69,629
Default
For the next update:
- Changed the last two setting for subreddit- and user profile crawling from a checkbox to a "limit how many of the last pages should be crawled" numerous setting
- Updated the wording to make it clear that the features of crawling complete user profiles and complete subreddits are both unfinished

Related internal ticket:



Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Reply With Quote
  #56  
Old 06.12.2022, 00:50
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
  
Join Date: Mar 2009
Location: Deutschland
Posts: 69,629
Default
Updated reddit plugin so "expected filenames" will also be found when directly adding single "v.redd.it" links.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Reply With Quote
Reply
Page 3 of 3 12 3

« Previous Thread | Next Thread »
Thread Tools
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

All times are GMT +2. The time now is 15:30.
JDownloader Community Board - Archive - Top
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.