JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 25.07.2020, 16:36
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 241
Default [LinkCrawler rule] request plugin oneclickchicks.com

example url - **External links are only visible to Support Staff****External links are only visible to Support Staff**

I would like it if the plugin automatically detects all images/gifs/videos in the complete forum post from all pages 1,2,3,4....etc.

The website requires account. Sent to psp in pm.
Reply With Quote
  #2  
Old 27.07.2020, 18:04
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

Hi,

this is just another generic forum.

A plugin for this is unlikely going to be added anywhere in the near future/ever.

I recommend trying it with our link crawler rules (see forum search) in combination with your cookies of this website.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #3  
Old 27.07.2020, 18:50
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 241
Default

is there a way to crawl through multiple forum pages through a link crawler rule?
Reply With Quote
  #4  
Old 27.07.2020, 19:00
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

there used to be a policy on not creating plugin support for forums also, not sure if that still exists.

@plaintext/bugmennot
sure, just need to create a rule or rules to find the content you want.
linkcrawler rules and decrypter plugins basically have the same outcome. though LC are for non complicated tasks (no wait times/captcha tasks etc)

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #5  
Old 27.07.2020, 19:03
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

Quote:
Originally Posted by plaintext View Post
is there a way to crawl through multiple forum pages through a link crawler rule?
To answer this more precicely:

Using only linkcrawler rules, this may be hard/impossible but at least single page parsing should be doable.

Also we're open source so you can write your own plugins at any time:
https://jdownloader.org/knowledge/wi...nt/get-started

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #6  
Old 27.07.2020, 22:06
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

I don't quite agree with psp statement above, if your first page (you example link looks like vbul url structure, quick check and it is) next page has unique url component its far from hard ('>' and its href). just supply DEEPDECRYPT rule (with thread url support with optional &page=\d+ listening) with correct deepPattern vs returning all (default). In this case a pattern for the post body and secondary for next page with | , then it should be possible to return everything you're looking for within a single rule.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #7  
Old 28.07.2020, 14:15
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

@raztoki
You are probably right

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #8  
Old 28.07.2020, 16:24
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 241
Default

I have started to work on a link crawler. Can you please tell me how to add oneclickchick cookies/login to jdownloader2 account manager? Maybe you can add a custom website option to the account manager so that people can add domains with cookies to login on unsupported website? Or is the cookie from linkcrawler rule also supposed to work while downloading?
Reply With Quote
  #9  
Old 28.07.2020, 18:04
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 241
Default

Code:
[
  {
    "cookies": [
      [
        "bbuserid",
        "removed"
      ],
      [
        "bbpassword",
        "removed"
      ]
    ],
    "pattern": "https?://forum.oneclickchicks.com/showthread.php\\?t=[0-9]+(&page=[0-9]+)?",
    "rule": "DEEPDECRYPT",
    "deepPattern": "https?://forum.oneclickchicks.com/thumbs/([0-9A-Za-z]*/)*([0-9A-Za-z]+)\\.([0-9A-Za-z]+)"
  },
  {
    "cookies": [
      [
        "bbuserid",
        "removed"
      ],
      [
        "bbpassword",
        "removed"
      ]
    ],
    "pattern": "https?://forum.oneclickchicks.com/thumbs/([0-9A-Za-z]*/)*([0-9A-Za-z]+)\\.([0-9A-Za-z]+)",
    "rule": "REWRITE",
    "rewriteReplaceWith": "**External links are only visible to Support Staff**
  },
  {
    "cookies": [
      [
        "bbuserid",
        "removed"
      ],
      [
        "bbpassword",
        "removed"
      ]
    ],
    "pattern": "**External links are only visible to Support Staff**,
    "rule": "DIRECTHTTP"
  }
]
This is what I came up with (for scraping links from a single page). The last two rules seem to work fine. But the first rule does not work. Can you please help with it.

Last edited by raztoki; 28.07.2020 at 20:18.
Reply With Quote
  #10  
Old 28.07.2020, 18:49
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 241
Default

Also, point to note - the regex in the first rule (deepdecrypt) works fine when tested on regexr or regex101 online. Maybe I messed up the escaping characters or something in the rule, or maybe jdownloader's regex engine doesn't support something... Idk. Please help whenever you have time. (Also, if have a bit more time, please make the deepdecrypt rule work for all pages)
Thank you.
Reply With Quote
  #11  
Old 28.07.2020, 19:17
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 241
Default

... I forgot to remove the userid and password cookies from the 2nd and 3rd rules pasted above. Please remove them @admins..
Reply With Quote
  #12  
Old 28.07.2020, 20:50
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

Quote:
Originally Posted by plaintext View Post
... I forgot to remove the userid and password cookies from the 2nd and 3rd rules pasted above. Please remove them @admins..
Already done.

Normal users can edit posts but not the bugmenot account.

Here is my attempt:
Code:
[ {
  "enabled" : true,
  "cookies" : [ [ "bbuserid", "CENSORED" ], [ "bbpassword", "CENSORED" ] ],
  "updateCookies" : true,
  "logging" : false,
  "maxDecryptDepth" : 0,
  "name" : "oneclickchicks.com directURLs",
  "pattern" : "**External links are only visible to Support Staff**,
  "rule" : "DIRECTHTTP"
},{
  "enabled" : true,
  "cookies" : [ [ "bbuserid", "CENSORED" ], [ "bbpassword", "CENSORED" ] ],
  "updateCookies" : true,
  "logging" : false,
  "maxDecryptDepth" : 0,
  "name" : "oneclickchicks.com grab URLs from single forum thread",
  "pattern" : "https?://forum.oneclickchicks.com/showthread.php\\?t=[0-9]+(&page=[0-9]+)?",
  "packageNamePattern" : "<title>(.*?)</title>",
  "deepPattern" : "(attachment\\.php\\?attachmentid=[^\"]+)",
  "rule" : "DEEPDECRYPT"
} ]
Rule as plaintext to get around our forum URL censor:
pastebin.com/mgqmS99K

This finds all(?) 40 items and puts them in one package - tested with the first page of your linked forum thread.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #13  
Old 28.07.2020, 21:39
plaintext plaintext is offline
BugMeNot Account
 
Join Date: Sep 2016
Posts: 241
Default

thank you!
Reply With Quote
  #14  
Old 29.07.2020, 14:38
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

Thanks for your feedback

You can now change the password of your account.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 19:53.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.