JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 12.06.2021, 23:42
BJN01 BJN01 is offline
JD Adviser
  
Join Date: Jan 2020
Posts: 113
Default [LinkCrawler Rule] request for: isekaiscan.com
hello ,
I would like to ask for something to download the chapters from the isekaiscan.com site .

Examples links

series page
**External links are only visible to Support Staff****External links are only visible to Support Staff**
chapter page
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**


I searched a bit in the html of the chapter 1 and I would have found:

title

Spoiler:
Code:
<!DOCTYPE html>
<html lang="en-US">
<head>
... omiss...
<title>My Gift LVL 9999 Unlimited Gacha - chapter 1</title>
...omiss...
</head>


link file :
Spoiler:
Code:
<input type="hidden" id="wp-manga-current-chap" data-id="138845" value="chapter-1" />
<div class="page-break no-gaps">
<img id="image-0" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-1" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-2" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-3" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
<div id="ads-between-images" class="advance-ads"> </div> </div>
<div class="page-break no-gaps">
<img id="image-4" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-5" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-6" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-7" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-8" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-9" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-10" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-11" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-12" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-13" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-14" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-15" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-16" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-17" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-18" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-19" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-20" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-21" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-22" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-23" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-24" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-25" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-26" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
 <img id="image-27" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-28" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-29" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-30" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-31" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-32" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-33" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-34" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-35" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-36" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-37" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-38" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-39" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-40" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-41" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-42" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-43" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-44" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-45" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-46" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-47" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-48" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-49" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-50" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-51" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-52" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
</div>
<div class="page-break no-gaps">
<img id="image-53" data-src="			
			**External links are only visible to Support Staff** class="wp-manga-chapter-img img-responsive lazyload effect-fade">
 </div>
</div>



would it be possible to achieve something like this? :

- My Gift LVL 9999 Unlimited Gacha - ch01
--------My Gift LVL 9999 Unlimited Gacha - ch01-Page_01.jpg
--------My Gift LVL 9999 Unlimited Gacha - ch01-Page_02.jpg
--------My Gift LVL 9999 Unlimited Gacha - ch01-Page_03.jpg
---------..............................................
--------My Gift LVL 9999 Unlimited Gacha - ch01-Page_XX.jpg


name package = "title" (My Gift LVL 9999 Unlimited Gacha - chapter 1)
file name = "title" + "-Page_" + fileNumber.jpg/png/

if it is too complicated "Chapter" can also remain, I change it with a script in "Ch"



example link with "."
ch 1.2.
**External links are only visible to Support Staff****External links are only visible to Support Staff**
ch 2.1
**External links are only visible to Support Staff****External links are only visible to Support Staff**
ch8.2
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Last edited by BJN01; 12.06.2021 at 23:56.
Reply With Quote
  #2  
Old 14.06.2021, 14:23
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
  
Join Date: Mar 2009
Location: Deutschland
Posts: 74,658
Default
Hi,
this website is using Cloudflare and also appears to be down (or GEO-blocked) at this moment.

English / Englisch
the issue you've reported has been caused by our current Cloudflare issues.
Please read the first post of the linked thread and post in this thread if you have further questions!
In case you've posted a new thread and it will either get merged with the linked Cloudflare thread or we will lock it.
Please post Cloudflare related questions in the above linked thread only!
Attention: The first post of that linked thread contains useful information for you AND website-owners and also the hint that website-owners can solve this issue on their side which would also make our plugin work again!
There is no ETA for a Cloudflare fix from our side so it might be faster to ask website admins! Please read the first post of the linked thread completely and consider sending it to the admins of the website you are having issues with!

Deutsch / German
Das von dir gemeldete Problem wurde durch die aktuellen Cloudflare Probleme verursacht.
Bitte lies den ersten Post des Cloudflare Threads.
Falls du einen neuen Thread dazu geöffnet hast wurde dieser entweder in den bestehenden Cloudflare Thread verschoben oder mit diesem Post von mir beantwortet und geschlossen.
Weitere Fragen bitte im oben verlinkten Cloudflare Thread posten!
Achtung: Im ersten Post ist u.a. beschrieben, dass auch Webseiten-Admins das Problem auf ihrer Seite lösen können --> Unser Plugin würde dann sofort wieder funktionieren - das bitte beachten, den ersten Post vollständig lesen und ggf. an den Support der betroffenen Webseite weiterleiten!
Wie können derzeit nicht sagen, wann es von unserer Seite aus eine Lösung für das Cloudflare Problem geben wird!

-psp-
EDIT

You can bump this thread once the website is back online so we can check on this again but du not expect a positive outcome!
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Last edited by pspzockerscene; 14.06.2021 at 14:36.
Reply With Quote
  #3  
Old 14.06.2021, 17:29
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
  
Join Date: Mar 2009
Location: Deutschland
Posts: 74,658
Default
Update:
Upon doing some more experiments it seems like this website is doing GEO-blocking via Cloudflare at least they do block my German IP.

I was able to access this website using an US proxy and then indeed even Cloudflare let me through.

This is a simple website.
A dedicated plugin is not required.
Here is a simple LinkCrawler rule which will crawl all photos inside a "chapter".
If you want it to auto-crawl other types of links too (e.g. all chapters of a series), you need to create more LinkCrawler Rules.

Here is the above mentioned rule:
(Logging in this rule is enabled to find Cloudflare issues more easily!)
Code:
[
  {
    "enabled": true,
    "logging": true,
    "maxDecryptDepth": 1,
    "name": "isekaiscan.com grab all images of a chapter",
    "pattern": "https?://isekaiscan\\.com/manga/[a-z0-9\\-]+/chapter-[0-9]+(-[0-9]+)?/",
    "rule": "DEEPDECRYPT",
    "packageNamePattern": "<title>(.*?)</title>",
    "deepPattern": "id=\"image-[0-9]+\" data-src=\"([^\"]+)"
  }
]
Rule as plaintext for easier copy & paste:
pastebin.com/uYQMBJ2S

In my tests, I was able to crawl 1-2 galleries and then got blocked by Cloudflare so please don't expect too much here!
If the rule fails for you, please post a log so we can find out if your issue is Cloudflare related or not.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Last edited by pspzockerscene; 14.06.2021 at 17:34. Reason: Updated rule once again to also match on "chapter-55-55"
Reply With Quote
  #4  
Old 14.06.2021, 18:20
BJN01 BJN01 is offline
JD Adviser
  
Join Date: Jan 2020
Posts: 113
Default
down ??
I checked here too: **External links are only visible to Support Staff****External links are only visible to Support Staff**
but it's accessible.
I tried to open the site in question (isekaiscan.com) with other browsers and it work ...

so ...

Damned Cloudflare ! ( I read the topic of the link, and I posted a question. )




I understood that there is no hope for plugins, but pasting the links of the files manually they are downloaded without problems ; so couldn't you use a pair of matched linkcrawler?



I searched a bit in the html of the various pages (if they could be useful I also saved them as txt) and follow a pattern:


es.1)
ch 1
**External links are only visible to Support Staff****External links are only visible to Support Staff**
and the images are in:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

name for pachage --> from the link (h##ps://isekaiscan.com/manga/my-gift-lvl-9999-unlimited-gacha/chapter-1/)
file --> in the "index"/html they are :
h**ps://isekaiscan.com/wp-content/uploads/WP-manga/data/manga_60b5ff4fdc93f/9ba67ea562d2fae119bb1e0bd97cd7ac/***.jpg
with
<h**ps://isekaiscan.com/wp-content/uploads/WP-manga/data/manga_>+<13 alphanumeric >+</>+<32 alphanumeric >+</>+<file jpeg,gif,..>



es.2)
ch 10.2
**External links are only visible to Support Staff****External links are only visible to Support Staff**

and the images are in:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**


name for pachage --> from the link (h##ps://isekaiscan.com/manga/NAME/chapter-10-2/)
file --> in the "index"/html they are :
h**ps://isekaiscan.b-cdn.net/manga_5d8d9d73ef11f/7f15665706291cdcf916d0c11cab71da/***.jpg
with
<h**ps://isekaiscan.b-cdn.net/manga_>+<13 alphanumeric >+</>+<32 alphanumeric >+</>+<file jpeg,gif,..>



all the file list is in the <html / "index"> so using linkcrawler is it possible?
Attached Thumbnails
isekaiscan-mygiftl999.jpg   isekaiscan-unsuccessfullyetblabal10.2.jpg  
Attached Files
File Type: zip isekaiscantest.zip (969.2 KB, 0 views)
Reply With Quote
  #5  
Old 14.06.2021, 18:26
BJN01 BJN01 is offline
JD Adviser
  
Join Date: Jan 2020
Posts: 113
Default
SORRY !! i wasted a lot of time writing the post and haven't seen your answer.

now I sign your rule and I try
Reply With Quote
  #6  
Old 15.06.2021, 00:17
BJN01 BJN01 is offline
JD Adviser
  
Join Date: Jan 2020
Posts: 113
Default
i did some tests and LinkCrawler rule works super fine.
After a few attempts, i also added a packaging rule like:
Code:
condition name = isekaiscan Filename
url source  =(contains)=  **External links are only visible to Support Staff**file name = <jd:orgpackagename> -Page_<jd:orgfilename>
and with the renaming scripts I get the desired result

Spoiler:

- My Gift LVL 9999 Unlimited Gacha - ch01
--------My Gift LVL 9999 Unlimited Gacha - ch01-Page_01.jpg
--------My Gift LVL 9999 Unlimited Gacha - ch01-Page_02.jpg
--------My Gift LVL 9999 Unlimited Gacha - ch01-Page_03.jpg
---------..............................................
--------My Gift LVL 9999 Unlimited Gacha - ch01-Page_XX.jpg


so, many thanks again!
Last edited by BJN01; 17.06.2021 at 14:21.
Reply With Quote
  #7  
Old 15.06.2021, 10:28
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
  
Join Date: Mar 2009
Location: Germany
Posts: 82,262
Default
@BJN01: Thanks for the feedback great to hear it's working and you even made a packagizer rule by yourself

Yes, that su.... when writing long post/answer, just to realize that the other person already has answered/updated a post
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 15.06.2021, 12:50
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
  
Join Date: Mar 2009
Location: Deutschland
Posts: 74,658
Default
Nice!

Regarding Cloudflare:
Now I can even view/open the picture directurls just fine but their main website still seems to block my German IP via Cloudflare...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

« Previous Thread | Next Thread »
Thread Tools
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

All times are GMT +2. The time now is 23:16.
JDownloader Community Board - Archive - Top
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.