JDownloader Community - Appwork GmbH
 

Notices

Closed Thread
 
Thread Tools Display Modes
  #41  
Old 22.02.2016, 21:00
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Wait for next core update
__________________
JD-Dev & Server-Admin
  #42  
Old 23.02.2016, 13:07
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Incomplete regex matching.
Jiaz could write correct(If you find free time), I will be very grateful.

  #43  
Old 23.02.2016, 13:33
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

maybe ???

Quote:
javascript\:__doPostBack\('lbNext',''\)
  #44  
Old 23.02.2016, 13:50
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Decrypt only 1 page (not work multipage)

**External links are only visible to Support Staff**...
**External links are only visible to Support Staff**...


Code:
[ {
"enabled" : true,
"maxDecryptDepth" : 1,
"id" : 1454334584729,
"name" : null,
"pattern" : "javascript:[\\w]+\\('lbNext',''\\)",
"rule" : "DEEPDECRYPT"
} ]
  #45  
Old 23.02.2016, 14:00
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

crawler rules are for two things
URL patterns, for URL listening, as in trigger when you copy URL.. eg. **External links are only visible to Support Staff****External links are only visible to Support Staff**
and if they are direct downloadable or deep decrypt task.

for deep decrypt, our parser looks for links and returns them all.
Its not designed to search for patterns that you provide it with!
It wont process javascript!

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
  #46  
Old 23.02.2016, 15:32
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

raztoki -
Some sites dynamicalu generate web content by using JavaScript. This cause JD2 to fail to detect when loading multipages.
can not force JD2 to process the JavaScript?(Eg. The simulation mouse "click")
:\
  #47  
Old 23.02.2016, 15:36
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

I can only advise on vague posts (millions of them), since you rarely give working links to sites your trying to support. I gave help based on the feature you're trying to use outside of design.

If you want complicated function, outside of the simple function that linkcrawler rules provide, _you_ will need to make your own hoster/decrypter plugins.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
  #48  
Old 23.02.2016, 15:42
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

I know, in this case, the script is not enough, you probably need an additional java class code...
  #49  
Old 23.02.2016, 16:24
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

In theory, it should work with the parser, but unfortunately it did not work.
I have no other ideas, but do not give up and still looking solutions

  #50  
Old 23.02.2016, 16:44
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

If matched, add link to download queue use backreferences as {$1}, OR {$2}, etc.?
  #51  
Old 23.02.2016, 16:48
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

JDownloader cannot evaluate Javascript without a real plugin. Only visible links can be processed!
__________________
JD-Dev & Server-Admin
  #52  
Old 23.02.2016, 17:15
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

I used a parser with "Evaluate calculation scripts" (Enabled)
but I get an error and abort parse.

href="javascript:__doPostBack('lbHome','')">Home</a><img src="../Images/spaceer"
  #53  
Old 23.02.2016, 17:22
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Code:
import  java.util.regex.Matcher;
import  java.util.regex.Pattern;

/**
   <P>{@code java ParseForclosureResultsXmpl}</P>
 **/
public class ParseForclosureResultsXmpl  {
   public static final void main(String[] igno_red)  {
      String sLS = System.getProperty("line.separator", "\n");

      StringBuilder sdInput = new StringBuilder().
         append("<a id=\"SheetContentPlaceHolder_ctl00_gvForeclosureResutls_lbCaseNum_0\" href=\"javascript:__doPostBack('ctl00$SheetContentPlaceHolder$ctl00$gvForeclosureResutls$ctl02$lbCaseNum','')\" style=\"display:inline-block;width:100px;\">CV-13-798497</a>").append(sLS).
         append("              </td><td align=\"center\">488-05-029</td><td align=\"center\">I</td><td align=\"center\">01/02/2013</td>").append(sLS).
         append("  </tr><tr style=\"background-color:Gainsboro;\">").append(sLS).
         append("      <td align=\"left\">UNKNOWN HEIRS, ETC OF D.C. RUFUS, ET AL  </td><td align=\"left\">10603 HAMPDEN AVENUE</td><td align=\"center\">CLEVELAND</td><td align=\"center\">44108-0000</td><td align=\"center\">").append(sLS).
         append("                  <a id=\"SheetContentPlaceHolder_ctl00_gvForeclosureResutls_lbCaseNum_1\" href=\"javascript:__doPostBack('ctl00$SheetContentPlaceHolder$ctl00$gvForeclosureResutls$ctl03$lbCaseNum','')\" style=\"display:inline-block;width:100px;\">CV-13-798498</a>").append(sLS).
         append("              </td><td align=\"center\">109-16-094</td><td align=\"center\">A</td><td align=\"center\">01/02/2013</td>").append(sLS).
         append("  </tr><tr style=\"background-color:LightGrey;\">").append(sLS).
         append("      <td align=\"left\">SHARECE MILLER, ET AL  </td><td align=\"left\">13514 ALVIN AVENUE</td><td align=\"center\">GARFIELD HTS</td><td align=\"center\">44105-0000</td><td align=\"center\">").append(sLS).
         append("                  <a id=\"SheetContentPlaceHolder_ctl00_gvForeclosureResutls_lbCaseNum_2\" href=\"javascript:__doPostBack('ctl00$Shee").append(sLS);

      String sRqdValuePrefix = "ForeclosureResutls_lbCaseNum_";
      Pattern checkerLoopPattern = Pattern.compile(sRqdValuePrefix + "\\d");
      Matcher m = checkerLoopPattern.matcher("");  //Unused. so the matcher can be reused in the loop.

      int iLn = 0;
      String[] asInput = sdInput.toString().split(sLS);
      for(String s : asInput)  {
         iLn++;    //1st iteration: Was zero, now 1

         //Resuing matcher instead of retrieving new one from Pattern each iteration
         m.reset(s);

         if(m.find())  {
            int iCheckerNumber = Integer.parseInt(s.substring(m.start() + sRqdValuePrefix.length(), m.end()));
            System.out.println("Found on line " + iLn + ", at index " + m.start() + " with checker number " + iCheckerNumber);
         }
      }
   }
}
  #54  
Old 23.02.2016, 19:03
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

E.g. On this page JD2 works perfectly
So there is some exceptions, it works.
**External links are only visible to Support Staff****External links are only visible to Support Staff**
  #55  
Old 24.02.2016, 11:50
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Quote:
Originally Posted by Jiaz View Post
JDownloader cannot evaluate Javascript without a real plugin.
Only visible links can be processed!
I was able to process javascript parser ~880,000 links direct links, now I can easily add links to JD2

Mini-example:
4800 links
**External links are only visible to Support Staff****External links are only visible to Support Staff**
  #56  
Old 26.02.2016, 18:32
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default Deep-Decrypt multipage - issue

What is wrong?
:confused:


Code:
[ {
  "enabled" : true,
  "maxDecryptDepth" : 1,
  "id" : 1454334584729,
  "name" : null,
  "pattern" : "/p/(\\d+)",
  "rule" : "DEEPDECRYPT"
} ]
**External links are only visible to Support Staff****External links are only visible to Support Staff**
  #57  
Old 26.02.2016, 18:41
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

"pattern" wrong?

Code:
/p/(\\d+)
.*?/p/(\\d+)
.+/p/(\\d+)
  #58  
Old 26.02.2016, 18:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

maxDecryptDepth=1 -> it will not deep decrypt other pages
You need to increase this value
__________________
JD-Dev & Server-Admin
  #59  
Old 26.02.2016, 19:02
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Regex needs for full-size photos because detect very small files.


Quote:
bizarre-collection\\.biz/d/.+\\.jpg$
  #60  
Old 26.02.2016, 19:04
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Quote:
Originally Posted by Jiaz View Post
maxDecryptDepth=1 -> it will not deep decrypt other pages
You need to increase this value
Unlimited "multipage" Value: 0 ?
  #61  
Old 26.02.2016, 19:07
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

there is no unlimited, you can simply set to high value. I doubt that there are millions of pages :p
__________________
JD-Dev & Server-Admin
  #62  
Old 26.02.2016, 19:23
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Deepdecrypt only detects "thumb"

**External links are only visible to Support Staff****External links are only visible to Support Staff**
  #63  
Old 26.02.2016, 19:24
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

because the full images are build via javascript.
__________________
JD-Dev & Server-Admin
  #64  
Old 26.02.2016, 19:28
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Javascript is not a problem
If I knew the regex to full-size photos is parser all links to the full-size photos.
Only needs a regex to match the full-size image.
  #65  
Old 26.02.2016, 19:35
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Example Full-Size link:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
  #66  
Old 26.02.2016, 19:40
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

yes, yes, yes !.
I tested this regex works detects full-size photos.


**<bizarre-collection.biz/thumb/>
  #67  
Old 27.02.2016, 05:09
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

once again, far as I'm aware, linkcrawler rules do not look for patterns within pages, they return all urls, and JD will add all that is 'supported'. The pattern you provide is to scan specific urls added to program (either clipboard, add links dialog, etc). Pattern as I see it isn't correct, you need at least "http.+/p/\\d+". () are not required in your patterns
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
  #68  
Old 28.02.2016, 11:38
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Quote:
Originally Posted by raztoki View Post
once again, far as I'm aware, linkcrawler rules do not look for patterns within pages, they return all urls, and JD will add all that is 'supported'. The pattern you provide is to scan specific urls added to program (either clipboard, add links dialog, etc). Pattern as I see it isn't correct, you need at least "http.+/p/\\d+". () are not required in your patterns
"http.+/p/\\d+"

Regex works, but not decrypts all multipages.
JD2 shows 240 links
Other parser ~537 links

example link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
  #69  
Old 28.02.2016, 11:41
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Probably should show.
I calculated
270 + 20 + 16 = 316 links
  #70  
Old 28.02.2016, 11:44
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

1 page = 20 images
28 page (1...28) = 280 images + first page(0) (20 images) + last page (16 images) = ?
  #71  
Old 28.02.2016, 13:16
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

once again 1 page = unknown results, since it doesn't filter results for specific item, it returns all img/hrefs/etc> then returned results match given plugins will show download link in GUI. If you want isolated/filtered download links please create decrypter plugins.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
  #72  
Old 28.02.2016, 13:38
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

I know it will not work in JD2, but some tools have built such a possibility for each multi-page

  #73  
Old 28.02.2016, 13:40
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Example:

*bootshare\\.com/(.*)\?pageNum\\_[a-z]+=(\\d+)
  #74  
Old 28.02.2016, 15:28
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

I have no idea what is wrong.

Error Occured:


[ {
"enabled" : true,
"maxDecryptDepth" : 1,
"id" : 1454334584729,
"name" : null,
"pattern" : "http.+(\\&start=(\\d+)|\\&page=(\\d+)$|\\?page=(\\d+)$|/page-(\\d+)$|\\?page=(\\d+)\\&|/page/(\\d+)/$|/page(\\d+)$|/page/(\\d+)\\?|/page(\\d+)/|\\?PageNo=(\\d+)|\\?paged=(\\d+)|/page/(\\d+)$|\\?st=(\\d+)|\\&s=(\\d+)+|\\&p=(\\d+)|\\&cat=(\\d+)|\\?_pgn=(\\d+)|\\?id=(\\d+)|/p/(\\d+)|\\?go=(\\d+)",
"rule" : "DEEPDECRYPT"
} ]
  #75  
Old 28.02.2016, 15:31
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

without first/end patern ? ()
  #76  
Old 28.02.2016, 15:36
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

my regex Lines, wants to join

hxxp.+MULTI-REGEX

http://jdownloader.org/pastebin/134850
  #77  
Old 28.02.2016, 15:37
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Correct...

"http.+\\&start=(\\d+)|\\&page=(\\d+)$|\\?page=(\\d+)$|/page-(\\d+)$|\\?page=(\\d+)\\&|/page/(\\d+)/$|/page(\\d+)$|/page/(\\d+)\\?|/page(\\d+)/|\\?PageNo=(\\d+)|\\?paged=(\\d+)|/page/(\\d+)$|\\?st=(\\d+)|\\&s=(\\d+)+|\\&p=(\\d+)|\\&cat=(\\d+)|\\?_pgn=(\\d+)|\\?id=(\\d+)|/p/(\\d+)|\\?go=(\\d+)"
  #78  
Old 28.02.2016, 15:42
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Is this again your RegEx special question or is that actually JDownloader related ?

.....
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
  #79  
Old 28.02.2016, 16:01
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

use regex budy/or regex testing website, it would have told you that your patterns are invalid!

missing closing ) on the first , second is even worse as your not sharing the protocol.+ component so 95% pattern is invalid.

no more regex based questions.. please use software or websites to test your patterns.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Closed Thread

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 00:29.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.