JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 11.10.2016, 14:24
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default Problem while analyse a page with jdownloader 2

Hi everyone!
I use jdownloader since years without problem and allways surprised by its power!!!

Unfortunately since the last update, when I try to analyse and find links through pages, it doesn't find anymore the links hosted on these pages!! And before it worked like a charm!!
What could I do?
Thanks alot for your help
Reply With Quote
  #2  
Old 11.10.2016, 14:43
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Please provide example links. Nothing has changed parser, so must be something with the website you are trying to parse
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 11.10.2016, 15:07
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

Yes thanks, but on the pages if I select and copy links, jdownloader detect it immediately:
for example:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

It should find:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
and
**External links are only visible to Support Staff****External links are only visible to Support Staff**

but it only find images from the pages and audio files from bandcamp, I don't understand what has changed. Last time I used it it was middle of july and I don't find any changement on the website.
Reply With Quote
  #4  
Old 11.10.2016, 15:07
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #5  
Old 11.10.2016, 15:56
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,507
Default

Have you checked page source of that link to find those 2 llinks?
I can't find it.
__________________
FAQ: How to upload a Log
Reply With Quote
  #6  
Old 11.10.2016, 18:25
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

Yes, when I see the code source (with Ctrl+U on firefox), we can find these links (I found it using Ctrl+F)
On line 496 there are the 3 links:It's a huge line ! firstly Bandcamp (which is found by JD) then zippyshare and psy-music.ru (which are not found by JD).Strange!
thank you

Last edited by flowlapache; 11.10.2016 at 18:33.
Reply With Quote
  #7  
Old 11.10.2016, 20:07
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

You need to add LinkCrawler Rules to tell JDownloader to auto parse this type of url.
JDownloader does not support this url and therefore does not autohandle/parse it.
With a customized LinkCrawler Rule (use board search) you can tell JDownloader how to handle this url.

You can also Copy/Paste the URL into JDownloader to force auto deep decrypt of the url
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 11.10.2016, 20:10
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

Ok, I will try this "link crawler rule". But it strange because it works since 4 years at least!! Maybe they change the type of url this summer. Thank you for your quick answers and help!
Reply With Quote
  #9  
Old 11.10.2016, 20:17
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

JDownloader does not support this type of url and never did.
Copy/Paste will work fine and a Linkcrawler Rule too
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 11.10.2016, 20:20
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

Ok, so the website have changed without something changed for me.
When I try with deep decrypt, it doesn't find what I search for.
I'm looking for linkcrawler Rule, JD is really surprising!
Reply With Quote
  #11  
Old 11.10.2016, 20:21
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Simply copy/paste the url into JDownloader, it will deep decrypt and show all supported links.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 11.10.2016, 20:26
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

Thats what I did, but it doesn't find the two links I would like. And I'm not sure what I should change in the linkcrawler rule. Can I create a new rule specific to this website? It seems to be allways on the line 96 with same type of codes, on every pages from this website (psy-music.ru) If I copy this line, JD find immediately what I look for!!
I don't find on other thread how to do..I see often "mail to support@..." to obtain help about making a rule for this website...

edit: I can't find how to tell JD to auto parse pages from this website. I look some other rules for specific sites but I don't understand which code I should add to linkcrawler rules....

Last edited by flowlapache; 11.10.2016 at 22:14.
Reply With Quote
  #13  
Old 11.10.2016, 23:49
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

I have forgotten something important! (thanks to mgpai from JDchat)
links are not visible unless I login. That's why you didn't find links when you asked me.
I'm still trying to make a rule for linkcrawler.

For this login, it was already this since years.When I logged and had the cookie of the website, it was visible for JD.
But now no more.

I tried this, but it's not a valid rule...
Code:
[ {
  "enabled" : true,
  "name" : "psy-music",
  "pattern" : " psy-music\\.ru/news/.+/[0-9-]+ ",
  "rule" : "DEEPDECRYPT",
} ]

I will sleep, maybe tomorrow regex will be clearer...

Last edited by flowlapache; 12.10.2016 at 00:08.
Reply With Quote
  #14  
Old 12.10.2016, 01:29
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

pattern requires protocol prefix
Code:
https?://
it shouldn't contain white space in the inside of quotation mark.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 12.10.2016 at 01:42.
Reply With Quote
  #15  
Old 12.10.2016, 09:40
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by flowlapache View Post
Code:
[ {
  "enabled" : true,
  "name" : "psy-music",
  "pattern" : "https?://psy-music\\.ru/news/.+/[0-9-]+ ",
  "rule" : "DEEPDECRYPT",
} ]
see fixed version
__________________
JD-Dev & Server-Admin
Reply With Quote
  #16  
Old 12.10.2016, 12:11
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

Ok, thanks guys, I'm learning....
I tried with this fixed version (without and with the space after the "+" in pattern) but JD doesn't accept it.
And one thing more, I have already a linkcrawler rule (I don't know where and when it comes from) for zip format:
Code:
[ {
  "enabled" : true,
  "maxDecryptDepth" : 0,
  "id" : 1476219085998,
  "name" : "psy_music",
  "pattern" : " psy-music\\.ru/news/.+/[0-9-]+ ",
  "rule" : "DEEPDECRYPT",
  "packageNamePattern" : null,
  "formPattern" : null,
  "deepPattern" : null,
  "rewriteReplaceWith" : null
} ]
I don't understand where to put my Rule for psy-music.ru? Before, After or Instead of this "zip" rule?
I tried the 3 solutions, but because it's not valid, the value doesn't change (only zip rule stays...)
Reply With Quote
  #17  
Old 12.10.2016, 12:13
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

your pattern is still invalid!
it must begin with https?://

an url does not start with _(space)
__________________
JD-Dev & Server-Admin
Reply With Quote
  #18  
Old 12.10.2016, 13:21
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

Sorry, it wasn't this one, I copied the (first) bad one.
I would like to show the existing one for zip format:
Code:
[ {
  "enabled" : true,
  "maxDecryptDepth" : 2,
  "id" : 1433746432948,
  "name" : "Learned file extension:zip",
  "pattern" : "(?i).*\\.zip($|\\?.*$)",
  "rule" : "DIRECTHTTP",
  "packageNamePattern" : null,
  "formPattern" : null,
  "deepPattern" : null,
  "rewriteReplaceWith" : null
} ]
Code:
[ {
  "enabled" : true,
  "name" : "psy-music",
  "pattern" : "https?://psy-music\\.ru/news/.+/[0-9-]+",
  "rule" : "DEEPDECRYPT",
} ]
still invalid but different.

And must I replace the ".zip" rule or I add it wrinting something like:
Code:
[ {
  "enabled" : true,
  "name" : "psy-music",
  "pattern" : "https?://psy-music\\.ru/news/.+/[0-9-]+",
  "rule" : "DEEPDECRYPT",
} ]
[ {
  "enabled" : true,
  "maxDecryptDepth" : 2,
  "id" : 1433746432948,
  "name" : "Learned file extension:zip",
  "pattern" : "(?i).*\\.zip($|\\?.*$)",
  "rule" : "DIRECTHTTP",
  "packageNamePattern" : null,
  "formPattern" : null,
  "deepPattern" : null,
  "rewriteReplaceWith" : null
} ]
I tried the 3 solutions, but because it's not valid, the value doesn't change (only zip rule stays...)
Reply With Quote
  #19  
Old 12.10.2016, 13:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

[{rule1},{rule2},{rule3}]
and NOT
[{rule1}][{rule2}][{rule3}]
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 12.10.2016, 14:50
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

Ok thanks! I see the importance of "," and priority of "[" and "{". I will try
Reply With Quote
  #21  
Old 12.10.2016, 15:04
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

It is json format. You can google for json beautifier. If it *eats* your input, it is valid json
__________________
JD-Dev & Server-Admin
Reply With Quote
  #22  
Old 13.10.2016, 15:06
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

Quote:
[ {
"enabled" : true,
"maxDecryptDepth" : 0,
"id" : 1476354396774,
"name" : "psy-music",
"pattern" : "http?://psy-music\\.ru/news/.+/[0-9-]+",
"rule" : "DEEPDECRYPT",
"packageNamePattern" : null,
"formPattern" : null,
"deepPattern" : null,
"rewriteReplaceWith" : null
}, {
"enabled" : true,
"maxDecryptDepth" : 2,
"id" : 1433746432948,
"name" : "Learned file extension:zip",
"pattern" : "(?i).*\\.zip($|\\?.*$)",
"rule" : "DIRECTHTTP",
"packageNamePattern" : null,
"formPattern" : null,
"deepPattern" : null,
"rewriteReplaceWith" : null
} ]
Yep, so with this code, it works but it's like before.The deep decrypt doesn't find the good links (logged with browser and with JD as basic authentification).

For the max decrypt depth, 0 is 0 or it's everything? Because now (with 0), when one link is parsed, it finds a lot of differents bancamp links until endless (maybe it parses all thye site). I will try with depth=1

Last thing, do you know if it's possible in a browser (firefox) to "copy the code source of the link" with a right clic?
It could save me to open each page and making Ctrl+A then Ctrl+C to obtain good links in JD.

Last edited by flowlapache; 13.10.2016 at 15:50.
Reply With Quote
  #23  
Old 13.10.2016, 15:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Depth=0 -> only parse the input url
Depth=1 -> also parse first level of found links

Ctrl+A = Select All + Ctrl+C = Copy = Copy all to Clipboard. Browser will also put html code into clipboard.

It works perfectly fine for me. I don't need any logins. Browser and JDownloader do show the links just fine without any logins?!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #24  
Old 13.10.2016, 22:52
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

WHAAAT!!! you are experiencing what happened since ever!! You really obtain the http link from psy-music.ru and zippishare or rusfolder and others?
it's crazy, I have only, images, bandcamp audio files, videos but no archive!!
I'm lost with this new news!
Reply With Quote
  #25  
Old 13.10.2016, 22:55
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

oops, sorry (i can't edit before it's approved...)I misunderstood...
In fact yes, it works with control+A when we are on the page.
But I would like to make it like before, just put the link (of the page) in JD without need to open the page... like parsing and analysing the page to find these archives!
Reply With Quote
  #26  
Old 14.10.2016, 14:44
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

I don't understand why "deepdecrypt" don't find it.
Is there a rule saying to JD to parse the code source of a page (Ctrl+A, paste by itself), because it's just what I want.

I search how "copy the code source of a page without open this page in a browser" but for now I have nothing. It's boring, it's only 2 or 4 actions to do what I want ( a link>the source code>JD find the archives!), But I can't do it manually for 100 pages!

I continue to search this "code source copy" feature in firefox or otherz...
In all cases, thank you for your support, I'm really goodly surprised by the support by chat, or board. It's so better than a lot of applications or webshops which are not free moreover!!!
Reply With Quote
  #27  
Old 14.10.2016, 15:17
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,507
Default

Do you have to log in to see those links?
__________________
FAQ: How to upload a Log
Reply With Quote
  #28  
Old 15.10.2016, 14:17
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

Thanks to come! Yes I mentionned it a bit before...

Normally I log in, then I just put links of all pages I want in the "parser" of JD and every archives are found!

it works with Select All + Ctrl+C (even without be logged! like tried Jiaz) after have opened the page.

But now, I can't obtain archives without open each page.

Even with the deepdecrypt linkcrawler rule
And I didn't find how to make a "copy the code source of this link" in Firefox. Thing which could do the job...
Reply With Quote
  #29  
Old 15.10.2016, 14:55
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,507
Default

I think previously the link was shown to everybody without login, not anymore.
I am not sure, but you can try to enter your credentials in Basic Authentication and try to parse it again.
__________________
FAQ: How to upload a Log
Reply With Quote
  #30  
Old 15.10.2016, 17:38
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

Yes, I try it firstly but it changes nothing.

The website has changed but I don't know how type... Previously links were already hidden to public.Like now, it has been allways necessary to log to see the links, mostly for the http links from psy-music.ru...
Reply With Quote
  #31  
Old 17.10.2016, 17:54
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

During my tests the links were public available without login. I guess this is changed now as I no longer can see them. When you are logged in, are the links plain visible or via rightclick or do you need to do a left click?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #32  
Old 18.10.2016, 01:03
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

I guess, it's the trick.

We need to left click to open it (or launch it for the http link). It's not plain visible. We see the adress only with the mouse on the link.
it's directly visible in the code source.

JD find it if we select it and copy.
Reply With Quote
  #33  
Old 18.10.2016, 10:32
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

In that case I would stick to *select and copy* as it seems to work
When you just copy the url then the links are no longer available as the site requires a login and your JDownloader is not logged in.
Is that solution acceptable for you?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #34  
Old 20.10.2016, 15:42
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

Hi, thank you.

If I found nothing else, yes I will use the "open>select all>copy".
But I was used to go every one or two months on this website et download around 100 links....Which is really "hard and time eater" if I must open each page and select+copy...instead of only copy each adress like before...

In the same time I'm trying with "menu wizard" on firefox to add a "copy the source code of the link" to my right click. If it works, I thing I could obtain links without open every pages.
For now the "source results" are different than the source on the opened page.


But they probably make it weel good to force to open each page to obtain the links...
maybe they are no more JD friendly!!^^
Reply With Quote
  #35  
Old 20.10.2016, 15:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

We are currently working on updated browser extensions, then it should be possible to add links from pages with logins as well
__________________
JD-Dev & Server-Admin
Reply With Quote
  #36  
Old 21.10.2016, 15:11
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,533
Default

@flowlapache,
Add this script to event scripter. Change the username and password. You should now be able to grab the links which were otherwise not detected by JD. Will only detect single URLs. Create linkfilter rule to block unwanted links.

@jiaz,
Thanks for the browser API and code
Code:
// Crawler: psy-music.ru
// Trigger required: "New Crawler Job"

var sourceURL = job.text;
var matchURL = (/^http:\/\/psy-music.ru\/news\/.+\/[\d-]+$/).test(sourceURL);

try {
    if (matchURL) {
        var mainURL = "**External links are only visible to Support Staff**;
        var username = "myUser"; // Change username
        var password = "myPasss"; // Change password

        var br = getBrowser();
        var postURL = mainURL + "index/sub/";
        var postData = "user=" + username + "&password=" + password + "&rem=1&a=2&ajax=1&rnd=691";
        var mainPage = br.getPage(mainURL);
        var mainPage = br.postPage(postURL, postData);
        var links = br.getPage(sourceURL);

        callAPI("linkgrabberv2", "addLinks", {
            "links": links
        });

        job.setText("");
    }
} catch (e) {
    alert("Error occured while adding \"" + sourceURL + "\". Please try again");
}
Reply With Quote
  #37  
Old 24.10.2016, 14:03
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

OOh yeah guys!!

Thanks so much! You made it!

It works very well and fast! Thank you for this nice support! very efficient!
Reply With Quote
  #38  
Old 24.10.2016, 14:18
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Thanks for the feedback
__________________
JD-Dev & Server-Admin
Reply With Quote
  #39  
Old 29.06.2022, 20:50
flowlapache flowlapache is offline
DSL Light User
 
Join Date: Oct 2016
Posts: 31
Default

I try the same thing for another website but it doesn't work an dI don't understand why...
Reply With Quote
  #40  
Old 29.06.2022, 20:53
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@flowlapache: can you provide example links?
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 23:54.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.