JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 13.01.2017, 23:04
Luke M Luke M is offline
Ultra Loader
 
Join Date: May 2012
Posts: 47
Default Crawler not finding link

The link looks like this **External links are only visible to Support Staff****External links are only visible to Support Staff**

It's inside some javascript, but the crawler should still find it, right? The only thing slightly unusual is that it's a https instead of http.

Last edited by raztoki; 13.01.2017 at 23:40.
Reply With Quote
  #2  
Old 13.01.2017, 23:43
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,556
Default

if you're deep analysing, typically this is done on url containing html which may or may not have javascript within. If the url is within this base page it should find it. If the url is within another <script src it wont be caught as deep analyse only checks the original url (doesn't follow).
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #3  
Old 14.01.2017, 00:49
Luke M Luke M is offline
Ultra Loader
 
Join Date: May 2012
Posts: 47
Default

The link is in the main page, yes. At least it is when I load the page with Firefox. It's possible that the site is serving jdownloader a different page. Can I tell jdownloader what browser to emulate?
Reply With Quote
  #4  
Old 14.01.2017, 00:59
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,556
Default

@Luke_M
simple answer is no
not so simple answer, yes. (via proxy, or editing our source code).

can you provide the page you're trying to deep analyse?

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #5  
Old 14.01.2017, 20:44
Luke M Luke M is offline
Ultra Loader
 
Join Date: May 2012
Posts: 47
Default

What's the easiest way to test the crawler? Can I point it to a local file somehow, or do I have to install a web server?
Reply With Quote
  #6  
Old 14.01.2017, 23:38
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,556
Default

setup eclipse & jd workspace ? scan the page/url that you have issues with?

That's what I was going todo for you, once I was given the page/url to test with.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #7  
Old 16.01.2017, 16:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

Please provide some example link, then we can check/fix or add support/help.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 19.01.2017, 10:33
lightwave3d lightwave3d is offline
JD Addict
 
Join Date: May 2009
Location: jdownloader.org
Posts: 173
Default

hello, also having problem like this....

JD use to be able to crawl links from this site: **External links are only visible to Support Staff****External links are only visible to Support Staff**

but now it wont.

Last edited by raztoki; 19.01.2017 at 10:42.
Reply With Quote
  #9  
Old 20.01.2017, 08:30
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,307
Default

Do you use "Add new links"?
Please provide example links, links with "__" should contain external link that can be found with "Add new links".
__________________
FAQ: How to upload a Log
Reply With Quote
  #10  
Old 20.01.2017, 15:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

You can easily create support for this with LinkCrawler Rules (use board search)
You have to create rules for the '__' links
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 03:58.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.