JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 04.08.2020, 17:33
Necrontyr Necrontyr is offline
Modem User
 
Join Date: Aug 2020
Posts: 3
Default Custom referer in crawljob file

I created a script that does some web scrapping and parses a certain web pages to get the (direct) download links. The script then creates a crawljob JSON file that I load into jdownloader.
It was working fine until recently the host admin decided to restrict access to files through Cloudflare if the request headers doesn't contain the website url as a Referer.
Is there a way to set a custom referer in the crawljob file?

Note: Selecting the link in the browser and using Ctrl+C and then Ctrl+V on JD loaded the link just fine. However right clicking and selecting "copy link" in the context menu gives the 403 Cloudflare error. I guess that in the first way, the referer is copied to the clipboard and imported into JD. But doing manual work in browser is not the purpose of my script.

Last edited by Necrontyr; 17.08.2020 at 20:25.
Reply With Quote
  #2  
Old 05.08.2020, 06:56
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,201
Default

hi to answer your query, it depends.
a) are you just deep analysing ?
or
b) do we have a plugin for said website. some plugins have url listener to additional parameter like &referrer=websiteurlencoded.

clipboard ctrl c and v will have access to htmlflavour which your browser provides additional info than say text you copied, html behind that selection is also copied to a different part of the clipboard event.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #3  
Old 05.08.2020, 13:59
Necrontyr Necrontyr is offline
Modem User
 
Join Date: Aug 2020
Posts: 3
Default

The website is a Wordpress blog I access though its API, get the content of the post and parsing it as html. It's a website I don't feel confortable sharing here.
The files are stored in the blog owner's private host, it's not a filesharing service.
The browser by default sends the referer when clicking in the link. The link also has referrerpolicy="origin". If I set it to "no-referrer", or remove the attribute and set a rel="noreferrer", then I get the 403. The referer must be sent via HTTP headers, setting it in GET or POST doesn't work. That's why I need JD to import the referrer from a file like it does from clipboard.

I got the crawljob template from here: https://board.jdownloader.org/showpo...21&postcount=4
I don't know how hard would it be to add another field to do that.

Last edited by Necrontyr; 05.08.2020 at 14:15.
Reply With Quote
  #4  
Old 17.08.2020, 20:24
Necrontyr Necrontyr is offline
Modem User
 
Join Date: Aug 2020
Posts: 3
Default

Just letting you know that I made the script to download the files too, so I don't need JD anymore. But anyway, it would be a feature that you might want to add.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 05:25.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.