JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 16.04.2022, 13:32
DeusExBestia DeusExBestia is offline
DSL User
 
Join Date: Feb 2018
Posts: 36
Question Redfin.com and Zillow.com and other realty sites

So one of the favorite pastimes of my husband and I is to browse super fancy over priced well out of our price range homes on websites like Redfin or Zillow. Problem is that some of our favorites eventually get removed because they are either sold or no longer on the market.
It’s just something we poor peasants like to do so we can see what the rich buttheads in society live like. I would like to request that we add support for those sites. Maps and address I’m not not that concerned with saving, because it’s not like I can go visit them, and it could lead to privacy concerns. But the pictures of the homes; how they’re presented for sale, and even the blue prints they may upload alongside the pictures can help a man dream and fantasize about a fancier, and maybe better future.

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #2  
Old 16.04.2022, 13:50
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@DeusExBestia: so basically you are interested in downloading the images and, if available, the blueprints, right?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 16.04.2022, 14:00
DeusExBestia DeusExBestia is offline
DSL User
 
Join Date: Feb 2018
Posts: 36
Default

Yeah! The blueprints are uploaded as pictures as well, so just whatever they have in the house picture gallery. Maybe a .txt of the house description and whatever? “Beds: 7482, baths: 1.”
I don’t want to be all picky, but I presume if it’s at all possible they’d be saved as one house/address per package?
Reply With Quote
  #4  
Old 16.04.2022, 14:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@DeusExBestia: i will check, we have public holiday here in germany on monday, so expect answer by next week
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 16.04.2022, 14:33
DeusExBestia DeusExBestia is offline
DSL User
 
Join Date: Feb 2018
Posts: 36
Default

Awesome! Thank you kindly, and enjoy your holiday/weekend!
Reply With Quote
  #6  
Old 04.05.2022, 14:36
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Hi again and sorry it seems like this thread has been forgotten which rarely happens here.

Both websites look pretty simple so you should be able to do this by either only using our default deep-parser (= just add the links and let JD scan for results) or using LinkCrawler Rules.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #7  
Old 04.05.2022, 14:57
DeusExBestia DeusExBestia is offline
DSL User
 
Join Date: Feb 2018
Posts: 36
Default

No worries! I’ve been dealing with a new puppy so lots of things have fallen by the wayside. Thank you for replying.

I will give them a shot once I get home after work.
Reply With Quote
  #8  
Old 04.05.2022, 16:24
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Okay.
Just come back to us if you can't get it working.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #9  
Old 11.05.2022, 12:08
DeusExBestia DeusExBestia is offline
DSL User
 
Join Date: Feb 2018
Posts: 36
Default

So I tried it out, Zillow comes up as a mess. If I just tell JD to crawl it without putting in a manual packagename it comes out as a giant mess, tons of folders with jpg/png and webp files, for every resolution available from the site, and it grabs all the analytics and some random stuff from their tiktok and their twitter. If I manually put in a package name everything for zillow is neatly tucked into the folder, with dozens of copies of each picture at every resolution, leading to thousands of files downloaded.

Redfin, for the same address gave a ton of twitter links? Still waiting for it to finish downloading.
Reply With Quote
  #10  
Old 11.05.2022, 12:28
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Looks to me like you didn't understand any of the concepts of our LinkCrawler Rules(?)
Please post the LinkCrawler Rule you've created which lead to those results.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #11  
Old 11.05.2022, 14:30
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Here is a really really basic rule for redfin.com that simply grabs all .jpg images from the source code and puts them into a single package.
You should adjust the deepPattern to further limit the results.
Rule:
Code:
[
  {
    "enabled": true,
    "logging": false,
    "maxDecryptDepth": 1,
    "name": "example rule for redfin.com",
    "pattern": "https?://(www\\.)?redfin\\.com/.*/home/[0-9]+",
    "rule": "DEEPDECRYPT",
    "packageNamePattern": "<title>(.*?)</title>",
    "deepPattern": "(https?[^\"]+\\.jpg)"
  }
]
Rule as plaintext for easier copy & paste:
pastebin.com/raw/tz7B45Pq

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by Jiaz; 11.05.2022 at 15:28.
Reply With Quote
  #12  
Old 26.05.2022, 12:03
DeusExBestia DeusExBestia is offline
DSL User
 
Join Date: Feb 2018
Posts: 36
Default

Quote:
Originally Posted by pspzockerscene View Post
Looks to me like you didn't understand any of the concepts of our LinkCrawler Rules(?)
Please post the LinkCrawler Rule you've created which lead to those results.

-psp-
You're not wrong. I don't understand any of the LinkCrawler rules, lol.

I added the linkcrawler rules you wrote up, it only downloaded 17 pictures, and none of the ones of the house. Looked like it grabbed a bunch of the sample houses on there. I'll try giving another crack at it once I can dedicate time to it.
Reply With Quote
  #13  
Old 26.05.2022, 20:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@DeusExBestia: I will try to help when I find some free time for this
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 30.05.2022, 21:25
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

@DeusExBestia
Indeed it seems like the way those image-URLs are encoded cannot yet be handled by our generic parser.
Here are possible solutions/workarounds you can do even without us changing anything (LinkCrawler rule is not required in those cases):
  • Find generic browser addons that can extract image/media URLs so you can copy them and add them to JD this way
  • or: open the image gallery in browser -> Press CTRL + A to mark all, then CTRL + C to copy them --> Add them to JD and use the available filters to filter out everything except images --> Download them and check which of the results are those you really wanted to have

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 03:19.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.