GreatRealEstateAgentsBot
What our web crawler does, how it identifies itself, what it collects and does not collect, how it respects robots.txt, and how to opt a site out.
Last updated
What it is
GreatRealEstateAgentsBot is the web crawler operated by Great Real Estate Agents. It visits real estate agency websites to find the agency's public team page and collect the professional details of the agents who work there, so that every agent in a suburb can be represented on our ranking pages whether or not they have claimed a profile.
How it identifies itself
User agent string:
GreatRealEstateAgentsBot/1.0 (+https://greatrealestateagents.com.au/bot/)
Requests come from a published set of IP addresses, listed at https://greatrealestateagents.com.au/bot/ips.txt, so you can verify that a request claiming to be our bot is ours.
What it collects
From agency team or "our people" pages: agent name, role, agency, office address, business phone number, business email address, licence number where published, and the profile page URL. From agency contact pages: office address, phone number and ABN where published.
What it does not collect
- Review text from any website, including RateMyAgent, realestate.com.au, Domain and Google Maps. These hosts are on a denylist the crawler cannot override.
- Listing data from property portals.
- Personal information about private individuals, including vendors and buyers.
- Anything from pages excluded by robots.txt.
- Anything behind a login.
How it behaves
- It reads and obeys robots.txt before every crawl, including crawl-delay directives.
- It requests no more than one page per two seconds per host by default and backs off further if a host responds slowly or with errors.
- It identifies itself on every request and never masquerades as a browser.
- It caches pages and re-crawls an agency site no more than once a month unless the site's team page changes.
- It does not execute scripts except where a team page is rendered only by script, and never submits forms.
How to opt out
Add this to your robots.txt:
User-agent: GreatRealEstateAgentsBot
Disallow: /
The bot will stop crawling the site at its next check, within 24 hours, and we will purge any pages fetched from it. You can also opt out, or ask us to remove particular information, by emailing bot@greatrealestateagents.com.au with the domain. We acknowledge within two business days and act within five.
Opting your website out of the crawler does not remove your agents from the ranking, which is built from licensed sales data and public business information; it means we will not collect their details from your site. Agents can always claim and correct their own profile.
Contact
bot@greatrealestateagents.com.au