Property Price Register — Mapped

As I previously wrote in my Property Price Register — A Rant, the Property Price Register (PPR) in Ireland is a mess. A while back as a side project I decided to take on a task of mapping it.

The steps were as follows:

• Get the public PPR csv, all records since it started. Parse it, clean up common issues I found, run reverse geocoding on each address and then store the results with the correctly formatted address and latitude and longitude.
• This then allows me to display properties on a map, filter and run queries on.

The website as it stands is here :

The Property Price Register Mapped.

I haven’t properly set up the domain, securing it etc. for reasons I’ll go into later.

For those interested I’ll go into technical detail about how I created it.

Backend:

• NodeJS with Express and Objection ORM
• MySQL

Front End

• React Bootstrap

Processing

• NodeJS
• MySQL

Processing

The Processing component reads the publicly available property price register public CSV file. A NodeJS script parses the CSV, reading each record line by line. There’s several thousand records so there are optimisations in place to ensure subsequent runs on the script don’t do the same work again.

For each record the app performs formatting and cleanup of the addresses, this removes some common issues and issues with Irish address as I wrote about previously.

The resultant address is then looked up in the database to see if we already parsed it before, if so then we can update the details with a new sale.

If not present the address is stored in the database and flagged as requiring a geocode.

On the server there’s a cron job running every few hours that looks at the database for latest records and performs a reverse geocode on the new records. When results are fetched the database is updated, if an accurate match is found the returned formatted address, latitude and longitude is stored. If an accurate match is not found, e.g. only a general area or bounding box is found, the address is flagged as rejected and put into a table into the database.

Geocoding

For geocoding I used a few different sources. This is ultimately where the biggest issue I found lies.

Google Maps was the most logical and first port of call. There are over 400 hundred thousand entries in the property price register csv, using Google Maps very quickly becomes very very expensive. As a side project I wasn’t willing to spend much on this so this was quickly ruled out.

Open Street Map was the next option. It is free and I thought it would be accurate. However the rejections I was seeing weren’t good, too many valid addresses were being rejected. So I decided to move onto something else.

Finally I landed on Bing Maps. It has a free tier which allows for a lot of free geocoding requests. They need to be batched up and processing is delayed which made things a little trickier. I created several accounts and API keys and fired off the cron job to cycle through the keys a few times a day. It almost certainly wasn’t allowed under their T&C but I just wanted to see how far I could go with it. The results are ok, Google Maps is more accurate but I couldn’t complain.

API

I created a REST API using Express with Node, it’s pretty straight forward — just pulls the data from the database really. I also added JWT to it, in the vain hope of others using the REST API in future.

Front end

The front is using React Bootstrap. It was my first time using React so a fun learning experience, though frustrating to do simple stuff sometimes. I added some fairly basic filters, they probably aren’t super robust but seem to do the job most of the time.

I’m using Google Maps for visualising the properties, their clustering is impressive. If a user does try to load all the properties at once, then it does spin the fans on your machine and is slow but for a year or so it works ok on my macbook.

The Future

I managed to get through the 300k records (at the time the full PPR) around2 years ago. However life got in the way and I had to side-line the project for some time. When I got back to it there was another 100k odd records and all my accounts had been disabled. Microsoft changed their policies and it makes it impossible for me to (ab)use the service again in the same way.

I revived it in recent months and improved it a bit but a few things happened that means I’ll probably just set this to the side and leave it as is.

• I discovered the site https://propertypriceregisterireland.com/ — they have done pretty much everything I’ve done. Clearly they’ve had more time and ability on front end dev to push it along than me.
• The problem with the reverse geocoding is not a cheap one to solve. Moving providers means I may have issues with existing records and matching in the future.
• The geocoding I’ve used isn’t very accurate anyway, there’s tons of problems. Possibly caused by my errors when parsing and processing the content and receiving geocoding results.

I’ll probably make the repo public after I clean it up a bit. For now, enjoy the site as it is, it’s got several years of data there. Much of it is accurate, some issues are there but hopefully it’ll be useful to someone!

Any feedback is welcome at eddie dot long dot dev @gmail.com