Geographic Open Data — Where to start?
So another #GISDay is upon us today (16th November) and the volumes of geo data available is growing and growing. Geographic Information Systems (GIS) are a critical tool for geospatial analysis and insights. I’ve made extensive use of geo data for analysis in the past and GIS is one of the most established areas of data analytics in many fields.
In the UK, Studies have illustrated that a cost-benefit ratio of a £4 return for every £1 spent[1] on the use of geospatial data ([1] ConsultingWhere (2016) ‘Cost Benefit Analysis of Address and Street Data for Local Authorities and Emergency Services in England and Wales — Final Report.’ St Albans: ConsultingWhere).
So, where to start? If you do not know much about geographic data or GIS it can be quite daunting with new types of data, tools and data sources. Some of the information below can help, but this is just a very small snapshot and there is so much more available. Beware though, the more you delve in, the possibilities become more and more innovative and you may just catch the geo bug.
Formats
There are lots of different file types that you may not have heard of before and there are too many to mention all of them here. However, the main ones I have used in the past that are frequently available are:
Gml
The Geography Markup Language (GML) is the XML grammar defined by the Open Geospatial Consortium (OGC) to express geographical features. GML serves as a modeling language for geographic systems as well as an open interchange format for geographic transactions on the Internet.
Source: https://en.wikipedia.org/wiki/Geography_Markup_Language
Geojson
GeoJSON is a format for encoding a variety of geographic data structures. Source: http://geojson.org/
You will see geojson being used quite commonly for web mapping using tools such as Leaflet which is a JavaScript library for interactive maps online.
Shp
The shapefile format is a popular geospatial vector data format for geographic information system (GIS) software. It is developed and regulated by Esri as a (mostly) open specification for data interoperability among Esri and other GIS software products.[1] The shapefile format can spatially describe vector features: points, lines, and polygons. Source: https://en.wikipedia.org/wiki/Shapefile. Some may argue that it is mainly used for ESRI products but I have included it here as I have downloaded a lot of shp files data over the years as open available data that can be easily transformed and it is being used more and more on geo open data portals as an available format.
Csv
Comma Separated Values (CSV) data files allow data to be saved in a structured way and are compatible with day-to-day desktop analysis tools such as spreadsheets. A geo csv file is usually a general csv file with geo coordinates included that can be matched to a map or a place. For instance, if I had a list of premises I may want to add the geographic coordinates to the csv file in the form of longitude and latitude coordinates. So when the data is mapped, the coordinates allow the map to match the place name by using the coordinates to geocode the point to the address.
If you’re interested in learning more on these data formats, there is a much larger list with explanations available here.
Sources
OS Opendata
This is a really valuable resource of UK open spatial datasets from Ordnance Survey. You can order OS Opendata products from their download page at: https://www.ordnancesurvey.co.uk/opendatadownload/products.html and there are loads of products available. From background maps, roads, postcode units, boundary lines, rivers, vector maps and to even terrain layers. Data is available for download in variable formats but look like shp, csv or GML files are the main formats.
Getting started with OS OpenData:
ESRI open data (ArcGIS Open Data) - I haven’t used this one myself but it gets mentioned quite a lot when discussing geographic data. I ran a a quick search for UK data http://opendata.arcgis.com/datasets?q=uk and it returned 3,274 results so there is quite a lot to exploit.
Data.gov.uk |Mapping- The UK’s national open data portal has masses of data, ~40,000 datasets last time I looked, and a large number of these are geography and mapping based datasets from national agencies, local authorities and other organisations. It’s worth a good dig through as you may be surprised what’s available. Similarly, check out it’s European counterpart in the European Data Portal as it says it has 15,782 datasets found for “mapping” in the UK!
Global Forest watch- Bit specific this one, forest and natural resource data, but a great example of a platform with a variety of data available usually in multiple formats. The dataset I looked at allowed downloads of data in spreadsheet, kml or shapefile.
Looking for more data? The Geovation website has a much more comprehensive list of data sources and worth a look at: https://geovation.uk/data-sources/
Tools
There are loads available these days , varying from desktop geospatial analysis to interactive web mapping tools. Some of the more popular players here are:
CARTO is an open, powerful, and intuitive platform for discovering and predicting the key insights underlying the massive location data in our world.
This platform allows quick analysis for the nonexpert and can provide some powerful geospatial and visualisation tools for the non-expert. It is available as a Free or Personal account, or more advanced paid licensing models for organisations.
D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG, and CSS. D3’s emphasis on web standards gives you the full capabilities of modern browsers without tying yourself to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation.
If you’re not used to coding or data design, D3.JS has a ton of documentation that will guide you through creating data visualisations, including maps. Some great examples here with sample code for creating maps, including:
- Click to zoom maps
- Chloropleth map of England and Wales House prices
- Animated Historic Weather-station data for the UK (1880–2013)
There are lots of spatial and non-spatial examples of the visualisations available with loads of sample code that can be used and trialled at:
OpenStreetMap is a map of the world, created by people like you and free to use under an open licence.
Another great online mapping tool that is free for anybody to create maps and data and update existing sources, download data and reuse. Has a whole range of online mapping tools and functions and there is a very strong community focus with this site and there are helpful tutorials, wikis and videos available. Some really useful sharing and map image creation functions as well. OpenStreetMap data is licenced under the Open Data Commons Open Database Licence (ODbL).
A Free and Open Source Geographic Information System. Create, edit, visualise, analyse and publish geospatial information on Windows, Mac, Linux, BSD
A great alternative GIS system especially for desktop users. As it is open source, it can be downloaded and installed on variety of operating systems. I highly recommend trying this out if you are interested in geographic data. It brings a whole new level of tools and functions in geospatial analysis and has tools and tutorials for every level from basic ‘I want to create a map’ to more advanced functions such as working with spatial databases and big data analysis.
The obvious one that most people have heard of but you can also use it to make maps yourself. If you have a google account you can start playing and adding layers and information in. I have used this in the past for quick fixes such as to make a quick display map or plot points of interest. The interface is intuitive and most users with an account should pick up the map elements of this very quickly.
Update 22/11/2016
It has been rightly pointed out to me that some of these tools are open source and as such need support from the community who use them. If you’re interested in supporting them or learning more about the organisations or projects, check these out:
- Donate to QGIS
- Donate to OpenStreetMap
So I like geo data, what’s next?
If you have looked at all this above and thought I would like to know more, check out groups like FOSS4g UK who bring together open source GIS users, developers and supporters from the UK and the Association for Geographic Information (AGI) who actively support the development and innovation of Geographic information and tools. In addition, Open Data Camp is an unconference entirely devoted to open data and will also have a major geographic element with lots of like minded folks. Well worth investigating and engaging with.
There are lots of data formats, tools and sources not mentioned here, so please do add any additionals in the contents below.
