From Doodle to Digital: Extracting Features from Hand Drawn Maps
In our society, technology is trapped in a never-ending state of flux, a forward acceleration thats driven by innovation, widespread access and advancements in hardware. As a consequence businesses are active participants in an ongoing cat and mouse chase attempting to keep up, adapting their business processes to take advantage of new technologies. This transition (from old to new) has a series of challenges, particularly around data that is attached to legacy systems, this is valuable information that needs to be integrated into modern day (digital) processes.
Kainos have vast experience with these challenges, especially in regard to digital transformation for large organisations (including the UK government!). On occasion, customers will approach Kainos with a problem that lives outside the reach of our traditional skill set, problems that often require cutting edge solutions. This is where the innovation capability comes into play, this is an area where the team adds signifanct value to Kainos. Lets dive into an example use case, which involved the innovation team leveraging computer vision to extract features from hand drawn maps for a customer (hereby denoted as ‘CX’).
It All Started with a Pencil and a Few Crayons
Lets jump back a few decades, back in the old days when all the information associated with an organisation was collected entirely on paper — written manually by humans and stored in multiple, large filing cabinets. Eventually as time went on the digital revolution began to make a big splash, this led to businesses start the slow (often still ongoing!) journey of digitising their own written information for storage on servers.
CX also found themselves smack bang in the middle of this transition, in the past they captured property information (land ownership etc.) by dispatching a case worker to visit a location (equipped with a pencil and crayons) to draw a birds eye view of the land (while also writing up supporting documentation). The output looked something like this…
These are whats known as title plans, a scanned drawing which represents a piece of land visually. The anatomy of a title plan is typically made up from colour shading, pencil outlines, north facing arrow and other textual identifiers. Accompanying the visual features associated with title plans are a set of interesting challenges, which wrap a series of constraints around a potential solution.
- Absence of a consistent key between colour to definition, the description of what each shading represents is contained in another distinct document. This means, a case worker manually has to read an entire associated text to determine what each of the coloured areas represent —just imagine how tedious this is! 😔
- The modern digital mapping system contains additional information that is complementary to the hand drawn maps — which means the hand drawn features need to be overlaid in a consistent manner 😬
- Lets not get too down in the dumps…at least the red border is consistent on every title plan 😏
Okay, so there a few interesting challenges, however we felt that these were barriers we could overcome! So, to summarise, the current business process for a case worker attempting to find all the information for a piece of land looks a little something like this:
This is not particularly efficient, however this understanding helps define a few core goals that we felt would prove most valuable for CX, including:
- Extract the coloured ‘features’ from the scanned title plans and overlay onto the modern digital maps used by CX.
- Enable easy searching of the associated textual document for the mapping colour key for each title plan.
Cool, so how we do we actually achieve these goals?
Lets Bring Computer Vision to the Party
Just incase you have never encountered this concept before, lets start with a handy dandy definition:
Computer vision is concerned with the automatic extraction, analysis and understanding of useful information from a single image or a sequence of images
Lets step back, basically what this entails is the extraction of useful information from images — in our case this includes the shaded areas of the scanned map. Within the innovation capablity in Kainos we have extensive experience in computer vision related tasks, using this experience we put together a plan to help achieve our goals, so what did our solution look like?
Fun Fact: CX posed this challenge to other competitors to see who could solve it most effectively…As you would expect this got our competitive juices flowing.
Putting Our Money Where Our Mouth is
We chose to put our faith into the grand daddy of computer vision tooling — OpenCV. OpenCV remains one of the best choices when taking on a vision task due to its robustness, integration with multiple languages (we chose Python), community support and the sheer volume of options avaliable which are hidden behind easy to use abstraction layers.
- Image Extraction
The original data was a scanned PDF containing the hand drawn map & additional supplementary machine text containing details like the title plan ID among other things. To extract the element we wanted (the map) we wrote a script to parse the PDF and output the images contained within it — phew, step one, complete.
2. Feature Extraction (Aka where the magic happens)
At this step we had in our possession a dataset containing hand drawn maps, now we had to extract the areas of interest — This is where OpenCV comes into play. We will use the image below as an example to demonstrate how we applied image processing to manipulate the images.
Step one, Binarization: We first had to convert the base image into what is known as a binary representation — essentially transforming the image to only show black and white pixels, where white are the ‘features’ of interest and black is everything else.
However, we encountered an unknown challenge, not all of the hand drawn maps in our dataset had a pure white background — given that these images were essentially scanned from paper we discovered that the paper could be discoloured (taking on a muted yellow appearance). This ruled out the possibility of a brute force strategy which simply removed the black pixels (the pencil lines) and the white pixels (the backgound) — we needed something a little bit more intelligent.
To overcome this challenge we converted our images to the HSV colour space (rather than the traditional RGB colour space) which enabled us to target the saturation of the image much more effectively. This enabled us to use the mean saturation of the image as a means of creating a mask which could filter out the pixels that are not within the colour ranges where the likes of green, red, bright yellow and blue reside —aka the features we want to extract.
We then applied this mask to our original image, which yeilded this as the result…
Step two, colour retrieval: We could perform a ‘bitwise and’ operation using the mask & the original image to extract a perfect cut out of the areas of interest, only this time with the colour information intact.
Success! I think that is pretty good feature extraction 😄
3. Digital Map Overlay & Association with the Colour Key
It turns out that overlaying the the hand drawn features on top of the digital mapping system was relatively simple (due to the output image having an alpha channel). Additionally the associated document also contained the coordinate information for the land, which helped us define the location where the overlay would be placed. Great, so this enabled us to achieve the first of our two core goals — blending two distinct sources of information into one central, up to date location.
However, there is still one big elephant in the room, how could we associate the extracted features with a description stored in a separate (digital) document? Luckily for us the colours leveraged by case workers are quite distinct from one another with very little potential for overlap, with a focus on primary colours, these have each quite different representations in the pixel domain (RGB, where a colour is made up of three brightness values representing the three colour bands).
Red Range = (255, 180, 180) → (180, 0, 0)
Green Range = (150, 230, 150) → (25, 100, 25)
Blue Range = (180, 210, 255) → (0, 70, 180)
Yellow Range= (255, 255, 150) → (230, 230, 0)
Using these pixel boundaries we could identify which colours are present in the extracted features. This could then be used as a means of searching the associated document for a key word match, which would enable highlighting the description defining what the colour represents. In one swoop we have removed the need for the case worker to read the entire document for the colour descriptions, enabling them to spend their time on more valuable tasks (while also minimising good old human error).
So I know what your thinking, what would this look like for a case worker? To showcase this we put together a demo portal which enable a case worker to search for land based on a code, view both the hand drawn and digital features associated with the land as well as extracting the colour key information, check out the screenshot below.
Fun fact #2: It took the core team (three people) only two weeks to create this tool
Future Work
This particular problem space has a host of additional interesting opportunities that would require computer vision techniques & our innovation team at Kainos sure does love interesting / challenging problems!
You may have noticed that some of the title plans contain a ‘north’ directional pointer, if we are able to extract this point and adjust for the ‘on paper offset’ it would enable a more accurate overlay of the hand drawn features onto the digital map. So how would we do this? well…
- Train a convolutional neural network to detect the north arrow in the title plan — extracting bounding box coordinate information.
Use the orientiation of the north arrow to automate the rotation of the extracted hand drawn features to enable consistency (removing the need for manual editing by a case worker) with the digital map.
Conclusion
Moving data to new systems is hard. Its an ongoing battle that organisations are currently waged in with no end in sight, often this phenemonem breeds some interesting problems, just like this one we faced with CX. Often the solutions are hidden behind interesting technologies like computer vision, where both specific expertise and demonstrable business agility (away from our core Kainos offerings) are given the stage to shine.
If you feel that anything in this blog is applicable to your own business (even outside of the scope of extracting features from maps), or if you are interested in contacting the innovation team to discover more about what we do, feel free to send us an email: appliedinnovation@kainos.com
