As part of our Geostore project, the World Bank wanted to know the road surface type and quality of the farm to market roads in the Philippines.
We had an image repository of many of the roads. Lots of glorious road images! To be precise, we had over 200,000 images of the roads taken during various phases of the construction projects, ie. survey, planning, in-progress, and completed.
Luckily, most of these images had their latitude and longitude coordinates embedded on the photograph as well. We could use this information to pinpoint the location of the image and plot it out on a map, and use it to determine the location of the road.
One challenge was that each road could have multiple photos taken at the same spot, but at different dates and times that corresponded to different phases of the road construction. For example, a photo could be taken when the road was still being surveyed and thus was still a dirt road, and then another photo was taken at a later time when the road was already concreted and completed.
Luckily again, the timestamp of when the photo was taken was also embedded into each image. This allowed us to pick the most recent photo when there were multiple photos of the same spot with different surface types. We could then tell what the latest state of the road was. Problem solved!
Another challenge was the labeling of the images and assigning it the correct surface type. How would we know when a dirt road was not a gravel road? What if it’s partially concrete? What if it was concrete with some asphalt on it? What if the person categorizing it made a mistake?
We had to clarify the surface types and we had to create a quality control process to ensure accuracy. It was then that the Geoprocessing Dashboard was born!
We created training documentation that gave several examples on each surface type so that all the people categorizing would be consistent in their labeling. We also created a protocol on what to do if the photo contained multiple surface types. For example, a photo of a single lane concrete road with the other lane being a dirt road would be labeled as both a dirt road and a concrete road. Yes, there were a LOT of roads like that.
People make mistakes. So what we did for quality assurance was we created a feature for multiple people to label the same images so that there was a check on inconsistent labels. These labels could then be flagged and other people could take a look at it and give the appropriate label when there were conflicts. This enabled efficient resolution on controversial images.
The labelers were staring at their screens the whole day. We realized that labeling the images using their mouse was taking a lot of time. There had to be a way to make this process more efficient.
What if we can label the images and proceed to the next one without having to lift our fingers from the keyboard? To quickly experiment if this was useful this, we created a special tool complete with ergonomic and intuitive shortcut key controls, and BOOM, productivity doubled!
But we were barely making progress towards completing all 200,000 images. Each image took about 2 minutes collectively to label as it was being labeled by multiple people. At that rate, even with 10 people working non-stop for 8 hours a day, it would take almost 4 months to label it all. We needed to find a faster way.
Artificial Intelligence! Our team was experimenting with Artificial Intelligence and Machine Learning technology that enabled automatic image classification and labeling. Could we use this to label photos of roads?
First, we needed to provide the Machine Learning (ML) model with a large number of examples of images for each category. Fortunately, we had that! We already had labeled about 10,000 images, and so we already had the base of what was needed to implement a custom ML model.
Screenshot from: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets
After several hours of getting it set up and running the training for the ML model, it started to work! We piped in the remaining 190,000 images into it, and in less than 24 hours, we had labeled all 200,000 images. Oh yeah!
But, we had to double check several images that the ML model reported that it wasn’t sure what the appropriate label was. It turns out, most of these images were selfies and group photos of the inspection teams. :)
In summary, we started with creating a labeling tool, made it more efficient, discovered that it wasn’t fast enough, and decided that it was a job for Artificial Intelligence.
And that, friends, is how we categorized over 200,000 road images with Artificial Intelligence.
Learn more about our work with the Geostore here.
Need some processes automated for your company's operations? Talk to us and we will see how we can help.
