AI + Humans: How Artificial Intelligence scales Human Expertise for Detecting Changes to the Earth

This post details how Orbital Insight’s partnerships, artificial intelligence expertise, software and systems engineering allows us to reliably detect semantic changes to the Earth’s surface. We will go through a generalized overview, as well as a specific case study to demonstrate how we can enable both AI and human analysts to each do what they are best at, and iterate towards a globally scalable solution.

Overview

Changes in land use and land cover are crucial for understanding socioeconomic behavior, as well as supporting various commercial use cases that require the latest infrastructure and connectivity data. Commercial mapping use cases, in particular, require an extremely high degree of fidelity and confidence, which includes the ability to track changes to key area features at massive scale. Features such as buildings and roads often change well before official mapping updates, which themselves can take substantial amounts of time and manual effort. Critical feature records may therefore be out of date, and if put into use lead to faulty analyses, planning, or operations.

Satellite hardware companies that have successfully launched fully operational constellations are extremely rare and provide a differentiated “high ground” solution to address these issues. Earth observation imagery from these will offer a good first source to search for new features and updates. This is dependent, however, on sufficient coverage over wide areas, as well as resources to search for and catalog relevant updates.

Orbital Insight is able to reliably monitor wide areas by stitching together imagery from multiple satellite providers and extracting features at scale with proprietary AI.

We have seen that official mapping records can be very out of date, especially in remote, rural areas. For example, Figure 1 shows a region where some mapping records maintain less than 20% of the actual road network, as detected by our fully automated analysis of recent satellite imagery. The same value proposition can extend to fast-developing urban areas, where manual updates may not keep pace with the rate of growth. Either way, this type of effort shows where existing records are out of date, and even provides a foundational dataset with which to update them.

Figure 1: An area in rural Cambodia where we see the value of pairing current wide-area imagery with scalable artificial intelligence. Left-to-Right: OSM road networks (cyan, displayed over older stock imagery); Google Street Maps; and our newly derived road networks (red, derived directly from current imagery, displayed over older stock imagery).

However, the accuracy one can derive from satellites alone may not support all commercial use case requirements. In which case, satellite-based analyses can tip-and-cue other satellite sources, non-imagery geospatial signals, and/or manual follow-on efforts. Given the financial and time commitments required for some of these use cases, commercial mapping professionals need to deploy resources to those areas where building footprints or road networks have indeed changed.

We will go through a case study demonstrating how Orbital Insight’s access to multiple earth observation satellite constellations, high-performing artificial intelligence (AI), and software/systems engineering can help solve these challenges.

This pairing offers an unprecedented foundational lift that empowers the human domain expert to accomplish their mapping updates faster, cheaper, and better than they would sans this technology.

Lessons from chess

“The key is in finding the right balance, and, as unlikely as it sounds, there are lessons from chess. After Garry Kasparov’s infamous defeat against IBM’s Deep Blue, he went on to consider the possibilities offered by playing chess in partnership with computers rather than against them.

Rather than admitting defeat, he invented a new form of the game, called Advanced Chess, where a human and AI work together. The brute force analysis of the computer system together with the more strategic thinking of the human player has taken the game to heights of skill never seen before, and it’s now an active sport around the world.”

The above anecdote is a transferable analogy about pairing AI with human professionals. Just as AI is able to detect minutiae and tactical level insights to empower human experts in chess or in medicine, our AI can scan vast quantities of satellite imagery for pertinent features that an analyst can use to enhance their existing processes and workflows.

Commercial Mapping: Challenges

The requirements for advanced commercial mapping are increasingly rigorous, and there are limited options to meet them. More specifically:

• Commercial road mapping use cases, like those supporting autonomous car technologies, demand highly exquisite and accurate maps.
• It is extremely costly for people to drive every mile of road to search for changes; end-state systems may require even more precise measurements to actually update their records, beyond just detecting that change.
• A single satellite constellation’s cadence and fidelity cannot keep pace with road changes nor provide the necessary accuracy in some of these cases. This is particularly true for rapid, hyper-local development in urban areas, and also for rural/remote regions that are imaged even less frequently, as shown in Figure 2.

Figure 2: Outlines of nearly 1,400 high-resolution (<1m GSD) images taken over Japan’s mainland, across the entirety of Q4 2018 (October through December). Rendered over a Bing imagery hybrid visualization, and filtered to include images with up to 40% cloud cover.

The need for both feature-level accuracy and nation-wide scalability makes Orbital Insight’s GO platform the ideal solution. By ingesting, interpreting, and analyzing exquisite high-resolution imagery combined with differentiated high-cadence imagery, commercial mapping companies are able to meet these challenges.

Commercial Mapping: Solution

Orbital Insight provides a systematic solution based on the partnerships and technologies we have cultivated, all accessed through our online GO Platform. We combine our self-developed AI algorithms, with integrated and scalable software, and leverage the multiple imagery constellations at our disposal.

The first, broad sweeping search for changes leverages exclusively AI and medium-resolution imagery. Medium resolution imagery has a significantly higher revisit rate, as shown in Figure 3, covering entire countries at the speed of change and in such high volume that human analysts are hard-pressed to search it rigorously and completely.

Figure 3: Outlines of every medium-resolution (3–5m GSD) image taken over Japan’s mainland, within the same time period as Figure 1 (Q4 2018). There are over 124,000 total (with the same 40% threshold for provider cloud score).

It is financially unfeasible for a human workforce to analyze this volume of imagery, however, AI can augment domain expertise. Figure 4 underscores the totality of this challenge, with a time series showing often several hundred thousand square kilometers of imagery collected daily. By extrapolating from previous hand-marking projects, we estimate it would take a team of 10 analysts working full-time approximately 388 business days to review the 124,000 tiles referenced in Figure 3’s caption above.

Figure 4: A time series of daily detections across the entirety of the Japanese mainland, showing the scale of combined coverage (in square kilometers) covered between a medium resolution imagery provider and multiple high-resolution imagery providers; both are again filtered to include up to 40% cloud covered imagery.

Our highly-trained computer vision algorithms allow us to interpret this wide-area imagery rapidly and at scale, detecting the same essential, optically recognizable features we would get from searching it by hand. We can also compress and compare different collection time windows and search for changes between them.

• Our AI can process this scale of information and deliver structured results in a matter of days. It could take over a year for a team of human analysts to search this same volume of imagery.

As noted earlier, however, the accuracy and fidelity of imagery are not always sufficient for updating datasets with more rigorous requirements. Our proprietary software lets human analysts easily iterate over and validate those results, leveraging multiple medium and high-resolution sets of imagery as well as other geospatial sources; Figure 5 below shows a sampling of what these different imagery sources look like.

Figure 5: A sampling of different types of imagery available to Orbital Insight for computer vision algorithms. We can optimize between the relative availability and fidelity among them for both initial detection as well as tipping-and-cueing and validation.

Using these in various orders and combinations, we can derive a higher confidence dataset with relevant semantic changes. This can then be used to target changes/updates at the required level of rigor (hand annotation, custom collection, ground truth/field work, etc.). We capitalize on the relative advantages of these multiple sources and have engineered systems and software for human analysts take over when and how it is most advantageous to do so. The entirety of this process is outlined in Figure 6, including a generalized concept of operations and more specific anecdotes.

Figure 6: Three Tiers of Effort, powered by Orbital Insight’s technologies and partnerships, enable substantial cost and performance improvements for updating commercial quality data. A generalized concept of operations is outlined on the left, while corresponding vignettes are on the right.

This process offers an iterative system of human-machine teaming powered by a virtual constellation of multiple satellite image providers, high-performance AI, and integrated validation software/processes. This will be demonstrated through a practical Case Study on how Orbital Insight leverages those elements to help our partners get highly accurate updates to countrywide road networks in a timely, cost-effective manner.

Case Study: Finding New Roads across Japan

Tier 1: Regions of Interest and Automated Land Use Analytics

The first step is to define the region we would like to search, or persistently monitor for land use updates. We can draw boundaries as large as entire cities, counties, or countries, and start an automated process that includes: ingestion from one or more satellite image providers, proprietary pre-processing and computer vision application, and advanced data science to determine changes from one time period to another. Figures 7 and 8 demonstrate the results of Orbital Insight’s land use classification algorithm, during a specific time interval; respectively, showing all land use classes across an entire city, and one specific sub-class (roads-linestrings) across a mixed urban/unpopulated area.

Figure 7: Single time period results from Orbital Insight’s land use algorithm, applied across an entire city scale, and based on medium-resolution imagery (Planet, 3–5m GSD).

Figure 8: Single time period results from Orbital Insight’s land use algorithm, showing specifically a roads-as-linestrings output class across a mixed urban/unpopulated area and challenging arid/desert conditions. These results are also based on medium-resolution imagery (Planet, 3–5m GSD), visualized over generic/older basemap imagery.

Through GO, and based on user-specified custom options, results like these can be automatically compared to other time periods to determine changes across semantic land use classes. This typically results in more changes identified, and far more quickly, compared to human review of such sources.

Tier 2: Efficient Validation of Changes Detected

The next step is to quickly validate changes that were detected from one time period to another, in this case: those involving road features specifically. Our proprietary data annotation software supports this in a rapid, easy-to-use and team-scalable interface. Figure 9 below shows how we provide a system for quickly visiting each detected change, and validating it using various imagery catalogs. The success of this model is predicated on our partnerships — the ability to access multiple image providers' products — and engineering aptitude to cross-analyze these with Tier 1 results. Multiple users can conduct this process in-tandem with one another which also helps improve the pace and scalability of such efforts.

OI’s internal software, as shown in Figures 9 and 10:

• Seamless management of multiple users and thousands of images
• Interactive GUI
• Users can toggle quickly between multiple data types
• Rapid and straightforward QA for land use management

Figure 9: Internally-developed software for validating detected road changes in Airbus SPOT imagery, providing multiple resources for validation, and capturing updated results in a structured manner. We help users compare before vs after images around one of our initial detections, letting them quickly validate AI detections using multiple imagery sources.

Figure 10: Results of software validation, again leveraging Airbus SPOT imagery to compare before vs after images around one of our initial detections; this lets the analyst quickly validate AI detections using multiple imagery sources.

Through this process, we have collected a large number of possible changes from our initial AI capability (Tier 1), and then successfully narrowed these down to the most high-confidence/relevant instances (Tier 2). To recap:

• We leveraged partnerships with the best satellite partners in the industry and AI technologies to detect any roads development, at any scale, and at any cadence. There was no human involvement in this beyond setting initial parameters for geographic boundaries and cadence of analysis.
• We validated this AI-derived list of possible changes, using our custom software engineering and multiple partnerships to put human analysts in the most advantageous position possible. They can view multiple imagery sources and before/after images, incorporate additional spatial datasets, and involve multiple teammates simultaneously, to quickly and flexibly narrow this list down to only high confidence instances.

We are now ready to share this refined list of highly likely road change detections with our partners.

Tier 3: Rigorous Road Data Curation and Updates

We will share this AI-derived, efficiently human-curated list of road change detections with our partners, who defined the need parameters in the first place. Depending on the level of rigor their requirements dictate, they may still have to deploy field resources and conduct the same manual curation they have always done. However they now have a significant advantage, knowing exactly where to conduct such efforts, and likely having a more replete set of updates needed overall. By comparison, they would otherwise have to rely on government announcements, manual and infrequent imagery searches, or physically driving around to look for changes.

Our partners can now bypass many of those efforts and skip straight to deploying their collection assets, targeting only those locations that have high confidence or recently occurred, relevant changes. The time and cost savings for this can be substantial, as can the improvement in result quality.

How much time and cost can GO save?

Orbital Insight’s solution costs ~15% of the traditional method. Time and cost savings can vary from one use case to the next, but a good example is shown in Figure 11 below. This is the estimated time and cost that would be taken by using two different time periods to search for change, in this case detecting roads changed all over Japan (area: 378,000 km²).

Figure 11: Example of cost comparisons for commercial-grade road detection of all over Japan.

Conclusion

Changes in land use or land cover are crucial elements for understanding socioeconomic behavior, as well as ensuring accuracy in managing nationwide infrastructures and the adoption of new technologies. Raw satellite imagery is often used as the first source of information for such updates. However, this relies on adequate wide-area coverage, and significant time and financial costs. Furthermore, a single satellite provider will not be able to simultaneously deliver the cadence and resolution needed to support most commercial use cases; nor can the level of accuracy derived from satellites support all commercial use case requirements.

Orbital Insight provides systematic AI solutions based on the valued strategic supply-chain and multi-sensor data partnerships we have cultivated over the past five years. We combine our self-developed AI algorithms and efficient validation process of any change detected. These are all accessible through our online GO platform. In this way, GO is able to accomplish substantial savings in time and cost and effort, with improved data accuracy and currency.