Blog

Using Satellite Imagery and Computer Vision for Automated Ship Detection

Ship aircraft carrier

Introducing multiclass ship detection algorithm for object detection and classification

Can geospatial analytics shine a light on the global supply chain disruption and answer what’s causing the traffic jam at the world’s biggest ports? How about using location analytics in financial analysis and equity research at investment banks? Monitoring port activity using alternative data would help predict delays and calculate the impact on sales. Geospatial intelligence would certainly empower military analysts, providing an exact count of the number of ships, enabling the analysts to monitor thousands of ports daily using satellite imagery.

Computer vision, particularly multiclass object detection, is the foundational technology that can power these applications spanning from understanding supply chain bottlenecks to performing anomaly detection on military ships. Orbital Insight GO platform provides insight into the events happening around the globe by combining geospatial data from multiple sensors and running analytics on them. The platform enables users to select any region on Earth and provides a comprehensive snapshot of the activities taking place in those regions by leveraging satellite imagery data and geolocation data.

At Orbital Insight's computer vision team, our goal was to build an algorithm to automate the process of detecting maritime vessels at various ports around the globe. This algorithm would be a multi-class object detection algorithm. That means, the algorithm would not only see a vessel(s) in a given satellite image but also categorize the detected object into one of the following types:

  • Aircraft Carrier
  • Cruise Ship
  • Cargo Containers/ Oil Tankers
  • Submarines
  • Military Warships
  • Tugboats

This algorithm would help monitor the supply chain network and provide insights into economic and trade activities across various regions. It would also automate the process of monitoring thousands of sites of strategic interest daily. Given the explosion of data available today, monitoring strategic sites and deriving crucial intelligence from the data has become an arduous task for defense analysts. The algorithm will assist in creating a tip-and-cue workflow to direct analysts to pay attention to sites where their expertise is needed.

Data

The maritime vessel detector was trained on Planet Skysat imagery, which is available globally. We used the red, green, and blue bands from their ortho visual product, which is pan-sharpened and orthorectified. We carefully designed the data collection process to ensure that we collected data uniformly across all regions worldwide and accounted for all seasonal variations, haze, cloudy, choppy waters, and snow regions in our dataset. While at the same time, we had to ensure that we maintained a near-uniform distribution of objects across all categories and all dataset splits.

We generated our training data in-house by sampling scenes over various strategic ports of economic and military importance worldwide. First, a polygon was drawn around each vessel, and each vessel was then categorized into one of the following categories listed above. Some examples of images from our dataset:

An image showing military warships docked at a port in Zhanjiang district, China.

Cargo containers docked at a port in Cotonou, Benin, Africa.

Model

We used TensorFlow to train the model. We experimented with both CenterNet and Mask R-CNN networks and their variants. In the end, we used the Mask-RCNN network to output the masks over the vessel. We post-processed the mask output to obtain a rotated bounding box over the object and the vessel’s category. We also output the orientation of the vessel with respect to the N-S axis of the compass.

Challenges

The project brought a lot of exciting challenges, especially in data collection. First, there exists an inherent bias in the occurrence of some categories of ships in the real world, which leads to skewed data distribution. For instance, the ubiquity of cargo containers versus the rarity of aircraft carriers in the real world is one example of this bias. There are 50 times more container vessels than aircraft carriers in our dataset. This led to an imbalanced dataset where the number of objects from a few categories significantly outweighs the number of categories from other objects. Hence, we had to experiment with novel techniques to overcome this bias while developing the model. The second challenge is the huge variance that exists for some types of ships. One example of this would be a military warship. The algorithm is expected to detect a military warship irrespective of the differences in the color, length, mounted ammunition, and width that exist between warships developed by different countries suited for their needs. Another challenge was the massive variance in the size of the objects of different categories. For instance, we had objects such as tugboats which were about 25 meters in length but also at the same time, we had huge cruise ships and aircraft carriers which were anywhere from 250 meters to more than 400 meters in length. Other factors that made the development of this algorithm even more challenging were variations in atmospheric conditions such as snow, haze, clouds, lighting conditions, view angles of the satellite, and variation in terrains across the regions of the world.

Solution

As part of the development of this model, we explored several novel techniques to deal with the challenges. To deal with the class imbalance in the dataset, we explored data oversampling techniques and experimented with different loss functions in model training. These techniques helped increase the performance of the model on rare categories.

To overcome the challenge of huge intra-class variance, we experimented with several data augmentation techniques which increased the robustness of the model to these variations. With careful design of cropping, padding, and resizing operations in our input data pipeline, we were able to detect objects of various sizes with high accuracy.

Due to the difficulty of this problem, choosing the right set of augmentations, learning rate schedules, anchor sizes, and weight initialization techniques played a significant role in determining the final model performance. Finally, introducing additional post-processing steps to eliminate false positives gave a huge overall improvement in the performance of the model.

Through several label campaigns, many model tuning experiments, and innovative ideas from the computer vision team at Orbital Insight, the multi-class maritime vessel detector was able to reach an F1-score of 0.733 on the test set. Imagine what the Orbital Insight team could detect for you - airplanes, vehicles, tanks, etc.!

Some visualizations of model output on test images:

The image on the left shows our ground truth polygons with military warships docked against each other at the naval base in La Spezia, Italy. The image on the right shows our model’s detections at this naval base. This image shows variation in the size, shape, ammunition, of the docked warships. Sometimes, the model finds it difficult to correctly identify mid to small seized warships which are more of military patrol boats.

The submarines camouflage with the background, making it very hard to detect them. Still, you can see in the above image that our model was able to pick up both the submarines with high confidence. The left image in the ground truth polygon and the right image shows our model’s predictions. The image is of Waalhaven harbor in Rotterdam, Netherlands.

Cruise ships docked at a port in Marseille, France. The model is able to detect and correctly identify all the cruise ships.

Container vessels carrying goods through the Huangpu River, Shanghai, China. There is even one military vessel docked at this port. Left is the ground truth, and the right image is the model predictions. The model can detect container vessels with high confidence.

The model is able to detect vessels across a wide range of sizes. In this image, one can see the model in action detecting those tiny tug boats docked at Port Hedland, Australia.

Curious to learn more, check out our airbase monitoring application using Orbital Insight GO: