Deep Learning – Instance Segmentation

In this blog, the focus will be on using the instance segmentation in the RedAI application.

Instance segmentation is a computer vision task for detecting and localizing an object in an image. Instance segmentation is a natural sequence of semantic segmentation, and it is also one of the biggest challenges compared to other segmentation techniques. The goal of instance segmentation is to get a view of objects of the same class divided into different instances. Automating this process is not easy, because the number of instances is not known in advance, and the evaluation of the obtained instances is not based on pixels as it was the case with semantic segmentation. Segmentation of image instances is not an explored area, but interest is motivated by the possibility of application in practice. Instance tagging provides us with additional information for inferring unknown situations, for counting elements belonging to the same class, and for detecting certain objects to be retrieved in robotic tasks.

It can be used to a large extent in autonomous driving, in medicine, surveillance, etc.

State-of-the-art instance segmentation can be found at Arxiv.org.

You can see the concept of the instance segmentation in the following figure:

So, unlike in semantic segmentation, all three dogs have different colors, or labels. Semantic segmentation does not recognize more of the same objects in one image as different, while instance segmentation does. For example, if there are multiple dogs in one image, the instance segmentation will recognize it.

Mask R-CNN

We will use the Mask R-CNN model to solve the instance segmentation problem. That’s a deep neural network based on a faster R-CNN architecture that, in addition to classification and detection, also provides mask prediction for each bounding box.

RedAi App

The RedAI app is an artificial intelligence software solution that enables FMCG/CPG companies in Retail and HoReCa to detect, classify and control their own and their competition’s SKUs (products) with a series of photos taken from a mobile device or tablet.

The RedAI application is based on computer vision tasks:

• object detection
• object classification
• semantic segmentation

Our RedAI team is in the phase of researching and implementing instance segmentation for two classes (bottle and price) in the RedAI application.

Masks would allow us finesse in the form of better augmentation (we could accurately crop the products and glue them on empty shelves). There is also a realistic chance that the use of masks would increase the accuracy of product classification (because they could put detection parts that are not inside the product mask to zero, so the classifier would not look at them)

The process consists of processing the data, then learning the model and testing the images outside the training distribution.

Masks would allow us finesse in the form of better augmentation (we could accurately crop the products and glue them on empty shelves). There is also a realistic chance that the use of masks would increase the accuracy of product classification (because they could put detection parts that are not inside the product mask to zero, so the classifier would not look at them)

The learning curves of our model look like this:

The figure below shows the final result of our model:

Conclusion

In this post, we discussed the concepts of instance segmentation based on deep learning.

We went through the model architecture and explained the concept of instance segmentation on the RedAI app. This should provide a basic understanding of instance segmentation as a topic in general.