#whiteandgold or #blackandblue? You’d be forgiven for choosing to live under a rock for the past two years, but in case you were doing so in 2015 and have no idea what those hashtags are referring to, look up: the dress.
In order to avoid reigniting colour perception havoc across the internet, for our latest hack day two teams decided to undertake the task of extracting product colours from images. Our content team carefully curates our products, but us engineers were feeling green with envy and wanted to see how our algorithms would fare.
We started the day with a high-level discussion about image processing and algorithms which could be useful for the task. Upon going through the product image dataset, it became clear that simply finding the most dominant colour in an image would be an insufficient approach. The first reason was that the background would manipulate the result. While this could be solved with some processing, the second reason precluded us from doing so: some product images contained an entire scene as opposed to featuring the product in isolation.
Example of a ‘busy’ product image with multiple objects.
With this in mind, we decided that it was necessary to first extract the object out of the original image and then apply a colour detection algorithm.
Both teams elected to use deep learning but differed on the models used. One team elected to implement salient object detection based on a VGG16 model. This model detects the region of an image which is most salient, ie. the part of an image which would immediately attract your attention. It was chosen due to immediately ignoring objects which were not important. By contrast, the other team used an object detection model to find all objects in the image. With all objects located, the largest was selected to be used downstream for colour extraction. Thankfully there was a divergence in approach here, otherwise we’d all be seeing red and throwing accusations of copying (not really, we’re very co-operative).
A mask of the above product image. The white region is the detected object.
With the desired object extracted from the image, it was time to find the colour. Both teams decided to use k-means clustering to do this. The algorithm partitions the image pixels into k clusters where the mean value of each cluster corresponds to that cluster’s pixel value, or colour. We rank the clusters according to their size to obtain the most dominant colours in the image.
Left: Input image. Right: Detected colours from the input image.
Finally, to make it friendly to humans, we used the webcolors module to convert the pixel hex-code into a textual name. In order to find the closest known colour to the cluster’s mean, one team used the k-d tree algorithm, while the other computed it according to the euclidean distance.
The entire colour extraction pipeline.
It was a great experience to quickly implement a solution using Python’s popular image processing and machine learning libraries. Most encouraging was that ideas for improvements continued to flow throughout the day and even afterwards. While we were feeling slightly blue that hack day was over, we are all looking forward to the next one!
