Visualising Multi-Sensor Predictions from a Rice Disease Classifier

The Microsoft Rice Disease Classification Challenge introduced a dataset comprising RGB and RGNiR (RG-Near-infra-Red) images. This second image type increased the difficulty of the challenge such that all of the winning models worked with RGB only. In this challenge we applied a res2next50 encoder that was first pre-trained with self-supervised learning through the SwAV algorithm, to represent each RGB and their corresponding RGNIR images with the same weights. The encoder was then fine-tuned and self-distilled to classify the images which produced a public test set score of 0.228678639, and a private score of 0.183386940. K-fold cross-validation was not used for this challenge result. To better understand the impact of self-supervised pre-training on the problem of classifying each image type, we apply t-distributed Stochastic Neighbour Embedding (t-SNE) on the logits (predictions before applying softmax). We show how this method graphically provides some of the value of a confusion matrix, by locating some incorrect predictions. We then render the visualisation by overlaying the raw images in each data point, and note that to this model, the RGNIR images do not appear to be inherently more difficult to categorise. We make no comparisons through sweeps, RGB-only models or RGNIR-only models. This is left to future work.