Thin Wire Segmentation and Reconstruction Based on a Novel Image Cropping and Stitching Algorithm in Apple Fruiting Wall Architecture for Robotic Picking

Abstract

The layout of orchards usually requires the use of wires, to provide sturdy support. Such is the case of apple trees in fruiting wall architecture, where wires are conducive for mechanical harvesting and especially robotic picking. However, wires may cause damage to the robotic gripper, especially when direct picking occluded apple fruits. Hence, the importance of identifying the wires of a fruit wall architecture. In this study, a pixel-level segmentation network, BlendMask, was adopted to segment wires. The wires are thin and normally behind the branches or leaves, making difficult for their identification. Therefore, a novel data processing algorithm called image cropping and stitching (ICS) is proposed for BlendMask to segment the wires. A total of 82 RGB (Red, Green, and Blue) images registered to create a raw dataset. The dataset was augmented and then cropped into 12736 images with a resolution of 800 × 1024 pixels and corresponding annotation files based on input size of BlendMask to make a cropped dataset. Then BlendMask was trained with the cropped dataset and tested on the cropped images, where additional stitching for the cropped images was needed in the image testing dataset. Results showed that BlendMask with ICS obtained IoU and pixel accuracy (PA) of 43.86% and 61.01%, respectively, where BlendMask achieved better average precision (AP) and AP with Intersection over Union (IoU) of 0.5 (AP50) of 13.42% and 38.75% in the cropped dataset, which were 13.29% and 38.42% higher than the uncropped dataset, respectively. Moreover, a reconstruction method based on feature point extraction and fitting (FPEF) was proposed to estimate wire skeletons, which achieved a reconstruction accuracy of 90.70%. These results showed a promising potential using segmentation and reconstruction methods for identifying wires and thus providing a basis for robotic picking in modern orchards.

Keywords: BlendMask, Feature point extraction and fitting, Fruiting wall architecture, Image cropping and stitching, Wire reconstruction