New Active Learning Approach for Seabed Segmentation

arXiv (Cornell University), 2024

Advances in underwater imaging enable the collection of extensive seafloor image datasets that are necessary for monitoring important benthic ecosystems. The ability to collect seafloor imagery has outpaced our capacity to analyze it, hindering expedient mobilization of this crucial environmental information. Recent machine learning approaches provide opportunities to increase the efficiency with which seafloor image datasets are analyzed, yet large and consistent datasets necessary to support development of such approaches are scarce. Here we present BenthicNet: a global compilation of seafloor imagery designed to support the training and evaluation of large-scale image recognition models. An initial set of over 11.4 million images was collected and curated to represent a diversity of seafloor environments using a representative subset of 1.3 million images. These are accompanied by 2.6 million annotations translated to the CATAMI scheme, which span 190,000 of the images. A large deep learning model was trained on this compilation and preliminary results suggest it has utility for automating large and small-scale image analysis tasks. The compilation and model are made openly available for use by the scientific community at https://doi.org/10.20383/103.0614. from image hosting websites. However, this dataset comprises terrestrial and anthropocentric objects and scenes, and does not represent subaqueous environments. The difference in the domain of the input data may limit the capacity for transfer learning. Development of large-scale models using compilations of benthic imagery that are suitable for transfer learning purposes would be ideal, yet this is made difficult by a lack of universal labels for seabed features. One of the primary difficulties associated with developing deep learning models in this context is that, unlike terrestrial and anthropocentric images, there is no objective label for many seabed habitats, biological communities, substrate types, or organisms. Indeed, a number of different classification schemes are used to label benthic features 33-35. Because no single vocabulary is universally applied to describe these features, we currently lack large sets of consistently labelled images that are necessary for training deep learning models for benthic environments. We note an outstanding need to develop standardized protocols for the translation of common marine image labelling schemes. Self-supervised learning (SSL) is a recent technique in which models can learn to understand their stimuli without the use of manually annotated data 36-44. Instead of using labelled data, self-supervised models learn to solve a pretext task that can be automatically constructed from the input data itself. SSL enables the training of large-scale models on unlabelled imagery, which can be collected at scale more easily than annotated imagery. Models trained with SSL have already learnt to see and understand the stimuli of interest, and can subsequently be used for transfer learning onto specific tasks, even if there is only a limited amount of annotated data available for the new task. SSL may enable the training of deep learning models on large-scale benthic image datasets for the purposes of transfer learning on smaller novel tasks (e.g. site-specific habitat labelling), despite the lack of large consistently labelled image datasets. Cumulatively, adequate volumes of benthic image data currently exist to support the development of SSL models, but they are spread globally among various research groups, government data portals, and open data repositories. There is a need to compile and curate datasets for the development of large-scale image recognition models. Such compilations must include images from a range of biomes, depths, and physical oceanographic conditions in order to adequately represent the global heterogeneity of benthic environments. Additionally, data should be included from an array of acquisition platforms and camera configurations to represent the variability in image characteristics (e.g. lighting, resolution, quality, perspective) that arise from non-standardized image data collection methods. The intended applications and scope for a benthic habitat machine learning image dataset dictate qualities that images should possess to be useful for automating tasks in this context. Unlike imagery that is focused solely on specific biota, benthic habitat images often depict a broader area (e.g. on the order of m 2), which necessarily includes the seafloor. The goal of analyzing such data is often to broadly categorize the benthic environment, potentially including both biotic and abiotic elements 45. Biotic characterization may include descriptions of individual organisms 46 or community composition 47 , while abiotic components include description of substrate, sediment bed forms, heterogeneity, rugosity, and relief 48-50. For these reasons, benthic habitat information is often summarized at the whole-image level-for example, by assigning one or several "habitat" labels to an entire image using a pre-defined scheme 34, 35, 51 , or by aggregating individual labels indicating presence or absence, abundance, or percentage cover of individual habitat components, which may be labelled using a more detailed vocabulary 33. It is therefore useful for benthic habitat images to depict a broad enough area so that both abiotic and biotic habitat components may be recognized. This may differ from other forms of marine image labelling that focus on locating specific objects, semantic labelling, bounding boxes, and masking. These forms of labelling are well suited to applications focusing on single taxa, pelagic biota, and object detection or tracking. Efforts to establish extensive image datasets for those applications are also underway 11, 52-61 , and several data portals and software packages support the labelling and centralization of data to support that work (e.g. CoralNet 11 , FathomNet 53 , SQUIDLE+ 62 , BIIGLE 63 , VIAME). Here we describe BenthicNet: a global compilation of seafloor images that is designed to support development of automated image processing tools for benthic habitat data. With this compilation, we strive to obtain thematic diversity by (i) compiling benthic habitat images from locations around the world, and (ii) representing habitats from a broad range of marine environments. The compiled dataset is assessed for these qualities. Additionally, we aim to achieve diversity of non-thematic image characteristics (e.g. image quality, lighting, perspective) by obtaining data from a range of acquisition platforms and camera configurations. The dataset is presented in three parts: a diverse collection of over 11 million seafloor images from around the world, provided without labels (BenthicNet-11M); a rarefied subset of 1.3 million images, selected to maintain diversity in the imagery while reducing redundancy and volume (BenthicNet-1M); and a collection of 188 688 labelled images bearing 2.6 million annotations (BenthicNet-Labelled). We provide a large SSL model pretrained on BenthicNet-1M, and demonstrate its application using examples from BenthicNet-Labelled. The compilation and SSL model are made openly available to foster further development and assessment of benthic image automation tools. 2 Methods In order to achieve a diverse collection of benthic habitat images for training deep learning models, data spanning a range of environments and geographies were obtained from a variety of sources. These initially included project partners and research 3/140 contacts, which were leveraged to establish additional data partnerships with individuals, academic and not-for-profit research groups, and government organizations. The largest data volumes were eventually obtained from several academic, government, and third-party public data repositories. The acquisition of labelled data was prioritized in all cases, but extensive high quality unlabelled data collections were also included where feasible. The desired format for each dataset was a single folder containing unique images, accompanied by a single comma separated value (CSV) file indicating, at a minimum, the dataset, file name, latitude, longitude, date and time of acquisition, URL (if hosted online) and label(s) (if provided) for each image. 2.1 Data compilation and quality control Labelled benthic image data was initially obtained from project collaborators, data partners, and opportunistic sources such as academic journal supplementary materials. The formats and varieties of data were diverse, including collections of images with spreadsheet metadata, images with metadata contained in file names, GIS files containing images from which metadata was extracted, lists of URL image links, and raw video with text file annotations. Datasets that were not formatted as a single folder of images or list of URL links with CSV metadata were re-formatted upon receipt. Metadata contained in image file names was parsed and used to construct a metadata CSV file where necessary. Image data contained within GIS files was extracted using ArcGIS Pro and the ArcPy Python package, along with geographic information and other metadata contained within the files. All geographic coordinates were converted to decimal degrees using the WGS 84 datum. Data obtained as video files were subsampled by extracting still frames according to their metadata using FFmpeg. After formatting, all datasets were subjected to quality control checks for missing entries, duplicates, label consistency, image quality, and matches between images and metadata. Data columns were renamed to match a standardized format for the BenthicNet dataset. All quality control and formatting was completed using R and Python. The dataset sources are summarized in Table 1. Additional detail on the individual datasets is provided in Appendix C. 2.1.1 Individual contributions A number of datasets...

Photogrammetric Engineering & Remote Sensing

In this paper, we propose a machine learning-based oil spill segmentation using aerial images. In detail, a novel deep neural network-based object segmentation, named Deep Believe Active Contours (DBAC), is introduced, where a pre-trained deep belief neural network is utilized to guide the moments of active contours. Results show that (1) Unsupervised pre-trained deep neural network can efficiently control the evolution of active contour segmentation of oil spill regions; and (2) When applying the proposed DBAC algorithm on the test data from an oil spill image database, it produced a recall rate of 66% and a precision rate of 60%, which outperformed the state-of-the-art methods in the range of 4% ~ 18% and 1% ~ 10%, respectively. Moreover, DBAC produced a better Hausdorff distance (an amount of 13.34) compared to the competing methods. These results show the promises of DBAC for the task of oil spill segmentation in ocean environment.

Deep learning, also known as deep machine learning or deep structured learning based techniques, have recently achieved tremendous success in digital image processing for object detection and classification. As a result, they are rapidly gaining popularity and attention from the computer vision research community. There has been a massive increase in the collection of digital imagery for the monitoring of underwater ecosystems, including seagrass meadows. This growth in image data has driven the need for automatic detection and classification using deep neural network based classifiers. This paper systematically describes the use of deep learning for underwater imagery analysis within the recent past. The analysis approaches are categorized according to the object of detection , and the features and deep learning architectures used are highlighted. It is concluded that there is a great scope for automation in the analysis of digital sea-bed imagery using deep neural networks, especially for the detection and monitoring of seagrass.

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018

Semantic segmentation is a fundamental research in remote sensing image processing. Because of the complex maritime environment, the sea-land segmentation is a challenging task. Although the neural network has achieved excellent performance in semantic segmentation in the last years, there are a few of works using CNN for sea-land segmentation and the results could be further improved. This paper proposes a novel deep convolution neural network named DeepUNet. Like the U-Net, its structure has a contracting path and an expansive path to get high resolution output. But differently, the DeepUNet uses DownBlocks instead of convolution layers in the contracting path and uses UpBlock in the expansive path. The two novel blocks bring two new connections that are U-connection and Plus connection. They are promoted to get more precise segmentation results. To verify our network architecture, we made a new challenging sea-land dataset and compare the DeepUNet on it with the SegNet and the U-Net. Experimental results show that DeepUNet achieved good performance compared with other architectures, especially in high-resolution remote sensing imagery.

2021