Photo-Identification of Marine Cetaceans Using Convolutional Neural Networks


Modelling cetacean (whale, dolphin, and porpoise) population dynamics and behaviour is paramount to effective population management and conservation. Robust data is required for the design and implementation of conservation strategies and to assess the risks presented by anthropogenic activity such as offshore wind turbines and commercial fishing.

Moreover, cetaceans make prime candidates for modelling ecosystem change under the ecosystem sentinel concept as they reflect the current state of the ecosystem and respond to change across different spatial and temporal scale. As the global climate changes and urbanisation of coastal areas intensifies, it is imperative to develop methodologies for quick and effective assessment of the biological and ecological impact of rising sea temperatures, pollution, and habitat degradation. This can be achieved through modelling the population, behaviour, and health of large marine species such as dolphins.

Methodologies of cetacean research includes photo identification (photo-id). Photo-id involves collecting photographic data and identifying individuals based on unique permanent markings, and has been used for more than 40 years for modelling cetacean population dynamics and ecology. Current identification techniques for cetaceans rely heavily on experts manually identifying individuals. This can often be costly due to the number of person-hours required for identification, as well as the large potential for error due to issues such as observer fatigue.

Further, individual identification of dolphins within a species is time consuming due to the nature of the task. Intra-species dolphins have very similar markings and body types making identifying an individual within a pod very difficult. Prominent features must be identified, such as small nicks to the fins or scars left from injuries to identify an individual. If these features are only prominent on one side of the individual, the task of identification becomes even more difficult.

With progressively more data being collected during fieldwork through increased use of technology, there is an urgent need for an automatic system for quick identification with reduced error rates. Previous efforts to photo-id individuals from underwater video stills from previous expeditions undertaken by Newcastle University’s School of Natural & Environmental Science’s Marine MEGAfauna Lab took around three months from raw video file to be completely catalogued. This project addresses these limitations by applying the methodologies, techniques, and computational power of deep learning to the field of marine biology. Deep learning models, specifically Convolutional Neural Networks (CNNs), will be trained on high-end computer clusters using the Microsoft Azure Cloud prior to field studies using existing data. Once trained, the models can be ran on field deployable computers to perform image analysis in real time from multiple data sources (underwater and above water images).

Data collection for this project focussed on a population of white-beaked dolphins (Lagenorhynchus albirostris) off the coast of North-East England. Recent research has identified sites where the species is regularly sighted and underwater image analysis has shown seasonal and multi-year residency. A health assessment based on underwater image analysis identified high incidence of skin disease and trauma suggesting conservation of this population should be high priority. The species would also serve as a prime sentinel for monitoring North Sea climatic changes as it shows preference for cold water with North-East UK coastal waters representing the southern limit of its range.

The Northumberland Dolphin Dataset

Conservation is an area with great potential for computer vision utilisation. Cetacean conservation could especially benefit from the introduction of computer vision aides. Current manual identification processes often takes many months to complete, and thus any help from computer vision systems capable of fine-grained cetacean classification would afford researchers more time in the field and less time processing collected data.

Very few open-source datasets exist for use within a conservation or ecological space; those that do often focus on simple object detection of animals in a scene. Datasets often focus on a particular subset of animals such as pets commonly found in homes or birds. Some large scale datasets showing animals in natural environments do exist, although these often only provide labels at a species level, which is not fine-grained enough for population estimation which requires the identification of individuals. Of the datasets which do allow for species identification currently, most primarily focus on the development of land- based camera trap systems, although work has also been undertaken in the development of marine life species detection systems.

In order to aid the development of automatic photo-id systems, including my own, a large part of my PhD has focussed on the development of the Northumberland Dolphin Dataset, a challenging image dataset annotated for both coarse and fine-grained instance segmentation and categorisation.

Currently the first release of the Northumberland Dolphin Dataset is available. This can be downloaded here.


Trotter, C., Atkinson, G., Sharpe, M., Richardson, K., McGough, A.S., Wright, N., Burville, B. and Berggren, P., 2020. NDD20: A large-scale few-shot dolphin dataset for coarse and fine-grained categorisation. arXiv preprint arXiv:2005.13359.

Trotter, C., Atkinson, G., Sharpe, M., McGough, A.S., Wright, N. and Berggren, P., 2019. The Northumberland Dolphin Dataset: A Multimedia Individual Cetacean Dataset for Fine-Grained Categorisation. arXiv preprint arXiv:1908.02669.