VISION: VIdeo StabilisatION using automatic features selection for image velocimetry analysis in rivers

VISION is open-source software written in MATLAB for video stabilisation using automatic features detection. It can be applied for any use, but it has been developed mainly for image velocimetry applications in rivers. It includes a number of options that can be set depending on the user’s needs and intended application: 1) selection of different feature detection algorithms (seven to be selected with the flexibility to choose two simultaneously), 2) definition of the percentual value of the strongest features detected to be considered for stabilisation, 3) geometric transformation type, 4) definition of a region of interest on which the analysis can be performed, and 5) visualisation in real-time of stabilised frames. One case study was deemed to illustrate VISION stabilisation capabilities on an image velocimetry experiment. In particular, the stabilisation impact was quantified in terms of velocity errors with respect to field measurements obtaining a significant error reduction of velocities. VISION is an easy-to-use software that may support research operating in image processing, but it can also be adopted for educational purposes.

How to cite: Pizarro, A., S.F. Dal Sasso, S. Manfreda, VISION: VIdeo StabilisatION using automatic features selection for image velocimetry analysis in rivers, SoftwareX, Volume 19,  101173, 2022. [pdf]

HARMONIOUS GENERAL ASSEMBLY

University of Debrecen, Hungary


Detailed program of the Workshop

08:30 – 09:30 Registration

09:30-09:50: László Bertalan, Brigitta Tóth: Welcome from local organizers
09:50-10:05: Salvatore Manfreda – The achievements HARMONIOUS COST Action
10:05-10:25: Plenary talk: Francesco Nex – Towards real-time UAV mapping: example, challenges and opportunities
10:25-10:55Plenary talk: James Dietrich – Drones for River Monitoring, a ten-year perspective

10:55-11:20 Coffee Break

11:20-11:35: Eyal Ben-Dor – Summary of WG5: Harmonization of methods and results
11:30-11:50: Sorin Herban – Summary of WG1: UAS data processing
11:50-12:05: Jana Müllerová – Summary of WG2: Vegetation status (part 1)
12:05-12:20: Antonino Maltese – Summary of WG2: Vegetation status (part 2)
12:20-12:35: Yijian Zeng – Summary of WG3: Soil moisture content
12:35-12:50: Dariia Strelnikova – Summary of WG4: River monitoring

12:50-14:30: Lunch break

14:30-14:45: Gábor Papp – HungaroControl’s Air-Ground-Air communication concept in order to enable UAVs’ ecosystem
14:45-15:00: Géza Király et al. – UAS and their application in forest monitoring
15:00-15:15: Gábor Bakó et al. – HRAM: High Spatial Resolution Aerial Monitoring Network for Nature Conservation
15:15-15:30: Ferenc Kovács et al. – Application of UAV imagery in environmental research at the University of Szeged
15:30-15:45: Anette Eltner et al. – Hydro-morphological mapping of river reaches using videos captured with UAS
15:45-16:00: Ilyan Kotsev et al. – UAS-aided bedform and habitat mapping of Bolata Cove, Bulgarian Black Sea

16:00-16:30: Coffee Break

16:30-16:50: Lance R. Brady – UAS for Research and Applied Science in the United States Geological Survey
16:50-17:05: Kamal Jain et al. – Crop identification and classification from UAV images using conjugated dense convolutional neural network
17:05-17:20: Nicolas Francos et al. – Mapping Water Infiltration Rate Using Ground and UAV Hyperspectral Data: A Case Study of Alento, Italy
17:20-17:35: Martin Jolley et al. – Considerations When Applying UAS-based Large-Scale PIV and PTV for Determining River Flow Velocity
17:35-17:50: Adrian Gracia-Romero et al. – UAS plant phenotyping under abiotic stresses
17:50-18:05: Shawn C. Kefauver et al. – High-resolution UAV Imaging for Forest Productivity Monitoring

Planning the future of Harmonious

The Department of Topography and Cartography of the Technical University of Madrid hosted our work group meeting of COST Action – HARMONIOUS from 27 up to the 30 of October.

During this meeting the WG1 finalized the Glossary of terms used for UAS-based applications considering the three macro categories : platform and equipment, software and outputs.

GLOSSARY

1 Category: Platforms and Equipment 

  • Global Navigation Satellite System (GNSS) is a constellation of satellites used for positioning a receiver on the ground.
  • GALILEO is the GNSS European solution used to determine the ground position of an object.  
  • GPS is the most common GNSS based on the reception of signals from about 24 orbiting satellites by the USA, used to determine the ground position of an object. This global and accurate system allows users to know their exact location, velocity, and time 24 hours per day, anywhere in the world.    
  • Light Detection and Ranging (LiDAR) is based on laser pulses to locate the acquired point cloud in a 3D remote sensing. LiDAR data products are often managed within a gridded or raster data format.
  • Multispectral imaging captures image data within specific wavelength ranges across the electromagnetic spectrum.  The used spectral regions are often at least partially outside the visible spectral range, covering parts of the infrared and ultraviolet region. For example, a multi-spectral imager may provide wavelength channels for near-UV, red, green, blue, near-infrared, mid-infrared and far-infrared light – sometimes even thermal radiation.
  • Near Infrared (NIR) is a subset of the infrared band that is just outside the range of what humans can see. Applied to cameras, NIR cameras cover the wavelength range of 900 to 1700 nm, a range that is best suited for absorption and radiation characteristics analyses.
  • Noise    is an irregular fluctuation that accompanies a transmitted electrical signal but is not part of it and tends to obscure it. The main sources of noise can be divided into two main categories: the physical noise, linked to physics constraints like the corpuscular nature of light, and the hardware noise, linked to mechanical issues in the camera.
  • Optical Camera is a photographic device aimed to form and record an image of an object. An optical camera sensor is an imager that collects visible light (400~700nm).
  • Payload is the weight a drone or unmanned aerial vehicle (UAV) can carry on board. It is usually counted outside of the weight of the drone itself and includes anything additional to the drone – such as extra cameras, sensors, or packages for delivery.
  • Pixel size of an image identifies the spatial resolution and it is dependent on the sensor capabilities. It provides a measure of the image resolution, which is higher with finer grids, where the degree of recognizable details increases.
  • RGB Camera is equipped with a standard Complementary Metal Oxide Semiconductor (CMOS) sensor through which the colourful images of persons and objects are acquired. In a CMOS sensor, the charge from the photosensitive pixel is converted to a voltage at the pixel site and the signal is multiplied by row and column to multiple on chip Digital-to-Analog Converters (DACs). In a RGB camera, the acquisition of static photos is commonly expressed in megapixels that define the amount of pixels in a singular photo. While, the acquisition of videos is usually expressed with terms such as Full HD or Ultra HD.        
  • Thermal Camera is a non-contact temperature measurement sensor. All objects (above absolute zero) emit infrared energy as a function of their temperature. The vibration of atoms and molecules generates infrared energy. The higher the temperature of an object, the faster its molecules and atoms move. This movement is emitted as infrared radiation, which our eyes cannot see but our skin can feel (as heat). Thermal imaging uses special infrared camera sensors to illuminate a spectrum of light invisible to the naked eye. Thermal energy is invisible to the naked eye and works in different ways; it can be emitted, absorbed, or reflected. Infrared cannot see through objects but can detect differences in radiated thermal energy between materials. This is known as thermal bridging or heat transfer. 
  • Unmanned Aerial System (UAS) is a remotely controlled professional system integrating several technological components (e.g., navigation system, gyroscope, and sensors) in order to perform spatial observations.
  • Unmanned Aerial Vehicle (UAV) is a remotely controlled vehicle able to perform several operations and observations.

2 Software 

  • Aero-triangulation is the method most frequently applied to the photogrammetry to determine the X, Y, and Z ground coordinates of individual points based on photo coordinate measurements. The purpose of aero-triangulation is to increase the density of a geodetic network in order to provide images with an exhaustive number of control points for topographic mapping. Deliverables from aero-triangulation may be three-dimensional or planimetric, depending on the number of point coordinates determined.
  • Checkpoints are Ground Control Points (GCPs) used to validate the relative and absolute accuracy of the geo-localization of maps. The checkpoints are not used for processing. Instead, they are used to calculate the error of the map by comparing the known measured locations of the checkpoints to the coordinates of the checkpoints shown on the map.
  • Flight Type refers to the flight mission mode (manual or autonomous). In the manual mode, a pilot manages the UAS during the flight. The autonomous mission is programmed to react to various types of events, in a preset and direct way by means of special sensors. This makes UAS flight predictable and subject to intervention by a remote pilot, only if necessary.
  • Flight Time is a measurement of the total time needed to complete a mission, from the first to the last image taken during a flight. Flight time can be used to characterize the wind impacts on flight performance of UAS.    
  • Ground Control Points (GCPs) are user defined and priorly determined tie points within the mapping polygon used in the process of indirectly georeferencing UAS images. Such tie points can be permanent or portable markers with or without georeferenced data.
  • Masking is the procedure of excluding some part of the scene from image analysis. For instance, clouds, trees, bushes and their shadows should not be considered in further processing, such as in vegetation studies for the evaluation of crop vegetation indices.        
  • Orthorectification is a process of linearly scaling the image pixel size to real-world distances. This is achieved by accounting for the impacts of camera perspective and relative height above the sensed object. The objective is the reprojection of the original image, which could be captured from oblique viewing angles looking at unlevelled terrain, into an image plane to generate a distortion-free photo. 
  • Point Cloud is a collection of data points in a three-dimensional plane. Each point contains several measurements, including its coordinates along the X, Y, and Z-axes, and sometimes additional data such as a color value, which is stored in RGB format, and luminance value, which determines how bright the point is.
  • Radiometric Calibration is a process that allows the transformation of the intensities or digital numbers (DN) of multiple images in order to describe an area and detect relative changes of the landscape, removing anomalies due to atmospheric factors or illumination conditions. 
  • Structure from Motion (SfM) is the process of reconstructing a three-dimensional model from the projections derived from a series of images taken from different viewpoints. Camera orientation and scene geometry are reconstructed simultaneously through the automatic identification of matching features in multiple images.        
  • Tie Point is a point in a digital image or aerial photograph that can be found in the same location in an adjacent image or aerial photograph. A tie point is a feature that can be clearly identified in two or more images and selected as a reference point and whose ground coordinates are not known. The ground coordinates of Tie Points are computed during block triangulation. So, Tie points represent matches between key points detected on two (or more) different images and represent the link between images to get 3D relative positioning.
  • Precision is a description of random errors in the 2D/3D representations.
  • Quality Assessment is an estimation of the statistical geometric and radiometric errors of the final products obtained using ground true data.           

UAS-based Outputs

  • 2D Model is a bidimensional representation of the earth that contains 2 coordinates X and Y.
  • 3D Model is a mathematical or virtual representation of a three dimensional object.
  • 2.5D Model (Pseudo 3D Model) is a three-dimensional representation that uses X, Y coordinates, which are associated to a single elevation value in order to relate different points.
  • Digital Elevation Model (DEM) or Digital Height Model (DHM) is a gridded image describing the altitude of the earth excluding all other objects artificial or natural.    
  • Digital Surface Model (DSM) is a gridded image describing the altitude of the earth including all other objects artificial or natural. For instance, the DSM provides information about dimensions of buildings and forests.    
  • Digital Terrain Model (DTM) is a vector or raster dataset consisting of a virtual representation of the land environment in the mapping polygon. In a DTM the height of the point belongs to the bare ground.
  • Orthophoto is an aerial or terrestrial photograph that has been geometrically corrected to make the scale of the photograph uniform and use it as a map. Since each pixel of the orthophoto has a X and Y, it can be overlapped to other orthophotos, and it can be used to measure true distances of features within the photograph.        
  • Orthomosaic    is a high resolution image made by the combination of many orthophotos. It is a single, radiometrically corrected image that offers a photorealistic representation of an area that can produce surveyor-grade measurements of topography, infrastructure, and buildings.    
  • Feature Identification is a vector information computed from images using artificial intelligence algorithms in order to identify objects (roads, buildings, bridges, etc.) automatically. 
  • Point Cloud is a set of data points in space representing a three-dimensional object. Each point position has its set of Cartesian coordinates (X, Y, Z). It can be generated from overlapping images or LiDAR sensors.
  • Point Cloud Classification is the output of an algorithm that classifies the points of a cloud by computing a set of geometric and radiometric attributes.
  • Image Segmentation is a process that detects the features of an image clearly distinguishable based on the image texture and color.
  • Triangulated Irregular Network (TIN) is a pseudo three-dimensional representation obtained from the  relations in a point cloud using triangles.   
  • Vegetation Indices (VIs) are combinations of surface reflectance at two or more wavelengths designed to highlight a particular property of vegetation. VIs are designed to maximize sensitivity to the vegetation characteristics while minimizing confounding factors such as soil background reflectance, directional, or atmospheric effects. VIs can be found in the scientific literature under different forms such as NDVI, EVI, SAVI, etc.                
  • Aerial photograph is an image taken from an air-borne (i.e., UAS) platform using a precision camera. From aerial photographs, it is possible to derive qualitative information of the depicted areas, such as land use/land cover, topographical forms, soil types, etc. 
  • Terrestrial photograph is an image taken from the earth surface using a camera with an orientation that in most cases is not Nadiral.               

A comparison of tools and techniques for stabilising unmanned aerial system (UAS) imagery for surface flow observations

While the availability and affordability of unmanned aerial systems (UASs) has led to the rapid development of remote sensing applications in hydrology and hydrometry, uncertainties related to such measurements must be quantified and mitigated. The physical instability of the UAS platform inevitably induces motion in the acquired videos and can have a significant impact on the accuracy of camera-based measurements, such as velocimetry. A common practice in data preprocessing is compensation of platform-induced motion by means of digital image stabilisation (DIS) methods, which use the visual information from the captured videos – in the form of static features – to first estimate and then compensate for such motion. Most existing stabilisation approaches rely either on customised tools developed in-house, based on different algorithms, or on general purpose commercial software. Intercomparison of different stabilisation tools for UAS remote sensing purposes that could serve as a basis for selecting a particular tool in given conditions has not been found in the literature. In this paper, we have attempted to summarise and describe several freely available DIS tools applicable to UAS velocimetry. A total of seven tools – six aimed specifically at velocimetry and one general purpose software – were investigated in terms of their (1) stabilisation accuracy in various conditions, (2) robustness, (3) computational complexity, and (4) user experience, using three case study videos with different flight and ground conditions. In an attempt to adequately quantify the accuracy of the stabilisation using different tools, we have also presented a comparison metric based on root mean squared differences (RMSDs) of inter-frame pixel intensities for selected static features. The most apparent differences between the investigated tools have been found with regards to the method for identifying static features in videos, i.e. manual selection of features or automatic. State-of-the-art methods which rely on automatic selection of features require fewer user-provided parameters and are able to select a significantly higher number of potentially static features (by several orders of magnitude) when compared to the methods which require manual identification of such features. This allows the former to achieve a higher stabilisation accuracy, but manual feature selection methods have demonstrated lower computational complexity and better robustness in complex field conditions. While this paper does not intend to identify the optimal stabilisation tool for UAS-based velocimetry purposes, it does aim to shed light on details of implementation, which can help engineers and researchers choose the tool suitable for their needs and specific field conditions. Additionally, the RMSD comparison metric presented in this paper can be used in order to measure the velocity estimation uncertainty induced by UAS motion.

How to cite: Ljubičić, R., Strelnikova, D., Perks, M. T., Eltner, A., Peña-Haro, S., Pizarro, A., Dal Sasso, S. F., Scherling, U., Vuono, P., and Manfreda, S.: A comparison of tools and techniques for stabilising unmanned aerial system (UAS) imagery for surface flow observations, Hydrol. Earth Syst. Sci., 25, 5105–5132, https://doi.org/10.5194/hess-25-5105-2021, 2021. [pdf]

Recent Advancements and Perspectives in UAS-Based Image Velocimetry

Videos acquired from Unmanned Aerial Systems (UAS) allow for monitoring river systems at high spatial and temporal resolutions providing unprecedented datasets for hydrological and hydraulic applications. The cost-effectiveness of these measurement methods stimulated the diffusion of image-based frameworks and approaches at scientific and operational levels. Moreover, their application in different environmental contexts gives us the opportunity to explore their reliability, potentialities and limitations, and future perspectives and developments. This paper analyses the recent progress on this topic, with a special focus on the main challenges to foster future research studies.

How to cite: Dal Sasso, S.F.; Pizarro, A.; Manfreda, S. Recent Advancements and Perspectives in UAS-Based Image VelocimetryDrones5, 81, 2021. [pdf]

Increasing LSPIV performances by exploiting the seeding distribution index

Image-based approaches for surface velocity estimations are becoming increasingly popular because of the increasing need for low-cost river flow monitoring methods. In this context, seeding characteristics and dynamics along the video footage represent one of the key variables influencing image velocimetry results. Recent studies highlight the need to identify parameter settings based on local flow conditions and environmental factors apriori, making the use of image velocimetry approaches hard to automatise for continuous monitoring. The seeding distribution index (SDI) – recently introduced by the authors – identifies the best frame window length of a video to analyse, reducing the computational loads and improving image velocimetry performance. In this work, we propose a method based on an average SDI time series threshold with noise filtering. This method was tested on three case studies in Italy and validated on one in UK, where a relatively high number of measurements is available. Following this method, we observed an error reduction of 20-39% with respect to the analysis of the full video. This beneficial effect appears even more evident when the optimisation is applied at sub-sector scales, in cases where SDI shows a marked variability along the cross-section. Finally, an empirical parameter t was proposed, calibrated, and validated for practical uses to define the SDI threshold. tshowed relatively stable values in the different contexts where it has been applied. Application of the seeding index to image-based velocimetry for surface flow velocity estimates is likely to enhance measurement accuracy in future studies.

Keywords: Image Velocimetry, UAS, river flow monitoring, LSPIV, seeding metrics, Seeding Distribution Index, frame footage.

How to cite: Dal Sasso, S.F., A. Pizarro, S. Pearce, I. Maddock, S. Manfreda, Increasing LSPIV performances by exploiting the seeding distribution index at different spatial scales, Journal of Hydrology, 2021. [pdf]


A comparison of tools and techniques for stabilising UAS imagery for surface flow observations

This research presents an investigation of different strategies and tools for digital image stabilisation for image velocimetry purposes. Basic aspects of image stabilisation and transformation are presented, and their applicability is discussed in terms of image velocimetry. Seven free-to-use open-source tools (six community-developed and one off-the-shelf) are described and compared according to their stabilisation accuracy, robustness in different flight and ground conditions, computational complexity, ease of use, and other capabilities. A novel approach for fast stabilisation accuracy analysis is also developed, presented, and applied to the stabilised image sequences. Based on the obtained results, some general guidelines for choosing a suitable tool for specific image velocimetry tasks have been obtained. This research also aims to provide a basis for further development or improvement of digital image stabilisation tools, as well as for the analyses of stabilisation impact on image velocimetry results.

How to cite: Ljubičić, R., D. Strelnikova, M. T. Perks, A. Eltner, S. Peña-Haro, A. Pizarro, S. F. Dal Sasso, U. Scherling, P. Vuono, and S. Manfreda, A comparison of tools and techniques for stabilising UAS imagery for surface flow observations, Hydrology and Earth System Sciences, 2021. [pdf]

VISION: VIdeo StabilisatION using automatic features selection

VISION is one of the stabilization algorithms tested in recent manuscript entitled “A comparison of tools and techniques for stabilizing UAS imagery for surface flow observations”. The Matlab code and an set of data can be downloaded on OSF.

The “StabilisationFunction.m” is a Matlab function aiming at stabilising videos for image velocimetry analyses in rivers. It is a command-line function without GUI at the moment. An example of how to call the stabilisation function is also provided in the file “ExampleScript.m”. All the codes were written in Matlab R2020a.

CODE: https://doi.org/10.17605/OSF.IO/HBRF2

How to cite: Pizarro, A., S.F. Dal Sasso, S. Manfreda, VISION: VIdeo StabilisatION using automatic features selection, DOI 10.17605/OSF.IO/HBRF2, 2021.

Optimal spatial distribution of tracers for velocimetry applications

River monitoring is of particular interest as a society that faces increasingly complex water management issues. Emerging technologies have contributed to opening new avenues for improving our monitoring capabilities but have also generated new challenges for the harmonised use of devices and algorithms. In this context, optical-sensing techniques for stream surface flow velocities are strongly influenced by tracer characteristics such as seeding density and their spatial distribution. Therefore, a principal research goal is the identification of how these properties affect the accuracy of such methods. To this aim, numerical simulations were performed to consider different levels of tracer clustering, particle colour (in terms of greyscale intensity), seeding density, and background noise. Two widely used image-velocimetry algorithms were adopted: (i) particle-tracking velocimetry (PTV) and (ii) particle image velocimetry (PIV). A descriptor of the seeding characteristics (based on seeding density and tracer clustering) was introduced based on a newly developed metric called the Seeding Distribution Index (SDI). This index can be approximated and used in practice as SDI=ν0.1/(ρ/ρcν1), where νρ, and ρcν1 are the spatial-clustering level, the seeding density, and the reference seeding density at ν=1, respectively. A reduction in image-velocimetry errors was systematically observed for lower values of the SDI; therefore, the optimal frame window (i.e. a subset of the video image sequence) was defined as the one that minimises the SDI. In addition to numerical analyses, a field case study on the Basento river (located in southern Italy) was considered as a proof of concept of the proposed framework. Field results corroborated numerical findings, and error reductions of about 15.9 % and 16.1 % were calculated – using PTV and PIV, respectively – by employing the optimal frame window.

How to cite: Pizarro, A., S.F. Dal Sasso, M. Perks and S. Manfreda, Identifying the optimal spatial distribution of tracers for optical sensing of stream surface flow, Hydrology and Earth System Sciences, 24, 5173–5185, (10.5194/hess-24-5173-2020) 2020. [pdf]

Seeding metrics for error minimisation

River streamflow monitoring is currently facing a transformation due to the emerging of new innovative technologies. Fixed and mobile measuring systems are capable of quantifying surface flow velocities and discharges, relying on video acquisitions. This camera-gauging framework is sensitive to what the camera can “observe” but also to field circumstances such as challenging weather conditions, river background transparency, transiting seeding characteristics, among others. This short communication paper introduces the novel idea of optimising image velocimetry techniques selecting the most informative sequence of frames within the available video. The selection of the optimal frame window is based on two reasonable criteria: i) the maximisation of the number of frames, subject to ii) the minimisation of the recently introduced dimensionless seeding distribution index (SDI). SDI combines seeding characteristics such as seeding density and spatial clustering of tracers, which are used as a proxy to enhance the reliability of image velocimetry techniques. Two field case studies were considered as a proof-of-concept of the proposed framework, on which seeding metrics were estimated and averaged in time to select the proper application window. The selected frames were analysed using LSPIV to estimate the surface flow velocities and river discharge. Results highlighted that the proposed framework might lead to a significant error reduction. In particular, the computed discharge errors, at the optimal portion of the footage, were about 0.40% and 0.12% for each case study, respectively. These values were lower than those obtained, considering all frames available.

How to cite: Pizarro, A., S. F. Dal Sasso, S. Manfreda, Refining image‐velocimetry performances for streamflow monitoring: Seeding metrics to errors minimisation, Hydrological Processes, (doi: 10.1002/hyp.13919 ), 2020.