Learning Fluid Flow Visualizations from In-flight Image with Tufts
Abstract
For better understanding of fluid flows around aerial systems, strips of wire or rope, widely known as tufts, are often used to visualize the local flow direction. This paper presents a computer vision system that automatically extracts the shape of tufts from images, which have been collected during real flights of a helicopter and an unmanned aerial vehicle (UAV). As images from these aerial systems present challenges to both the model-based computer vision and the end-to-end supervised deep learning techniques, we propose a semantic segmentation pipeline that consists of three uncertainty-based modules namely, (a) active learning for object detection, (b) label propagation for object classification, and (c) weakly supervised instance segmentation. Overall, these probabilistic approaches facilitate the learning process without requiring any manual annotations of semantic segmentation masks. Empirically, we motivate our design choices through comparative assessments and provide real world demonstrations of the proposed concept, for the first time to our knowledge.
Note: data-set from this publication can be downloaded at: https://zenodo.org/records/10539950
Data collection with manned helicopter
The Fenestron is an anti-torque device for helicopters, also known as a fan-in-fin or ducted tail rotor. Many helicopter models of Airbus Helicopters are currently fitted with a Fenestron. Compared to a traditional tail rotor the Fenestron has the benefit of increased protection to foreign object collision, noise shielding and increased aerodynamic efficiency (at least in hover). One of the drawbacks is a more complex design due to many aerodynamic flow regimes for different flight conditions. In order to better understand the flow inside and around the Fenestron duct, for different flight conditions (including significant side slip), tufts were placed in and around the Fenestron duct and tail surfaces. The tufts consist of black polypropylene rope with a length of 160 mm and a diameter of 3 mm, which are taped to the surface with 3M aluminum tape. In general thinner tufts display better flow alignment, but are more difficult to observe at large distance. The initial idea was to film the tufts with one or two small cameras mounted on the horizontal stabilizer. However, the estimated certification effort was high, the field of view would not include the inner forward part of the Fenestron duct and there were concerns about vibrations that would blur the images. As an alternative it was decided to film the tufts from a second helicopter flying in close formation. In addition to the tufts, temperature sensor strips were placed on the tail surface to record the maximum temperature due to the exhaust gases. This was necessary to ensure that the tufts would not burn or detach due to reduced adhesion of the tape that was used to fix the tufts to the surface.
Similar results, to that obtained with tufts, can be obtained with oil flow visualization. However, the oil visualization provides a time averaged flow field, whereas tufts provide instantaneous data. For flight testing oil visualization is less appropriate since the flight condition of interest must be reached, before the oil settles, unless a sophisticated oil/ink release system is used. This problem is less pronounced in wind tunnel testing, where the condition of interest can be reached quite quickly and kept for sufficiently long time.
For the tests EC135-ACT/FHS helicopter was equipped with tufts and the Bo105 helicopter was used as a camera platform. Examples of the images that were gathered are given in Figure. The left shows an image of the left-hand side while a view of the right side is shown on the right. The individual tufts are marked with a letter/number combination in red (the tufts in the stator blades and hub are not annotated in order to avoid clutter). To allow for a good optical view one side door of the Bo105 was removed, which limited its maximum airspeed to 110 kt. Flying in close (<50 m) formation requires good coordination and communication between both flight crews. Maintaining the correct relative position with respect to the formation leader becomes even more challenging when the formation leader is flying with large side slip. The reference points, used by the wingman to estimate his relative position to the formation leader, change with side slip. Furthermore, high side slip of the formation leader, with the nose towards the wingman, creates the optical impression of being on a collision course. All these conditions results in high pilot workload and increased collision risk. Therefore, formation flying was limited to one hour per flight. For example, see the figure below for some examples.
For image capturing a Nikon D7100, with a 55-300 mm zoom optic, vibration reduction and autofocus, was used. Time synchronization between the EC135-ACT/FHS helicopter and the camera was done by taking an image of the clock before take-off. This yields a time synchronization of about 1 second. This synchronization allows correlating the images with data recorded by the helicopter. The camera has a resolution of 6000x4000 pixels in photo mode and 1024x800 pixel in video mode. The camera was handheld and care was taken to avoid direct contact between the camera and the cabin because of vibrations. Video mode was attempted, but the relative motion between the helicopters was too large and too severe to obtain useful results. Furthermore, the video resolution proved insufficient to enable image processing techniques. Instead, the camera was used in burst mode where a series of images were recorded in a short time. The timing of individual images was rather random since it depends on the autofocus. In a second test campaign extra tufts were fitted to the stator blades inside the Fenestron duct and on the inner hub. Circular encoded markers were applied to the tail surface to enhance digital processing of the images, see left image in the figure above.
Data collection with a UAV in stratosphere
High Altitude Long Endurance (HAPS) platforms are the stratospheric unmanned aerial vehicles (UAVs) which are designed to fly in the stratosphere for a long period of time. As these HAPS platforms can operate above the clouds for a long period of time, their future applications, ranging from mobile communication networks to long term observations and environmental measurements, are envisioned. One of the challenges herein is obtaining detailed aerodynamic model of the UAVs flying in the stratosphere, which can be used for the flight controller synthesis and stratospheric mission simulation. System identification -- the process of determining the detailed aerodynamic model from the sensor measurements of the platforms -- is challenging due to a large flight envelope that covers altitudes up-to 25 km at varying Mach numbers. Computational Fluid Dynamics (CFD) can provide an alternative, but CFD require verification and validation with real-flight measurements. Application of tufts with in-flight images could therefore visualize a fluid flow for a HAPS platform, which may help in verifying and analyzing the fidelity of obtained aerodynamic model for real flight missions. Moreover, along with helicopter flights, having a method to work on the second test scenario can show applicability of the developed tuft recognition system to different in-flight scenarios.
For the data collection, the UAV HABLEG was used. The UAV is a small HAPS platform with a wingspan of 3 m and a mass of 7.4 kg. This platform is designed as a balloon launched experimental glider, where a balloon carries the platform to stratosphere, and the platform glides back to the ground station with its autopilot system. The main advantage of such test platform is the ability to test the flight components, wing designs including structural stability and airfoil choices, etc. without having to build up the costly HAPS platform. With tufts attached to the wing, the UAV fits the current use-case of developing automatic tuft recognition system. Figure shows overall data collection procedure, where the UAV was launched to the stratosphere with a balloon, and glided back to the ground station. When the balloon was detached, the UAV was on stall. Then, the flight controller was activated to glide in a stable manner. The platform reached stratosphere with maximum altitude of approximately 19.55km. Overall mission duration was 145 minutes with 169 km as the total distance traveled.
For image capturing, three GoPro Hero (2014) are used to capture rear and hinder of the UAV, as well as to monitor the full wing span of the UAV. In the current paper, we utilize the in-flight images that capture the full wing span of the UAV during the entire mission duration. An example image is depicted in the above figure with 19 tufts. Specific labels for each tuft are also provided in the figure, which are again marked with a letter/number combination in red. Locations of tufts were chosen to examine the local air flow around the wing during stratospheric flight. The obtained images has a resolution of 1920x1080 pixels. Unlike in the helicopter flight, several tufts are clearly visible in the relatively low resolution images, which is due to the proximity placing of the camera near tufts.
Measures of Tuft Segmentation
Implementation Details
References
[1] Olsman, W. F. J. "Experimental Investigation of Fenestron Noise." Journal of the American Helicopter Society (2022).
[2] Wlach, Sven, Marc Schwarzbach, and Maximilian Laiacker. "DLR HABLEG–High Altitude Balloon Launched Experimental Glider." 22nd ESA Symposium on European Rocket and Balloon Programmes and Related Research. No. SP-730. ESA Communications, 2015.