In the visual inertial sensor fusion problem, the visual measurements are the ones provided by a monocular camera, the inertial measurements the ones provided by a triaxial accelerometer and a triaxial gyroscope. From now on, we refer to this sensor setting as to the standard visual inertial sensor setting. By applying basic results on nonlinear observability, both introduced during the seventies by the automatic control community and also new concept developed by myself (e.g., the concept of continuous symmetry introduced here), I obtained all the observability properties of this sensor fusion problem. The results are published here. The main result is the following: In the standard visual inertial sensor setting all the independent observable modes are: the positions in the local frame of all the observed features, the three components of the speed in the local frame, the biases affecting the inertial measurements, the roll and the pitch angle, the magnitude of the gravity and the transformation between the camera and IMU frames. This holds independently of the number of observed point features.
Pushed by curiosity, I investigated other sensor setting that are less informative than the standard sensor setting of above. Specifically, I considered the case when they do not include a complete triaxial accelerometer since the acceleration is perceived only along two axes or even along a single axis. This curiosity was driven by the fact that in the vestibular system, the sensors that perceive acceleration, are the otoliths, i.e., the saccule and the utricle. Both of them perceive the acceleration only on a plane. The question that I wanted to answer was: Is it possible to explain, by an observability analysis, that in these conditions there is enough information to have self-motion perception? Note that, to properly use the information provided by an otolith, it is necessary to know the relative configuration of the visual sensor with respect to the otolith (in computer vision this is called camera extrinsic calibration that is a part of the camera calibration process). Hence, the system to be analyzed is the one that consists of a camera and a single otolith, where the relative transformation is unknown. The questions to answer are:
Are observable the extrinsic camera parameters for this setting?
How the observability properties change with respect to the ones of the standard setting?
What is it observable if the accelerometer only perceive acceleration along a single axis?
Provide a complete answer to these questions demands to solve a fundamental open problem in control theory, which is the unknown input observability problem. Indeed, in the observability analysis, we do not want to constraint the motion such that the acceleration vanishes along the axes where it cannot be perceived. In contrast, the motion is not constrained and, the acceleration along the axes where it cannot be perceived, acts on the dynamics as an unknown input. For this reason, I spent five years to derive the analytic solution of this open problem (see here the solution).
By applying this solution I finally could answer all the previous questions. All the results can be found in the last application discussed in chapter 5.7 of my book. A short summary of these results is:
The information provided by the standard sensor setting is equivalent to the information provided by a monocular camera and only 2 accelerometers. This means that an otolith, which is able to sense the acceleration on a plane, provides the same information of a complete Inertial Measurement Unit (IMU), once its measurements are fused with them of a monocular vision sensor.
In the case when the inertial sensors only consist of a single axis accelerometer, it is still possible to get the absolute scale but the self-perception of motion is altered. For instance, the perception of the speed is incomplete since it is not possible to distinguish the speed from the same speed rotated around the accelerometer axis. This degree of freedom will be removed by adding a further accelerometer (and this is consistent with the first result of above) or by adding a further single axis gyroscope, provided that the accelerometer axis does not coincide with the gyroscope axis.