DeepSoro Proprioception Research

Intro

In hazardous environments, navigating unpredictable obstacles and unstable terrain poses risks to conventional robots. A soft robotic sphere, with its adaptable and deformable design, offers a safer solution, minimizing damage to itself and its surroundings. Its ability to roll, compress, and maneuver through complex areas allows it to detect paths and construct real-time maps, making it invaluable for exploration, search-and-rescue, and disaster response.

I jointed AI4CE to work under PhD Zhao Hanwen who is in charge of building a self-propelled soft spherical. Hanwen planned to use two internal camera to observe the point clouds drew to the shell of the soft robot, therefore calculating its movement through neural networks. He sets up the environment for simulation of robot's interaction with its surrounding. I was responsible to have a second environment set up on my computer that will be later used to run more models or variation in setup. Here is a demo of PolyFem solved scene where the robot is drop from the sky to a terrain:

Vertices Mismatch Issue

The simulation information is stored in three kinds of files, vtu, stl, and vtm. Despite they have different format, they all store geometric information using corresponding vertices and face. Because the solver we use, PolyFem and Pymesh, have multiple version, compatibility issue occur when wrong version are used.

For example, one error occur during simulation:

It appears that the shape of the ball is completely ruined, which suggest wrong matching between vertices and faces, as it's no longer a smooth surface but with ridges.  The third image above shows a camera capturing inside the sphere. The interior is blacked out by intersected faces, and sphere's shape under this condition cannot be distinguished.

After some time, I pinpointed the issue in the source code:

This issue occure when the current Pymesh no longer support a function called CleanPolyData that used to clean up redundant face or vertices. Therefore, I turn to use the clean function in vtk, and this function incorrectly arrange geometric positions.

After numerous try, I finally figured out how to resolve this issue no matter what version user have for their packages. Below is a powerpoint shwoing my approach in details. With this method, the simulation will not be dependent on the package version no more.

After the correction using pymeshlab's multiple filters, the sphere restore it's shape, and we can place cameras inside the hollow sphere.

We now can monitor the internal deformation by using the camera functionality in Blender app. The picture on the left shows the point cloud that simulate the distortion, and the picture on the right shows what the Blender camera capture. We can see that the shpere is being squeezed from the right indicated by the shading for the right picture.

ResNetLSTM with Hallucination

We need a powerful model that can observe many features captured by the internal camera, while using those features to deduce deformation and translation of the robot. Zhao Hanwen builds a model called ResNetLSTM, which apply pretrained Resnet18 for feature extraction, while feeding into a LSTM for time series prediction.  Here is the total structure of the model (I used silly block diagram as the paper has not been published, this model belongs to Zhao Hanwen):

Here is a demo of using hallucination vs without, since the experiment is still in progress, I cannot show the actual robot simulation, here is the effect of hallucination in the Moving MNIST dataset as shown. 

Because this is a fairly simple dataset, I keep the epoch to a small number in order to see the difference in performance for hallucination feature vs without. Here is the result:

Here are more updates:

Real World Experiment

The simulation is very successful as the translation prediction reaches around 90 percent of accuracy. However, we have to test it out in the real world.

The real world spherical robot should be self-propelled and deformable in every single place. This is hard because we have to put 3D printings or other inductile things inside a sphere. 

The single camera model, spherical robot, and real world experiment is coming...