16-824 Group 18 Project Website

Future Directions

The first step to take in terms of future directions is to accommodate for the issues we encountered during our development and training of our model, specifically the limitations in our dataset, our masks, and the latter inpainting architecture.

With regards to our dataset limitations, some approaches we could take are to collect our own data in the field, augment our current dataset through translations and cropping, or otherwise artificially apply raindrop distortions and blurs to images of scenes. This would allow us to increase the size of our dataset and thus provide our models with more examples to learn from.
In order to improve the quality of our raindrop masks, we would consider approaching the problem through less traditional image transformations (i.e. morphology operations, filters, and Canny edge detector) and instead take the time to label masks from the rainy image. This would allow for the training of convolutional neural networks through supervised learning to learn how to automatically generate the raindrop masks through semantic segmentation of the image into raindrop and non raindrop pixels.
We also consider changing the inpainting architecture into a conditional GAN. Due to the fact that we completely remove portions of the image in the process of masking out the raindrops, oftentimes the GAN does not have enough information in order to properly reconstruct a realistic representation of the derained scene. Implementing a conditional GAN would allow the model to retain a correspondence between the input and output and thus learn to derain scenes more realistically.

In terms of future developments past the corrections, our model is currently only trained to handle raindrops of varying sizes on windows and camera lenses, creating distinct regions of distortion to detect and replace. However, there are a multitude of other rain scenes that may be encountered outdoors. For example, there could be heavy rain creating large rain streaks across the image, the rain could be striking different surfaces and creating rain splatters in the image, or the rain could creating a mist and fog up the scene. Further developments in our model would seek to take into account these varied rain scenes in order to make our model applicable to any type of rain that it may encounter, thus resulting in an all purpose deraining model.

Fig. 13: Here, heavy rain is falling and creating thick streaks across the image. Depending on the speed and density of the rain, the image may be obscured to different levels or even view rain splatters on objects in the scene. [b]

Fig. 14: In this image, the rain causes a fog on the windshield to form. However, it is often common to observe rain creating a mist that fogs the area without directly affecting the window or camera surface. [c]

Conclusion

Overall, though our method demonstrated a clear ability to reconstruct a derained image from rainy ones, it does not perform as well in comparison to other approaches such as Attentive GAN, Eigen13, and Pix2Pix. However, through our analysis of the results, we have identified clear reasons why we observe this behavior and solutions to them as well. Despite these issues, our model's approach towards deraining images still shows definite potential to perform better given the proper adjustments to the dataset and tuning to the architecture. The model at a base state without retraining performed better in both PSNR and SSIM metrics in comparison to the scores from the rainy input and clear ground truth images. Additionally, when retraining the model with all of these issues in consideration, the model showed a significant improvement in its performance. Thus it is reasonable to expect that once the issues we experienced are resolved through the methods detailed in our future directions, our model will likewise see improvements in its final performance.