Ashwin's Research

Introduction to project topic and research question:

I plan on researching how camera image data can be used to orient the Pioneer 3 Dx robot frame with respect to the cornhole tables so that we can improve the accuracy of the bean bag launches. The evaluation criteria will be how close the final orientation of the robot frame is to the cornhole table holes.

Discussion of related work:

1. Ji Zhang et al. used cameras and lidar to compare the motion estimation accuracy. Sensor suite robustness when moving at high speed and when subject to significant ambient lighting changes, were also evaluated. In this paper, the authors used the V-LOAM algorithm upon data from a depth camera data at 60Hz, to estimate the location of dynamic objects in the environment. Upon this, Lidar data at a much lower, 1Hz was used to refine this location estimate. This differs from my approach which uses Hough Transforms upon images provided by an ordinary 30Hz RGB camera. One useful lesson from this paper was their conclusion that fisheye camera lenses provided greater robustness at the cost of robustness when compared to wide-angle lenses. This analysis helped us identify the lens of the raspberry pi camera to buy.

2. Y. Ebisawa used an infrared light source that was irradiated to an eye to help track the pupil movement. Two feature areas (the corneal reflection light and pupil) were detected in the image obtained from a video camera and then the eye-gaze direction was determined by the relative positions between the two. This method was different from my approach in the sense that the camera was very close to the object being detected (the pupil) and it used two light sources that helped produce two images, the differences of which helped identify the pupil. One useful lesson learnt from this paper is the effects of shadows and lighting conditions on pupil detection.

3. Danti et al. uses Hough Transformation method for Lane detection, where as Color Segmentation and Shape Modeling with Thin Spline Transformation (TPS) is used with nearest neighbor classifier for road sign detection and Classification. Further, K-means clustering based algorithm is adopted for pothole detection. This is in contrast to our paper which utilizes Hough transform to detect holes in the cornhole table. This paper offered us additional context on how color segmentation could be used as an additional feature to improve hole detection.

4. Moghadam et al. used a stereo vision camera system to dynamically plan and navigate a mobile robot in cluttered and complex environments. A robust estimator is used to detect obstacles and ground plane in 3D world model in front of the robot, based on disparity information from the stereo vision system. Based on this 3D world model, a 2D cost map is generated for navigation, and localization can be achieved with respect to objects in the 3D world model. This method uses a stereo camera system upon which occupancy grid mapping techniques are used for localization. However, in our approach, we are using a single camera system employing Hough transforms. This paper gave us additional insight on how 2D lidar data may be fused with the stereo vision data to improve the accuracy of the occupancy grid generated. Thus, this technique could potentially be used to improve the performance of the approach I used as well.

5. Kammel et al. uses a 2D lidar and camera images to detect lane markers for driver assistance systems as well as for autonomous vehicles. The system allows the robust estimation of deviations between a digital map and the real world. This paper introduces the idea of using Hough transform to detect lines of parking spots and thus identify the center of the parking spot. However, in our approach we use Hough transforms to directly detect the cornhole table holes. This paper has given me the idea of using Hough transform to also map the edges of the cornhole table - this is something that can be used to filter out holes detected, such that only ones within the box made by the detected edges, are accepted (in case we detect more than one hole).

Methods used to investigate the research question:

I used a raspberry pi camera mounted on the upper surface of the robot to capture images of the cornhole table. The cornhole table images were then processed using the raspberry pi 3 located at the back of the robot. The study was conducted in Rogers 330 as there were cornhole tables lying around.

As a part of the data collection process, I placed the cornhole table on one end and the robot away from it at a distance of 3 meters (distance to launch in competition). Next, I tried manually reorienting the robot at different angles with respect to the cornhole table and examined how well Hough transforms could be used to detect the ellipse outline of the cornhole table hole. Sample images of the cornhole table with detected holes is shown below:

Once the ellipses were detected, using trigonometry and the properties of the camera's field of view, I was able to calculate the offset orientation. Using this offset, the pioneer was reoriented to face the cornhole table hole. The accuracy of the final orientation of the robot was the evaluation metric used to judge the effectiveness of the approach. This angle difference was measured using a string and protractor for simplicity. All data collected was stored in an excel file for processing at a later stage.

Figures summarizing results and description of all results gathered :

The results obtained are similar to what was expected when I developed the previous report (getting deviations in the order of +/- 5 degrees). Due to small errors in the image processing, camera alignment, measurement error and the inability to get the pioneer to orient itself as specified, we will get the observed deviations in the final orientation. Also, the results mostly represent a normal distribution, as would be expected for this particular application. An image of the results obtained from the 30 trials conducted can be seen below:

For the actual competition, I was not able to implement this approach on the robot due to the limited time available. During the first round, our robots loading mechanism broke due to a jammed load - Servos gears had broken . In the second round, we lost, mostly because we forgot to turn on the pioneer during the transition phase and we could only launch one bean bag as we decided to go ahead without our loading mechanism.

Conclusions and future work:

The results obtained indicate fairly accurate orientation with respect to the cornhole table holes. Due to small errors in the image processing, camera alignment, measurement error and the inability to get the pioneer to orient itself as specified, we will get the observed deviations in the final orientation. Also, the results mostly represent a normal distribution, as would be expected for this particular application. Overall, we can conclude that robot cornhole table hole alignment using Hough transforms is a feasible strategy for alignment with the hole.

My results and similar experiments on localization can help me with my active research area. I am currently working on assistive robots for improving the mobility of disabled infants. Having the ability for the robot carrying the infant to self-localize in the environment is an important step towards assisted driving. This, in addition to helping the infants navigate their environment in a safer way while avoiding unwanted collisions, can be used to redirect them towards other groups of infants, thereby increasing the amount of social interaction for them. Currently we are researching on using cameras, eye trackers, 2D Lidar, ultrasonic sensors, Kinect 360 sensors etc., to facilitate localization. This research has provided me a greater understanding on how cameras could be used for this purpose. Through the other research papers, I've reviewed, I also got to understand the potential of stereo vision cameras and 2D lidar. Thus, this will make part selection much easier for my personal research and help me formulate a better strategy faster.

References:

1. Zhang, Ji, and Sanjiv Singh. "Visual-lidar odometry and mapping: Low-drift, robust, and fast." 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2015.

2. Y. Ebisawa, "Improved video-based eye-gaze detection method," in IEEE Transactions on Instrumentation and Measurement, vol. 47, no. 4, pp. 948-955, Aug. 1998.

3. Danti, Ajit, Jyoti Y. Kulkarni, and P. S. Hiremath. "An image processing approach to detect lanes, pot holes and recognize road signs in Indian roads." International Journal of Modeling and Optimization 2.6 (2012): 658.

4. Moghadam, Peyman, Wijerupage Sardha Wijesoma, and Dong Jun Feng. "Improving path planning and mapping based on stereo vision and lidar." 2008 10th International Conference on Control, Automation, Robotics and Vision. IEEE, 2008.

5. Kammel, Soren, and Benjamin Pitzer. "Lidar-based lane marker detection and mapping." 2008 IEEE Intelligent Vehicles Symposium. IEEE, 2008.