Government Datasets -> Color-coded Sidewalk Segments

We take GIS Street and LIDAR data, plus a limited curb ramp dataset including only state-owned roads, and transform it to populate the database with sidewalk segments and elevation information. We used the Census Bureau’s roads shapefile to go from the full set of roads to a set of likely sidewalk segments, removing highways, for example. We used Wisconsin’s LiDAR dataset containing elevation contours to determine the maximum steepness for each sidewalk segment, which determines their color on the map. We did these calculations in ArcGIS and had to combine data from city, state, and federal databases to get our final output – no single website had all the data we needed.

Street View Images -> Curb Ramp Labels at Crossings

We pass crops of Google Street View images through a deep learning model to predict missing vs present curb ramps. To begin, we sampled full panoramas from about 20,000 intersections in the city of Milwaukee, resulting in 80,000 images from the 4 corners of each intersection. We then labeled a small sub-set of 2500 images in order to train the model that would predict the presence or absence of curb ramps on the remaining panoramas. We manually annotated these 2500 panoramas using the VGG Image Annotator, to following the Project Sidewalk labeling guidelines.

User Reports -> Timely Insights

The "Network Effect" of our Sidewaukee website. The users of our application can submit reports about temporary obstacles and other surface problems that our map is missing. Our custom API updates our database and displays it when a user hovers over the reported corners on the map. We prioritized this feature after hearing from our users that temporary obstacles like snow, ice, and construction are critical to their experiences navigating the city and that information typically isn’t available in other applications. We were able to implement this feature using MapBox and React.

Our plan was to use the pre-trained Project Sidewalk ResNet-18 model and apply transfer learning to the labeled Milwaukee image data and then classify the remaining images. From our 2500 manually labeled panoramas, we found meaningful sidewalk features to be present only in the middle one third of the image. This band captured 95% of the labels. So, we made six crops out of the middle band of every image to pass into the model. Unfortunately the pre-trained ResNet-18 model did not yield expected results.

We switched from the pre-trained Project Sidewalk ResNet-18 model to a generic ResNet-34 model trained on ImageNet and we started seeing better results. We suspect that the Project Sidewalk model was overfit to their particular set of images and thus incorporating a larger model performed better than the domain-specific pre-trained model.

We also improved our model's performance by switching from Project Sidewalk’s 5-class problem to a tiered binary classification. We first predict if an image is of a street crossing, then predict whether it has a missing or present curb ramp. We knew from our users that curb ramps are the most important feature, so we didn’t mind sacrificing the ability to predict other less-common classes like “surface irregularity” to improve the performance at the curb ramp prediction task. We achieved a 76% recall on predicting a curb ramp that is present.