Milestone 3

Using Tableau for data exploration and cleaning is a common and recommended practice, especially when dealing with large or complex datasets. It enables user to:

1.     Visualize Data: Tableau provides powerful visualization capabilities, allowing us to create interactive charts, maps, and dashboards. Visual exploration often reveals insights that might be hard to discern from raw data alone.

2.     Data Cleaning and Transformation: Tableau allows us to clean messy data, and transform variables. In our case, filtering based on track IDs and identifying empty spaces in xCenters and yCenters are examples of valuable preprocessing steps.

3.     Iterative Analysis: User can iteratively explore the data, apply different filters, and visualize the impact of these filters in real-time. This iterative process can help in refining the criteria for identifying anomalies effectively.

4.     Data Subset Selection: By filtering data based on specific criteria in Tableau, user can create a focused subset of the data. This subset can then be exported and further analyzed using Python, as demonstrated in our provided Python program output.

DATA MODELING

1. Loading and Preparing Data:

·       The script starts by loading data from the 'intersection1_sampled_data.csv' file using pandas.

·       It filters the data to include only vehicles of the 'car' class and within a specific trackId range (from 100 to 350).

·       The features used for modeling are 'xCenter' and 'heading'.

2. DBSCAN Clustering:

·       The script uses the DBSCAN algorithm from scikit-learn to cluster the data points based on their spatial density.

·       DBSCAN groups together data points that are closely packed together and marks data points that are isolated (not part of any dense cluster) as anomalies.

3. Identifying Anomalies:

·       Anomalies are identified by finding data points with cluster labels of -1, which indicates they are not part of any dense cluster.

·       The script retrieves the track IDs of these anomalies.

4. Data Visualization:

·       The script visualizes the clustered tracks and anomalies using Matplotlib.

·       Normal tracks are plotted as scatter points, anomalies are marked with red 'X' markers, and a specific track (if provided as a command-line argument) is plotted as a black triangle.

5. Result Output:

·       The script prints the track IDs of isolated tracks detected as anomalies by DBSCAN.

·       It checks if the selected track (provided as a command-line argument) is an anomaly and outputs the result.

In summary, the script applies DBSCAN clustering to identify anomalies in our dataset, visually represents the clustered tracks and anomalies, and provides information about the selected track's anomaly status if provided as input. This approach is particularly useful for detecting isolated tracks or outliers that do not conform to the dense patterns observed in the majority of our data.

The choice between using DTW distances and DBSCAN for anomaly detection depends on the nature of our data and the specific requirements of our use case. Here are the key considerations for each approach:

Using DTW Distances:

·       Advantages:

·       Fine-Grained Comparisons: DTW allows for fine-grained comparisons between time series data, capturing similarities even in the presence of distortions, time shifts, or speed variations.

·       Domain Adaptability: DTW is versatile and applicable to various types of time series data, making it suitable for a wide range of applications.

·       Considerations:

·       Computational Intensity: Computing pairwise DTW distances for large datasets can be computationally intensive, especially if we have a large number of data points or long time series. Downsampling and careful consideration of feature selection can help mitigate this.

·       Threshold Selection: Choosing an appropriate threshold for anomaly detection can be challenging and may require experimenting with different values.

Using DBSCAN:

·       Advantages:

·       Efficiency: DBSCAN is relatively efficient, especially for large datasets, as it groups similar data points into clusters without the need for pairwise distance computations for all data points.

·       Automatic Cluster Detection: DBSCAN can automatically detect the number of clusters based on data density, which is beneficial if we are unsure about the number of clusters in our data.

·       Considerations:

·       Parameter Tuning: DBSCAN has hyperparameters (such as eps and min_samples) that need to be tuned carefully for optimal cluster detection. The choice of these parameters can significantly impact the results.

·       Sensitivity to Density: DBSCAN's performance is sensitive to the density of data points. If our data has varying densities, DBSCAN might not work well out of the box.

Recommendation:

·       If we require fine-grained anomaly detection and our data is not excessively large, using DTW distances with proper downsampling and feature selection could provide valuable insights into our data. However, we need to be prepared for the computational cost associated with pairwise comparisons.

·       If computational efficiency is a primary concern, especially with large datasets, and we are willing to experiment with parameter tuning, DBSCAN might be a more practical choice. It's particularly useful when our anomalies form low-density regions in our data.

Ultimately, the best approach often involves experimentation and validation. We might consider comparing the results of both methods on a subset of our data to see which one aligns better with our understanding of anomalies in our domain. Additionally, our domain knowledge and understanding the specific characteristics of our data can guide the selection of the most appropriate technique for our anomaly detection task.

Given our dataset consisting of tracks with multiple rows of xCenter and yCenter data points, we would recommend using DTW distances for anomaly detection. Here's why:

1.     Fine-Grained Comparisons: DTW allows for fine-grained comparisons between time series data, which is well-suited for comparing tracks with varying lengths and shapes. It can capture subtle similarities and anomalies in tracks, making it a valuable choice for our dataset.

2.     Track-Based Comparison: DTW enables us to compare entire tracks (sequences of xCenter and yCenter points) with each other. This is particularly important if we want to understand the overall shape and movement patterns of tracks, considering the entire trajectory rather than individual points.

3.     Versatility: DTW is versatile and applicable to various types of time series data, including trajectories. It accommodates different speeds, accelerations, and shapes in tracks, making it suitable for a wide range of tracking data scenarios.

4.     Domain Relevance: Given the nature of our dataset (tracking data with multiple rows of spatial coordinates), DTW aligns well with the domain relevance of our problem. It respects the temporal order of data points, which is crucial in trajectory analysis. While DTW computations can be computationally intensive, careful preprocessing (such as down sampling) and feature selection can mitigate these challenges. Additionally, the insights gained from the fine-grained analysis provided by DTW often outweigh the computational costs, especially when we are dealing with tracking data where capturing subtle differences is essential for accurate anomaly detection.

While DTW computations can be computationally intensive, careful preprocessing (such as down sampling) and feature selection can mitigate these challenges. Additionally, the insights gained from the fine-grained analysis provided by DTW often outweigh the computational costs, especially when you are dealing with tracking data where capturing subtle differences is essential for accurate anomaly detection.

Applying Research Findings to Improve Traffic Management: Recommendations

1.     Keep an Eye on Traffic in Real Time: Use technology to monitor traffic constantly. This helps in quickly noticing any unusual situations on the roads.

2.     Predict Traffic Issues: Use computers to predict future traffic problems based on past data. This way, we can prepare in advance to deal with these issues.

3.     Use Smart Signs and Alerts: Install electronic signs on roads that can change messages. These signs can warn drivers about traffic problems ahead. Also, connect these warnings to GPS and phone apps to suggest different routes.

4.     Smart Traffic Lights: Have traffic lights that can change their timing based on how much traffic there is. This helps traffic move smoothly and saves time and fuel.

5.     Educate People: Teach people about traffic rules and why they are important. Use social media and events to spread the word. When people understand the rules, they are more likely to follow them, making traffic flow better.

6.     Work with Tech Companies: Partner with companies that make smart transportation technology. Test new ideas like self-driving cars and smart parking. These innovations can greatly improve how traffic works.

7.     Keep Learning and Improving: Have a team that keeps studying traffic data. By understanding patterns, we can make our solutions even better. Regular research helps us stay ahead of traffic challenges.

8.     Government Support: Create laws that support smart transportation. Encourage people to use eco-friendly vehicles like electric cars or bicycles. These laws can speed up the shift toward better and greener transportation.