Introduction to the Uncore CyberAI program
Attended a presentation on various careers in research by Professor Uthurusamy
Presentation on research procedures by Professor Hana
Learned how to sharpen skills in reading and writing research papers (Professor Hana)
Introduction to Responsible Conduct of Research (RCR) training by Professor Siadat
Project descriptions overview:
Autonomous Security Patrol (Professor Hana)
AI in Short-Term Trading (Professor Siadat)
Adversarial Attacks and Defenses for Panoptic Perception Models in Autonomous Driving (Professor Xu)
Raising User Awareness of AI Profiling (Professor Wilson)
Software Engineering and Deep Learning (Professor Wardat)
Group/topic assignments completed
Began setting up the simulation environment
Read research papers to gather more context on our topic
Watched educational videos on machine learning, convolutional neural networks (CNN), and object detection
Progressed through the RCR training course
Continued working on cybersecurity modules
Started working on the deliverable due on Tuesday 5/28
Continued researching and taking notes on related papers
Further progress on cybersecurity modules
Presentation on collaboration by Professor Debnath
Reviewed weekly deliverables and addressed questions from readings in group meeting.
Polished presentation for Tuesday.
Continued cybersecurity modules.
Presented weekly report.
Configured Google Colab for adversarial driving simulation.
Continued research on adversarial attacks through literature.
Collaborated with Harry to finish simulation environment setup in Google Colab.
Consulted with Dr. Xu for further assistance with setup.
Continued setting up the adversarial driving simulation environment using PyTorch.
Started setting up the environment for the YOLOP model.
Read more research papers on adversarial attacks and their defenses.
Continued working on website.
Finished reading literature provided by Dr. Xu.
Attempted performing training necessary for outdated adversarial driving simulator.
Started compiling report for next week Monday's group meeting.
Made progress with the adversarial attack portion of the simulator.
Still needed the actual driving simulation provided by Udacity. It was outdated, so time was spent trying to work around the broken or missing dependencies.
Gave a presentation on our activities from last week.
Spent the day playing with the YOLOP model; took a long time because the repository we were working with was broken.
Downloaded the necessary bdd100k dataset. The process took about 4 hours because of the size.
Performed YOLOP Segmentation Label (Mask) Generation, but the loading process took around 3 hours.
We faced a challenge as we only had one device with a GPU, and it was being actively used to set up the adversarial driving simulator.
Had a virtual meeting with Dr. Xu to update her on YOLOP progress
Discussed next steps with Dr. Xu
Fixed issues in test.py, demo.py, and postprocess.py files
Issues were causing problems with evaluating inference results and data conversion
Watched tutorials on YouTube for performing adversarial attacks on previous YOLO models using PyTorch
Started first attempt with Fast Gradient Sign Method (FGSM) adversarial attack.
Realized that despite there being 10 classes, only cars and roads were being detected.
More specifically, vehicles seemed to be getting thrown into the same category.
This caused some confusion about how FGSM may play out, as FGSM is mainly a misclassification attack.
Was able to implement FGSM on the demo.py file for inferences.
Implementing FGSM in test.py presented some challenges that required further deconstruction of the code.
Picked up on troubleshooting issue with FGSM integration into test.py.
Realized that instead of misclassification post FGSM, we can expect to see:
False positives
Incorrectly placed bounding boxes
Incorrect drivable area segmentation
Incorrect lane segmentation
Momentarily paused on YOLOP work to read documentation for CARLA driving simulator.
Continued watching YouTube tutorials.
Delivered weekly report presentation.
Worked on integrating FGSM into test.py. This file is crucial for obtaining valuable metric data for analysis.
Encountered a bug preventing the evaluation script from running.
Spent considerable time debugging the code.
Attempted to resolve indexing errors.
Met with Dr. Xu to receive help with debugging the code.
Discovered the issue was a coding oversight: a variable was used in the wrong place, causing the image batch shape to no longer match.
Fixed the code, but encountered memory leak problems, necessitating a reduction in the batch size.
Finally got the code to work and observed changes in accuracies for segmentation tasks but not for object detection.
Realized that the object detection metrics weren't changing because the Non-Maximum Suppression (NMS) portion of the code that handles bounding boxes needed separate tweaking.
After correcting all the variable mismatches, I observed that the loss was finally responding correctly to perturbations, and the object detection results were now updating.
Began research on JSMA, skimming online resources to gain a general understanding of it.
To implement the JSMA function, I needed to understand what a Jacobian is and what saliency maps are.
Spent some time taking notes to comprehend JSMA and its previous implementations.
Suspected that the iterative nature of saliency map creation might require significant computational time and memory.
Met with Dr. Xu to discuss this concern and potential next steps. She proposed localizing JSMA to a random portion of the image.
Began coding the JSMA implementation.
Initially attempted to extract a portion of the image, apply the perturbation, and then reattach the perturbed chunk.
Encountered numerous syntax errors that hindered my approach. When I extracted the image, the new chunk was treated as a numeric tuple pair, making it unusable for further operations.
Scrapped my previous JSMA attempt and decided to extract the chunk, save it as a separate image to work with, perturb it, and then reattach it. This approach was computationally complex and required extensive pre- and post-processing of the image.
When I attempted to pass it through the validation function, I encountered errors about incompatible shapes. (sigh)
Decided to abandon this method as well and try again. Realized that by extracting chunks or introducing a mask to isolate the portion of the image to be perturbed, I would also need to adjust other information such as target values and shapes.
Decided to scrap the chunk idea altogether to save time.
Began working on JSMA helper functions that the input images would be passed through, such as calculating saliency scores and perturbing the highest-scoring pixels.
To reduce computational strain, we introduced limits such as a "number of pixels to perturb" argument.
This approach finally yielded success, and I observed the JSMA helper functions impacting the model's performance, albeit not significantly.
Met with Dr. Xu to discuss and review my code. I explained the difficulties I was encountering.
She suggested introducing different methods of attacks, as I was merely adding noise to the pixels with the highest scores. I introduced methods such as set, noise, and add.
Paused the implementation of attacks to review the data being produced.
Attempted generating graphs for the data and had suspicions about what was happening post-FGSM and JSMA.
Noticed that the function.py file had become unkept and disorganized. Spent time reorganizing the code and adapting it for the attacks and defenses.
Modified the logger and updated code in the core file that handled result displaying.
Added helper functions to record data more efficiently.
Took a break from code implementation to check in on the CARLA simulator setup and YOLO-P integration that Harry was handling.
Began piecing together the midterm presentation.
Spent some time tweaking the FGSM functions to run multiple experiments simultaneously to eventually create a graph. Due to constraints with data collection, we opted to create a table first.
Applied the same adjustments to JSMA.
Continued working on the midterm presentation.
Noticed the results were looking even more suspicious, so I set up a meeting with Dr. Xu to discuss whether the results we were getting were typical or not.
Gave the midterm presentation and shared findings and progress up to this point.
Continued working on the progress for JSMA.
Set up a GitHub repository.
Obtained remote access.
Due to difficulties with the simulator, we decided to switch to Color Channel Perturbation (CCP) instead of Split Second Phantom Attacks (SSPA).
Began researching existing implementations of Color Channel Perturbation (CCP) and Universal Adversarial Perturbations (UAP).
Started writing the UAP attack implementation.
Found an existing implementation that uses stochastic gradient descent.
Paused the UAP implementation to visualize the results of JSMA. Began working on a script to create graphs for each metric perturbed by an attack.
Tweaked test.py to be more interactive with arg-parsers that accommodated the various experiment modes, parameters, and attack types.
Attempted testing existing attacks on the CARLA image set generated by Harry.
Ran into difficulties with missing annotations (labels, masks).
Attempted to find existing annotations online, but none fit BDD100k's layout.
For the sake of time and to stay within the scope of the research, I returned to attack implementation and paused the CARLA testing, since we still have real-world images from BDD100K.
Nearly finished the UAP implementation; took a break to start working on CCP.
Finished both CCP and UAP implementations.
Attempted collecting data for both types.
Refactored the code to accommodate the various changes: cleaned comments, reorganized functions, and structured the code for better readability.
Started working on the presentation for next week.
Tested Harry's pre-processing defenses on the attacks.
Encountered issues due to the nature of the defenses, which required additional refactoring.
Delivered the weekly presentation.
Found a workaround for the initial processing issue.
Attempted testing the pre-processing defenses on a small, manually created dataset but encountered some challenges.
Wrote scripts to automate dataset creation.
Goals:
Observe the effectiveness of the defenses.
Evaluate how varying defense parameters perform against different attack parameters.
Visualize the defense performance data.
Met with Dr. Xu to discuss progress so far and potential next steps.
Spent the day collecting data from previous attacks for analysis.
Attempted to fix the code to visualize the results using the original image instead of the perturbed or perturbed/defended images.
Independence Day!
Started creating the presentation for next week.
Continued debugging issues with scripts and YOLO-P code for better data analysis.
Gave the weekly report.
Cloned the repository and attempted data collection using the lab machine, but it was unsuccessful.
Spent the day debugging an issue that prevented the code from compiling.
Realized that YOLOP won't function with certain defense types (resize defense specifically)
Finally fixed the issue with visualizing the results using the original image instead of the altered images.
Attempted data collection for pre-processing (again).
Incorporated certain features to efficiently collect data:
Skip if performing low.
Skip combinations that have already been seen.
Prioritize certain combinations that make sense.
Real-time logging.
Collected all data and started drafting the poster.
Ran into size issues due to project size and had to do some reorganization.
Finished the poster, but needed to generate graphs.
Generate the necessary graphs for the poster.
Review and finalize the poster draft for Monday
Worked on deliverables.
Added missing items to the website and Google Drive.
Added better code documentation.
Continued modifying the poster
Worked on deliverables
Worked on deliverables
Worked on deliverables
Worked on deliverables
Worked on deliverables
Worked on deliverables
Big presentation :)
Finalized Deliverables
Exit day