Week One (5/20 - 5/24)
Week One Review:
Introductions
Developing a Research Career Presentation (Professor Uthurusamy)
Title IX Training Presentation (Chad)
Introduction to the UnCoRe Program (Professor Fu)
Reading and Writing Research Papers (Professor Siadat)
RCR Training (Professor Siadat)
Project Descriptions by REU advisors
Teams Formed, First Meeting with Advisor (Dr. Xu)
Team Work Presentation (Professor Debnath)
Fundamentals Modules
Start of Website
First Presentation Created
Week Two (5/27 - 5/31)
Week Two Review:
5/28
Week Two Presentation
Continue Working on Website
Setup CoLab Online Coding Platform for Testing
Attempt to Start Setting up Wuhan Studio Adversarial Driving Simulator
5/29
Continued work on Google CoLab (debugging, fixed missing dependencies, found more issues)
Continued Literature Review: Reviewed Adversarial Attack and Defense of YOLO Detectors... Phantom of the ADAS...
Started Creation of Week Three Presentation
5/30
Abandoned Google CoLab Project Due to Technical Limitations
Pursued Creation of Simulation Environment Within the Ubuntu Server we will Use for the Remainder of the Project
Finalized Initial Literature Review: Reviewed A Survey on Physical Adversarial Attacks... Panoptic Segmentation...
Continued Work on the Website
Created a Document Containing a List of All Adversarial Attacks & Defenses Found Within the Project Literature
5/31
Heavily Enhanced Website
Continued Working on the Week Three Presentation
Worked on Initial Adversarial Driving Model Training and Installation & Optimized File Pathings
Added All Relevant GitHub Repositories to the Site
Added All Relevant Sources & Master List of Attacks & Defenses to the Site
Week Three (6/3 - 6/7)
Week Three Review:
6/3
Week Three Presentation
Continued work on setting up the Adversarial Driving Simulator
Successfully installed the remaining parts of the model and data for training the simulator
Decided upon which attacks & defenses to use for the project
Registered for MIDSURE
6/4
A LOT of work on installing the UDACITY Autonomous Vehicle Driving Simulator
Further Model Training
Verification of simulator assets and features completed
Unity Hub Installed (v 3.8.0)
Unity Editor Installed (2020.3.of1)
6/5
Continued work on UDACTIY Autonomous vehicle simulator and the adversarial driving simulator remotely
Had to take care of getting a new military ID card and Driver's License
6/6
Rewrote a good amount of the code for the driving simulator, only to find that it still did not work
Attempted to solve a myriad of dependency issues and version mismatches
Uninstalled and reinstalled almost all of the programs relating to the simulator.
6/7
Due to a large amount of issues with the Adversarial Driving Simulator and UDACITY, I researched to find a new simulator
Discovered CARLA driving simulator. Read documentation and began installation process.
Installed a pre-packaged version of CARLA to check its functions
Researched methods for how to install a YOLO model in CARLA
Began work on week four presentation
Week Four (6/10-6/14)
Week Four Review:
6/10
Week four presentation
Uninstalled the pre-packaged CARLA
Re-compiled CARLA from source
Re-compiled Unreal Engine 4 from source
Ran into MANY errors, most of which were caused by the PCs resources being entirely used.
Researched how to fix errors with Carla to compile without error properly
6/11
Realized Linux has a feature called swap which can act as virtual RAM in cases where you do not have enough RAM to run certain tasks.
Analyzed swap settings
Changed swap partition size from its initial size of 1GB to a size of 8gb
Further compilation errors
increased swap size again, start recompile, (compilation takes roughly 2-3 hours every time)
6/12
Researched methods to integrate YOLOP within CARLA, and found a single example on github
Attempted first stages of retooling CARLA dataset generator scripts from https://github.com/SekiroRong/YOLOP
Researched various CARLA settings, features, and tools
Started testing using map 01 for CARLA, quickly switched to map o5 due to its lesser size and smaller processing overheads
Began rebuilding CARLA dataset generator and fixing broken code.
6/13
Continued work on Carla Dataset Generator
Broken code is almost completely fixed, though the car still is not being controlled properly
Further research into CARLA
6/14
FINALLY rewrote the dataet generator script, which now functions as expected
Current problems in the script
Bounding Boxes are not being generated properly
Raw Data Conversion script does not function, so we cannot translate our frame by frame images into a video at present
We will then need to ensure we can integrate YOLOP with CARLA
Begin work on week 5 presentation
Week Five (6/17-6/21)
6/17
Week Five Presentation
Took note of a few suggestions from peers
Began YOLOP-CARLA integration
6/18
Since YOLOP's source code is not used to control a simulated vehicle, and the SekiroRong YOLOP-CARLA code was a bit outdated and broken, I attempted to find a NEW solution to this issue: building a completely new script to accomplish our tasks from scratch
This proved to be incredibly difficult, as someone who does not specialize in programming.
researched the way vehicles are controlled in CARLA
Researched implementations of other YOLO models in CARLA
6/19
Continued work on my CARLA-YOLOP.py script
Was met with a great deal of failure
Nothing seemed to function correctly, and I was having issues with both the code itself and the physical machine we were using
6/20
Worked closely with our advisor to find new solutions to the issues we experienced
Began implementing fixes, was met with some success
Still could not implement vehicle control based on YOLOP predictions
Spent the rest of the day attempting to fix broken code
6/21
Met with project advisor and mentor once more
Collectively decided to move on from the YOLOP-CARLA integration, as it proved to be beyond the scope of our project
Began work on midterm presentation and a new plan for YOLOP
Week Six (6/24-6/28)
6/24
Week 6 (MIDTERM) Presentation
Properly discussed new plan with team
Decided to generate high-quality images using our functional Carla Dataset generator, and pass them through YOLOP
Began Collection of CARLA simulator images
6/25
Collected over 300 CARLA images, and passed them through YOLOP
YOLOP results were... bad
Accuracy was incredibly low, with a very high loss
Attempted to research how to modify textures in CARLA
Began modifying textures for the road, lane lines, and crosswalks to be simple, bright colors without intricate patterns (such as those simulating a real road)
With the lane lines set to be bright white, and the roads a dark black, I began generating a new set of images from CARLA
6/26
Returned to the lab and examined each of the collected images, then passed them through YOLOP
YOLOP results were better, but still not as good as we needed
Attempted to create a "CARLA Dataset"
CARLA Dataset creation proved to be incredibly time consuming, and we did not have masks to train YOLOP on, nor did we have the time to re-train YOLOP once more.
6/27
Shifted focus to the defense aspect of our project, limited scope to cover just the attacks and defenses, cut out the simulator entirely
Began researching a pre-processing defense for YOLO models
Identified potential weaknesses of the defense
Began drafting a script to run a number of pre-processing tasks, based upon user arguments.
Simultaneously began training YOLOP adversarially
6/28
Began refining the script, establishing the full list of desired functionalities, and limiting scope to those tasks
Identified a selection of pre-processing methods to test with YOLOP:
Image Resizing
JPEG Compression
Bit-Depth Reduction
Gaussian Blurring / Noise Reduction
Noise Generation / Noise Addition
Implemented all of the mentioned pre-processing methods as their own individual functions within the pre-processor script
Began work on Week 7 Presentation
Week Seven (7/1 , 7/2 , 7/3 , 7/5)
7/1
Week 7 presentation
Created 500 adversarially perturbed images from BDD100k
Began fine-tuning YOLOP with this imageset
Began making changes to the YOLOP training scripts to enhance adversarial training
Started training (this takes time)
7/2
Returned to the machine to check training progress early in the morning, the computer had crashed and not finished training
Restarted training due to the previously mentioned issue
Since the machines resources were being almost entirely used by the training process, I began work on another plan
Worked diligently to properly install Ubuntu on my machine
Ran into a number of issues related to version mismatches and incompatibilities
Continued to identify and fix issues
Began Researching a third defense, the DEFENSE GAN
7/3
Further research into a myriad of image-to-image GANs
Identified a short list of options, but finally decided to use Pix2PixHD from Nvidia to create the GAN
Researched how to use pix2pixhd, and broke down their scripts into chunks to teach myself the code.
After I had a good grip on the process, I began testing and training the GAN using our previously generated 500 FGSM training examples. We took these 500 examples, matched them with their non-perturbed "real image" counterparts, and fed these image pairs into the GAN for training
Ran into a LOT of issues while training, GPU seems to be having issues. PC crashed
Research methods to fix hardware and software issues for my PC
Began the incredibly long and convoluted process of finding a specific Ubuntu version that worked with the incredibly specific versions of CUDA, Tensorflow, CUDNN, AMP, etc.
Reinstalled Ubuntu at least 3 times in 4 hours
Began training the model once more, and attained a few results
Stepped away from the machine to allow it to train
Returned to a black screen...
Realized the training loop was near-infinite, with no defined stop epoch
Refined the training function to ensure it would only train for my specified amount of epochs
7/5
Reinstalled ubuntu AGAIN
Although the 4th was a holiday, I battled issues with GAN training for hours
Returned to my PC, training is now at around 163 epoch
Since the Adversarial training Defense seemed to be going nowhere, I moved my focus to strictly the defense GAN
The defense GAN is currently intaking images at quite a low resolution, upscaling the images, then downscaling again. This needs to be changed
Ensured image resolution was consistent and high-quality
Began training on our new resolution
After training another 100 epochs, I realized my Ubuntu install was still having a large amount of issues.
Decided to attempt to collect real results from the GAN at this point using the testing script.
Collected Results
Started work on the Week 8 Presentation
Week Eight (7/8-7/12)
7/8
Week 8 presentation
After training the GAN, the results were still not great, so I continued training
Adversarial training proved to be much more complicated than we had initially expected, and we would have needed to alter YOLOP's model structure to preoperly adversarially train the model.
Due to this, we moved away from adversarial training entirely.
Though the GAN was training, I was still experiencing a lot of hardware issues with my PC
Reinstalled Ubuntu once more due to a Gnome Display crash
Somehow networking was disabled , and most features were purged from Ubuntu
Reinstalled AGAIN
7/9
Though I was still experiencing crashes and issues here, training was crashing the PC LESS than before
Stopped training, now at around 170 epoch trained at our new resolution
Began testing results against YOLOP
YOLOP results were very , very bad
Met with advisors to discuss findings and found a new way to move forward
Began re-training at a resolution truly compatible with YOLOP's architecture
7/10
After weeks of stress and confusion due to my PC's poor performance, I finally decided to take my PC apart and clean it very well.
During this teardown, I noticed a potential pinched wire that was part of my 3090's GPU riser.
This lead me to believe the issue was the riser, so it was removed
I also removed a GPU power cable extender, to ensure the GPU is able to take full advantage of my 1000W PSU
After all of this, I detailed the interior of my case, leaving no dust
Finally, I reinstalled Ubuntu one FINAL time
This time, once I started training, the PC had no issues at all.
Temps were stable, clocks were stable, VRAM usage and RAM usage were all stable.
This proved my theory that all of these issues were caused by this riser cable.
Finally, I was able to continue training
7/11
After nearly the whole day of training, the model performance was much better but still not perfect.
Perfect may not have been a true possibility in this GAN, but I wanted to get as close to perfect as possible
After meeting with my advisors once more, we discussed the possiblity that the GAN may have not reached the point of convergence
In an attempt to reach convergence, I trained our model up to 370 epochs.
Convergence was reached around 220 epoch.
Loss metrics for the GAN seemed to never truly stabilize, leading our team to believe this GAN may have some broken code or the model itself may not function properly at this point
7/12
Collected a LARGE amount of synthesized images from the GAN, and their matching FGSM examples (500 each)
Passed all 1000 images into YOLOP for validation, and attained results.
These results were truly shocking, and quite disheartening
Our GAN, for reasons unknown, performed worse than the original perturbed images in YOLOP
This made very...very little sense to me, and caused me a great deal of stress
After meeting with my partner and advisors once more, we decided it would be best to include these finidings, although they were not truly effective or positive
Now, we call upon the research community to further this work, we truly believe a Defense GAN has the possiblity of working if implemented correctly. I think our biggest mistake was not using a more advanced GAN model.
Worked diligently on the final poster.
Week Nine (7/15-7/19)
7/15
Poster presentation and review
Made some edits to the poster to enhance clarity and improve readability
Enhanced various aspects of this site to ensure easy navigation
Remade the graphs for my Defense GAN adversarial defense
7/16
Caught up on some missing information on this page, and ensured I included good details of our process and findings.
Edited the site a bit more, further enhancing features
Edited the GitHub ReadME to enhance readability
Finally finished the "fundamentals" modules
7/17
Worked on Website
Worked on Github
Worked on Documentation
More work on poster
7/18
Further work on website
Worked on deliverables
7/19
Worked on deliverables
Week Ten (7/22-7/26)
7/22
Work on deliverables
7/23
Worked on deliverables
7/24
MID-SURE At MSU in Lansing
7/25
Finalizing deliverables
7/26
Exit day.