Naman Kukreti

In the case of ChatGPT as a Personal Cyber Security Assistant

Abstract

Computer science (CS) has been rapidly accelerating in terms of job popularity (Burea of Labor Statistics, 2022). Two of the most prominent studies in CS have been Cybersecurity and Artificial Intelligence (AI), in particular, Generative AI (GenAI). GenAI, almost entirely popularized by the introduction of ChatGPT (Zandt, F., & Richter, F. 2024 ), was launched in late November 2022 by OpenAI and has greatly impacted research since (Lund, B. D., & Wang, T. 2023). ChatGPT’s potential deployment in Cybersecurity as an assistant (R. Dreyling, E. Jackson and I. Pappel, 2021) is of great interest, and its potential abilities were tested with three courses offered at Marist College: Intro to Cybersecurity (416N), Hacking and Penetration Testing (417N), and Mobile Security (418N). These courses were chosen to gauge a general understanding of GPT’s Cybersecurity knowledge, which we hoped to extrapolate to its potential effectiveness as a Cybersecurity assistant. The courses’ tests were input into GPT and then their grades were calculated, scoring a mean of 82.7% on 415N, 87% on 417N, and 80% on 418N when using GPT 3.5. The results on 416N, when compared to a class mean of 93.4% in 416N with a standard deviation of 0.7%, give us a z-score of -13.8, suggesting a major gap in accuracy. These scores may be impacted by hallucinations, a phenomenon where GenAI models provide inaccurate information due to their training (Lund, B. D., & Wang, T. 2023). These hallucinations were categorized as Contradictory Hallucinations or Out-of-Range Hallucinations. The poor performance of GPT 3.5 on these college courses as compared to students allows us to conclude that GPT’s effectiveness as a Cybersecurity assistant is dismal.