Picture yourself after leaving an interview, replaying every answer in your head, wondering: “Did I really bomb that question about what makes me a great leader?” This nerve-wracking ritual plays out daily for job seekers across the globe. In today's competitive job market, candidates spend hours on end preparing for interviews, yet often lack the comprehensive feedback needed to truly improve their performance. Current AI interview tools promise convenience but often deliver surface-level and generic feedback that fails to capture the nuances of successful responses. As a result, candidates are left in the dark about critical factors such as body language, tone, and authenticity—key elements that can be the deciding factors between securing a job or missing out. 

AceInterview fills the gap in accessible interview coaching by providing a platform that transforms practice sessions into detailed learning experiences. Our interview analysis platform combines computer vision, audio signal processing, and AI to analyze all aspects of user performance - from facial expressions and body language to the clarity of responses and overall engagement. Users receive actionable and comprehensive feedback on key behaviors and soft skills, along with specific recommendations for improvement.

Our mission is to create an accessible project to help job seekers secure their dream jobs.

Overall, our solution aggregated the outputs of 7 feature extractors into a prompt, sent to a Large Language Model (Gemini) to produce feedback for the user. To evaluate these results, the generated feedback was sent to an LLM as a judge to determine its utility and iteratively improve model performance.

To evaluate the performance of our solution, the aforementioned LLM as a judge assigned scores on a 7-point Likert scale for the feedback’s actionability, clarity, constructiveness, specificity, and depth of insight. Improving these metrics was our primary goal to provide the best feedback to our users. Our solution was tested on all 138 videos to see how much we could improve this metric.

Detecting hallucinations is an unsolved problem in the generative AI space. While Retrieval Augmented Generation approaches can help, which inspire our feature extractors, there is currently no way to ensure an absence of hallucinations. Our team has sampled feedback from our generative model and spot-checked references the model alludes to regarding specific actions or words in the videos and their associated timestamps. So far we have not encountered egregious instances of hallucinations.

Lacking human labels is a problem often faced by researchers. Our project is no different. We have focused on generating feedback, but due to time and resource constraints have not been able to acquire human ratings to evaluate the generated feedback. To save time and resources, we opted to use an LLM as a judge. Zheng et al. 2023 (https://arxiv.org/pdf/2306.05685) showed that LLM judges can be highly correlated with human judgments. This is aided by the human feedback reinforcement learning scheme used to train these LLMs to align these models with human preferences. As such, we attempted to build our own automated LLM as a judge system. Our initial iteration also uses Gemini as the model Judge.

We had our LLM judge on 5 categories: 

• Actionability: how actionable the feedback is based on the specificity and clarity of the recommendations provided

• Clarity: how clearly and logically the feedback is presented

• Constructiveness: how constructive and motivational the feedback is.

• Specificity: how specific and detailed the feedback is.

• Depth of Insight: the depth of analysis in the feedback.

These were rated on a 7-point Likert scale similar to those found in psychological studies where 1 indicates a poor score and 7 indicates an excellent score. We selected a couple of feedback sample pairs for comparison and generally found that our own preferences were directionally captured by the LLM judge. 

We run our baseline feedback and our current solution feedback through the LLM as a Judge and extract the scores.  Our current solution scores better on all categories relative to our GPT-4o baseline with the most gain in actionability with a gain of almost 2 full points. There is only a slight increase in the Constructiveness attribute across our models. Overall, these results show that our approach is promising but requires further validation by human evaluators and in practice by users.