Abstract:

The employment opportunity for Computer Science (CS), Information Technology and Software Engineering and Development (SE) related occupations is projected to grow much faster than the average of all other occupations. Therefore, increase in student enrollment, retention and graduation rate is becoming very important, so is the need for effective teaching in these subjects. Many universities commonly use formal, institutional Student Evaluation of Teaching (SET) systems to measure the teaching effectiveness. After each semester, through SET, students provide feedback and comments for their courses and instructors. However, evaluations are private and only a handful people have access to these. Therefore, these evaluations cannot be utilized to create a common understanding of the students’ expectations, perspective, desired characteristics of the courses and instructors. On the other hand, third party online platforms like RateMyProfessor.com (RMP) are public, solicit anonymous student feedback and host tremendous amount of data about the instructors and their courses. These platforms are also popular among students. We mined and analyzed the RMP data for some research questions, e.g.: What are the common characteristics of the popular CS instructors? How different are they for the SE instructors? Are there any examples of special characteristics, tools and techniques popular CS instructors use? We captured and analyzed more than 9,000 students’ comments for over 300 CS instructors for the top 20 universities in the U.S. and Canada. The paper contributes by presenting the findings for the research questions and making the data and the scripts available for public use for future research.

Technical Paper:

In the proceedings of the 42nd IEEE/ACM International Conference on Software Engineering, Software Engineering Education & Training Track (ICSE - SEET 2020): [PDF] [Slides] [Talk]

Research Questions:

RQ1: What are the common characteristics of the popular CS instructors?

• Picture below you can see a graph of the most popular tags that students used to describe "popular" CS instructors on RateMyProfessor. Looking at this data we can see that tags like "amazing lectures", and "respected" appear more frequently than tags like "test heavy", and "lecture heavy".

RQ2: How do SE instructors perform compared to the popular CS instructors?

• In the two graphs below, you'll see one that compares tag frequency between "popular" CS instructors and "popular" SE instructors, while in the other graph, it compares the two, without the "popular" filter on the SE data. You'll notice that they share some similarities but there are some tags that grow in frequency with the "popular" filter on SE instructors such as "amazing lectures", and "respected".

RQ3: What makes the popular CS instructors so popular? Any examples?

• We manually analyzed over 9000 RateMyProfessor comments and automatically analyzed the same comments for certain keywords.

• We discovered students appreciate more "soft-skills" such as humane, humble, kind and life influencer.

• We also found that students appreciate instructors who provide online video lectures and materials, in fact, 38% of the instructors we examine provided some sort of online materials.

• Students also commented on instructor availability outside of class as a mark of a good instructor.

• Career focus was another big part of student satisfaction. Students seem to find it very helpful when internship preparation is a part of class lectures.

• Finally, students expressed that being responsive to student feedback helped them engage in the class and the materials better than if their feedback was not considered.

Download:

Python Web Crawler:

• Required Libraries: Python 3, Selenium Firefox WebDriver

• Download Script

• Known Issues: The web crawler is very much dependent on the RateMyProfessor.com (RMP) website UI. RMP may have changed its UI since this script is published and may need significant rework before it can work.

Captured Raw Data from RMP:

• Popular Computer Science Professors: json data

• Popular Software Engineering Professors: json data

Data Analysis Code:

• Please contact us for the analysis code

Data Visualization Code:

• Google Colab Python Script: code

Contact:

Aliaksei Kavalchuk
Email: ajk6204 [at] psu [dot] edu
Pennsylvania State University - Abington

Alec Goldenberg
Email: ajg6148 [at] psu [dot] edu
Pennsylvania State University - Abington

Ishtiaque Hussain [corresponding author]
Email: ihussain [at] psu [dot] edu
Pennsylvania State University - Abington