Alexa, Who Am I Speaking To?

On this page, we present all the videos we displayed to participants during our survey. The results are presented in our paper: "Alexa, Who Am I Speaking To? Understanding Users’ Ability to Identify Third-Party Apps on Amazon Alexa".

Below, we give each video the same labels as in the paper, and we list the question preceding each section of videos.

[Paper Abstract]: Voice User Interfaces (VUIs) are increasingly common on many Internet of Things (IoT) devices. Amazon has the highest share in the voice-assistant market and supports more than 47,000 third-party applications (“skills”) on its Alexa platform to extend functionality. We study how Alexa’s design decisions when integrating these skills may create unintended security and privacy risks. Our survey of 237 participants finds that users do not understand these skills are often operated by third parties. Additionally, people often confuse third-party skills with native Alexa functions. Finally, they are unaware of the functions that the native Alexa system supports. These misunderstandings may allow attackers to develop third-party skills that operate without users’ knowledge, or even to masquerade as native Alexa functions, posing new threats to user security and privacy. Based on our survey data, we make design recommendations, including visual and audio feedback, to help users distinguish native and third-party skills.

Paper Findings:

1) Many survey participants were unaware that Alexa skills can be (and often are) developed by third parties

2) Some survey participants could not differentiate between native and third-party Alexa skills and messages

3) Some survey participants do not know what voice commands Alexa understands

Survey Videos:

Videos 1A and 1B:

These videos tested whether participants understood Alexa had third-party skills before we told them it does.

Video 1A:

Question: "Immediately as result of the following exchange, what parties do you think know David added a rubber ball to his Amazon cart?"

(Options: Only Amazon, Only Other Parties, Both, Not sure)

Video 1B:

Question: "Immediately as result of the following exchange, what parties do you think know your name is ‘Benji’"?

(Options: Only Amazon, Only Other Parties, Both, Not sure)

Videos 3A-3E:

These videos tested whether participants, once educated that Alexa has skills built by third parties, could differentiate between native and third-party skills.

Question: "Please watch the following videos of a person interacting with Alexa and answer whether they interacted with Alexa's native skills (i.e. skills built into the device by Amazon) or a non-native skill (built by a developer who is not affiliated with Amazon)."

(Options: Native, Non-native, Not sure)

Video 3A: Tell a Joke (Native)

Video 3B: Jeopardy (Non-native)

Video 3C: Baseball Scores (Non-native)

Video 3D: Rain Sounds (Non-native)

Video 3E: Parental Controls (Non-native)

It is worth noting that we built this skill to sound like a native Alexa skill (in reality, parental controls cannot be configured on Alexa through voice).

Audios 3A-3E:

These videos tested whether participants, once educated that Alexa has skills built by third parties, could differentiate between real and fake Alexa system messages.

Question: "Which of the following do you think are real Alexa system messages (i.e., real phrases the device says to users when there are errors, connectivity issues, or other concerns)."