My son loves features, so I get to discover interactions :-)
Problem: When System Preferences->Speech->Speakable Items is enabled, the Esc is grabbed as a switch to turn on and off the microphone. The Esc key therefore never gets to Terminal.app nor vi.
Workarounds:
1. Press F1. This opens the help window and ends the insert/open mode. Then Press either :q and Enter
or press ZZ to exit help.
2. Press Control-Shift-[ . Yes, it is three keys but it does work. I learned this when DEC's VT220 Terminal eliminated the ESCAPE key.
3. Turn of Speakable Items.
4. Change the Listening Key to something else.
Discovery: I was using Terminal.app and I couldn't exit INSERT mode in vi (really vim). The problem was the "Speech Commands" was on. The "Listening Key" which allows the Mac to listen for Spoken commands, is by default, you guessed it, Esc. I checked it out by typing: echo Control-V and Escape which yielded nothing but the echo command. Whereas echo Control-V and the F1 key yielded echo ^[OP which mean the escape encoded the the F1 key was making it through. The Escape OP goes back to the VT100, see
http://vt100.net/manx/details/1,2934 and check out PF1 in "Table A-8 ANSI Mode Auxiliary Keypad Codes"
on Page A-10 which is page 302 of 348 of the PDF.
Cheat sheet for vim http://people.csail.mit.edu/vgod/vim/vim-cheat-sheet-en.png