Audio & Video Coding How-To's

How do I record audio from the built-in microphone?

Use the interfaces in Audio Queue Services (AudioToolbox/AudioQueue.h) and Audio File Services (AudioToolbox/AudioFile.h). For details and sample code, see Audio Queue Services Programming Guide.

How do I record audio from the headphone microphone?

After a user plugs in an Apple headphone which contains a microphone, the system automatically uses audio from the headphone microphone instead of from the built-in microphone.

How do I control recording input level?

iPhone OS uses a fixed recording input level.

How do I indicate recording level?

When you use an audio queue object (from Audio Queue Services, AudioToolbox/AudioQueue.h) for recording, you can query the audio queue object's kAudioQueueProperty_CurrentLevelMeterDB property to obtain the current recording level. The value of this property is an array of AudioQueueLevelMeterState structures, one per channel, as seen in Listing 1.

Listing 1: AudioQueueLevelMeterState structure

typedef struct AudioQueueLevelMeterState {

    Float32 mAveragePower;

    Float32 mPeakPower;

}; AudioQueueLevelMeterState;

How do I play short sounds and alerts?

You can play your own short sounds of less than 5 seconds by using System Audio Services, declared inAudioToolbox/AudioServices.h. Follow these steps:

1. Add the sound file to your application bundle. The file has certain restrictions when using System Audio Services:

   • You can use only .caf, .aif, or .wav files.

   • The audio data in the file must be in PCM or IMA/ADPCM (IMA4) format.

   • The file's audio duration must be less than 5 seconds.

2. Create a sound ID object as shown in Listing 2.

3. Listing 2: Creating a sound ID object

   AudioServicesCreateSystemSoundID (fileURL, soundID);

4. Play the sound when appropriate, as shown in Listing 3.

5. Listing 3: Playing the sound

   AudioServicesPlaySystemSound (soundID);

In typical use, which includes playing a sound occasionally or repeatedly, retain the sound ID object until your application quits. If you know that you will use a sound only once--for example, a startup sound--you can destroy the sound ID object immediately after playing the sound, freeing memory.

If your application requires playback of sounds longer than 5 seconds or requires level control or scheduling, use Audio Queue Services. See "How do I play arbitrary length sound files?" and "How do I choose the best audio formats in iPhone OS?"

You cannot play the alert sounds that are built into iPhone OS.

How do I control playback level?

On iPhone, the user controls playback level using the hardware volume control. You can control relative playback level when playing sounds with Audio Queue Services (AudioToolbox/AudioQueue.h). To control relative playback level, create a playback audio queue object and use the AudioQueueSetParameter function with the kAudioQueueParam_Volume parameter.

System Audio Services, AudioToolbox/AudioServices.h, does not offer level control.

How do I indicate playback level?

When you use an audio queue object (from Audio Queue Services, AudioToolbox/AudioQueue.h) for playback, you can query the audio queue object's kAudioQueueProperty_CurrentLevelMeterDB property to obtain the current playback level. The value of this property is an array of AudioQueueLevelMeterState structures, one per channel, as shown in Listing 4.

Listing 4: AudioQueueLevelMeterState structure

typedef struct AudioQueueLevelMeterState {

    Float32     mAveragePower;

    Float32     mPeakPower;

}; AudioQueueLevelMeterState;

How do I play sound files of arbitrary length?

Use the interfaces in Audio Queue Services (AudioToolbox/AudioQueue.h), along with Audio File Services (AudioToolbox/AudioFile.h). When playing sound with Audio Queue Services, you write a callback that sends short segments of audio data to audio queue buffers. In some cases, loading an entire sound file to memory for playback is best, because it minimizes disk access. In other cases, loading just enough data at a time to keep the buffers full is best. Test and evaluate which strategy is best for your application.

For more information, see "Audio and Video Technologies" in iPhone OS Programming Guide, and see Audio Queue Services Programming Guide.

How do I play multiple sounds simultaneously?

Use the interfaces in Audio Queue Services (AudioToolbox/AudioQueue.h). Create one audio queue object for each sound that you want to play. Then specify simultaneous start times for the first audio buffer in each audio queue, using the AudioQueueEnqueueBufferWithParameters function.

The following limitations pertain for simultaneous sounds in iPhone OS, depending on the audio data format:

• AAC, MP3, and ALAC (Apple Lossless) audio: no simultaneous playback; one sound at a time.

• Linear PCM and IMA/ADPCM (IMA4 audio): You can play multiple linear PCM or IMA4 format sounds simultaneously without CPU resource concerns.

How do I choose the best audio formats in iPhone OS?

Apple recommends that all audio playback on iPhone and iPod touch take into account CPU efficiency and battery life. To this end, Apple recommends:

• For uncompressed audio, use 16-bit, little-endian, linear PCM audio packaged in a CAF file. See "How do I convert an audio file to the preferred format for iPhone OS?"

• If you need to play multiple compressed sounds simultaneously, use the IMA/ADPCM audio (IMA4) format.

• If you only need to play a single compressed sound at a time, you can use any of ALAC (Apple Lossless), AAC, or MP3 audio formats, in addition to IMA4.

How do I minimize playback latency?

If immediate playback is important, such as for sound effects in games or time-critical user feedback in applications, make direct use of the IO Remote audio unit. You can also use OpenAL in iPhone OS, which employs the IO Remote audio unit for low latency. See "How do I use OpenAL to play and position sounds?"

How do I trigger vibration in iPhone?

To trigger vibration in iPhone, use System Audio Services (AudioToolbox/AudioServices.h). Call theAudioServicesPlaySystemSound function with the kSystemSoundID_Vibrate constant as an argument. Vibration takes place for a fixed duration.

How do I use OpenAL to play and position sounds?

iPhone OS supports OpenAL 1.1, using a simple panning model to maximize performance and battery life. For OpenAL documentation, see the OpenAL website.

How do I play streamed audio?

To play streamed audio, you connect to a network stream using the CFNetwork interfaces from Core Foundation, such as those in CFHTTPMessage. You then parse the network packets into audio packets using Audio File Stream Services (AudioToolbox/AudioFileStream.h). Finally, you play the audio packets using Audio Queue Services (AudioToolbox/AudioQueue.h). You can also use Audio File Stream Services to parse audio packets from an on-disk file.

How do I convert an audio file to the preferred format for iPhone OS?

The preferred full-quality audio format for iPhone OS is 16-bit, little-endian, linear PCM packaged as a CAF file. To convert an audio file to this format, use the afconvert tool at the command line in Mac OS X, as shown in Listing 5.

Listing 5: Converting an audio file to the preferred format for iPhone OS

/usr/bin/afconvert -f caff -d LEI16 {INPUT} {OUTPUT}

To see all the options available for the afconvert tool, enter afconvert -h at the command line.

How do I use the built-in audio units in iPhone OS?

iPhone OS provides three audio units: a Converter unit (of type (AUConverter), a 3D Embedded Mixer unit (of type AU3DEmbeddedMixer), and an IO Remote unit (of type AUIORemote). Use the interfaces in the Audio Unit framework to open, connect, and use audio units in iPhone OS.

iPhone OS does not support the creation of audio units.

How do I access the hardware volume controller?

Global system volume, including your application's volume, is handled by iPhone OS and is not accessible by applications.

How do I control where sounds play (built-in speaker, dock connector, headphones)?

iPhone OS sends audio to the appropriate device, depending on user preferences.

How do I access audio input and output on the dock connector?

iPhone applications have no direct access to the dock connector. When a user attaches an accessory to the dock connector, the system makes that accessory available for recording or playing audio automatically.

How do I play video on iPhone?

iPhone supports the ability to play back video files directly from your application. Video playback is provided by the methods available in the Media Player framework.

Video playback is supported in full-screen mode only, with a simple fade-in transition to the full screen playback on the device.

How do I use the Media Player framework for video playback?

You can use the methods provided in the Media Player framework to display full-screen video from files in either H.264 (Baseline Profile Level 3.0) format or MPEG-4 Part 2 video (Simple Profile) format.

The Media Player framework comprises a single class, MPMoviePlayerController, which provides support for three most commonly used video playback methods: play, stop, and initWithContentURL:. The play method starts the movie playing, if it is not already visible. The stop method stops the movie playing and if it is visible, pauses if necessary and then hides the movie. If you call the play method again, the movie starts playing from the beginning. The initWithContentURL: method creates a full-screen player for the movie specified by the URL.

How do I initiate video playback in my code?

To initiate video playback, you must know the URL of the file you want to play. The URL is typically a pointer to a file in your application bundle, if your application provides the file. It can also be a pointer to a file on remote server.

You use the URL of the file to instantiate a new instance of the MPMoviePlayerController class. This class presides over the playback of your video file and manages user interactions, such as user taps in the transport controls (if shown).

To initiate playback, you simply call the play method of the controller.

The code in Listing 6 shows a sample method that plays back the video at the specified URL. The playmethod is an asynchronous call that returns control to the caller while the movie plays. The movie controller loads the movie in full-screen view (that is, in widescreen), and animates the movie into place on top of the application's existing content. When playback is finished, the movie controller sends a notification to the object, which releases the movie controller, as it is no longer needed.

Listing 6: Initiating video playback

-(void)playMovieAtURL:(NSURL*)theURL

{
    MPMoviePlayerController* theMovie=[[MPMoviePlayerController alloc] initWithContentURL:theURL];
    theMovie.scalingMode=MPMovieScalingModeAspectFill;
    theMovie.userCanShowTransportControls=NO;

    // Register for the playback finished notification.

    [[NSNotificationCenter defaultCenter] addObserver:self
                                            selector:@selector(myMovieFinishedCallback:)
                                                name:MPMoviePlayerPlaybackDidFinishNotification
                                              object:theMovie];

    // Movie playback is asynchronous, so this method returns immediately.
    [theMovie play];
}

// When the movie is done,release the controller.
-(void)myMovieFinishedCallback:(NSNotification*)aNotification
{
    MPMoviePlayerController* theMovie=[aNotification object];
    [[NSNotificationCenter defaultCenter] removeObserver:self
                                                    name:MPMoviePlayerPlaybackDidFinishNotification
                                                  object:theMovie];

    // Release the movie instance created in playMovieAtURL
    [theMovie release];
} 

How do I take advantage of video playback if I'm a game developer?

If you are a game developer who wants to play cut scene animations (that is, a sequence in a video game that is not controlled by the player), you can use the methods provided in the Media Player framework, which enable you to play or animate video.

If you use OpenGL ES in your game, the amount of VRAM you use may influence video playback performance. As a result, you might have to completely shutdown your OpenGL ES contexts and surfaces before playing video.