iPhone/iPad開發札記2012/03/11 －2 Audio Unit

1. Reading 

"Audio Unit Hosting Guide for iOS" /

 "Audio Unit Processing Graph Service References" / - for managing the graph ( a collection of units )

"Audio Component Services Reference" / - for locating/loading a audio unit component 

"Audio Unit Component Services References" - for managing a specific audio unit itself

* threading and locks 

A graph object’s state can be manipulated in both the rendering thread and in other threads. Consequently, any activities that affect the state of the graph are guarded with locks and a messaging model between any calling thread and the thread upon which the graph object’s output unit is called (the render thread).

* identification of an Audio Unit 

    typedef struct AudioComponentDescription {

        OSType  componentType;            // mandatory

        OSType  componentSubType;         // mandatory

        OSType  componentManufacturer;    // mandatory 

        UInt32  componentFlags;           // always 0

        UInt32  componentFlagsMask;       // always 0

    } AudioComponentDescription;

注：iOS system has already registered 7 Audio Unit for us to use. Use the above structure to find one.

* AURenderCallback

Called by the system when an audio unit requires input samples, or before and after a render operation.

^What's this ?

typedef OSStatus (*AURenderCallback) (
    void                        *inRefCon,          // meaning "reference context" 
    AudioUnitRenderActionFlags  *ioActionFlags,
    const AudioTimeStamp        *inTimeStamp,
    UInt32                      inBusNumber,
    UInt32                      inNumberFrames,
    AudioBufferList             *ioData
);

ioData

The AudioBufferList that will be used to contain the rendered or provided audio data.

^ When "rendered" and when "provided" ?

* scope and element 

scope=input ( stream )  / output ( stream ) / global ( for whole Audio Unit ) 

element(bus) = some kind of function 

A mixer unit, for example, might have several input elements but a single output element.

Note: This means an element(bus) can only be decided by both scope+bus number. 

* I/O Unit Essentials 

always be used 

only Unit to start/stop the AUGraph processing

input element + output element + callback ( my own app ) 

* AUGraph / AUNode

Provides the capabilities of managing all Units and Callbacks.

It is THREAD SAFETY, which means you could change the configuration when the audio is played. ( this is the only place this kind of thing could happen. )

AUNode = proxy of Audio Unit under AUGraph context ( like adding an Audio Unit into a AUgraph)  , however, need to handle Audio Unit directly when you want to configure the Audio Unit.

* Constructing an AUGraph

1. Adding nodes to a graph
2. Directly configuring the audio units represented by the nodes
3. Interconnecting the nodes

* Figure 4-1: a simple AUGraph for playback 

Mixer Unit + Remote I/O Unit 

Q: output element of Mixer Unit connected to the output element of I/O Unit ( why not input element ? ) 

* Render Callback Function 

render callback function lives on a real-time priority thread on which subsequent render call comes asynchronously, which means the render call must run FAST ENOUGH!

NOT to do the following: 

# lock 

# memory allocate 

# file system 

# network connection 

* AURenderCallback 

As a notification listener, the system invokes this callback before and after an audio unit’s render operations.

As a render operation input callback, it is invoked when an audio unit requires input samples for the input bus that the callback is attached to.

* Parameter: inNumberFrame 

inNumberFrames parameter indicates the number of audio sample frames that the callback is being asked to provide on the current invocation. You provide those frames to the buffers in the ioData parameter.

Q: How does the system decide the value of inNumberFrame ???

* AudioUnitSampleType 

The canonical audio data sample type for audio processing.

typedef SInt32 AudioUnitSampleType;

#define kAudioUnitSampleFractionBits 24

Discussion

The canonical audio sample type for audio units and other audio processing in iPhone OS is noninterleaved(?) linear PCM with 8.24-bit fixed-point samples.

Q: What does this really mean ?

* ASBD

layout of bits: 

kAudioFormatFlagsAudioUnitCanonical = 

kAudioFormatFlagIsFloat |

kAudioFormatFlagsNativeEndian |

kAudioFormatFlagIsPacked |

kAudioFormatFlagIsNonInterleaved

* unit connection & stream format propagation

one direction 

happening when AUGraph is initialized. 

* hardware sample rate 

To use hardware sample rate whenever possible

Q: Should we use 44100 HZ rather than 16000 HZ for our iPad conference client 

* configure Audio Session: sample rate  

set the hardware sample rate to intended sample rate, which avoids sample rate conversation ( IMPORTANT ! )

[mySession setPreferredHardwareSampleRate:graphSampleRate // 2

error: &audioSessionError];

* configure Audio Session: ioBufferDuration

There’s one other hardware characteristic you may want to configure: audio hardware I/O buffer duration. The default duration is about 23 ms at a 44.1 kHz sample rate, equivalent to a slice size of 1,024 samples. If I/O latency is critical in your app, you can request a smaller duration, down to about 0.005 ms (equivalent to 256 samples), as shown here:

self.ioBufferDuration = 0.005;

[mySession setPreferredIOBufferDuration: ioBufferDuration

error: &audioSessionError];

* 雜問

Q: 似乎沒有見到對ASBD清0的地方？

2. Studying MixerHost sample project 

3.Write my own Audio Unit in conference project. 

4. Conference project 

1:30 PM: 其它程序裏也出現 Notification錯誤提示，以下問題需要弄明：

＊是否只有speex打印相關問題？

＊爲什麼昨天沒有相關問題出來？什麼變了？昨天用了黑蘋果今天用真機了？

5. 深度G711版本的conference

* 啓動會議失敗

root cause：*.ini文件需要同時改meeting no和start time

* iPad加會失敗

console顯示：40206 ????

[Session started at 2012-03-11 20:05:53 +0800.]

2012-03-11 20:05:54.509 test555[921:207] xxxxxxxxxxxx

2012-03-11 20:05:54.613 test555[921:6803] onResetVideoSize

2012-03-11 20:05:54.614 test555[921:6803] joinConf

2012-03-11 20:05:54.749 test555[921:6803] 40206