Test Plan and QA Strategy for PortAudio

Our goal is to have a set of tests that can be used to quickly verify PortAudio on multiple platforms.

Please see our test status for each platform.

We have two kinds of tests.

• Self Checking Unit Tests – These do not require a human to verify the results.

• Listening Tests – These require a human to run a program and listen to the results. We are converting these to unit tests.

We recommend that developers periodically run the Self Checking Unit Tests and a few selected Listening Tests before checking in changes to PortAudio.

Self Checking Unit Tests

These tests can be run from the command line. They have a return code of zero if they pass, non-zero if they fail. This allows them to be run as part of an automated test suite.

qa/paqa_errs.c – This test deliberately generates errors then verifies that the appropriate error message is returned.

qa/paqa_devs.c – This test tries to output audio using many combinations of device, numChannels, format, interleave, etc.

qa/paqa_latency.c – This test verifies that the default latencies for each device are non-zero, and that suggestedLatency is honored.

Loopback Test – This test listens to its own output using a loopback cable. It plays sine waves then analyzes the audio input. It can detect pops, noise and phase errors. The loopback tests will ultimately be able to replace most of the Listening Tests. The loopback test can be built on Linux and Mac using “make loopback”.

Please see this file for loopback instructions: http://www.assembla.com/code/portaudio/subversion/nodes/portaudio/trunk/qa/loopback/README.txt

Listening Tests

We highly recommend that the following tests and examples be run before checking in changes to portaudio.

• examples/pa_devs.c – List the available devices and host APIs. Please verify that it seems complete and correct.
• examples/paex_sine.c – Plays a sine wave using a callback.
• examples/paex_write_sine.c – Plays a sine wave using a blocking write.
• examples/paex_record.c – The program will ask you to record a few seconds of audio then play it back. You will need a microphone or other audio source.

• test/patest_multi_sine.c – Listen for sine wave with no pops or clicks on every channel. This is a good test for N>2 channel devices.
• test/patest_leftright.c – A tone will be played on the left and right channel. Make sure you can hear both tones.
• test/patest_stop_playout.c – Tests stopping of a stream in various ways.
• test/patest_wire.c – Test full duplex input and output on one stream. You should hear the audio input passed to the output.

A list of additional tests can be found here: http://www.portaudio.com/docs/v19-doxydocs-dev/group__test__src.html

Most of those additional tests are useful and are expected to build and run. But the following tests have problems.

Broken Build Tests

These tests do not build. They should be reviewed and probably deleted.

• patest_converters.c – Link error with PaUtil_SelectConverter.

Failing Tests

• patest_in_overflow.c – Good test. But on a fast computer it takes too long to load up the CPU.
• patest_out_underflow.c – Good test. But on a fast computer it has trouble causing underflows.
• patest_clip.c – Test expects over-range sound to wrap and sound nasty. But Mac uses floats at device level so it does its own clipping. It does not sound nasty. It just works.
• patest_latency.c – Does not respond to key presses on Mac.
• patest_prime.c – Latency is never high enough on Mac to hear the difference between primed and non-primed.
• patest_read_write_wire.c – Use blocking read/write in full duplex mode. Fails quickly on Mac with input overflow so I had to disable the error checks. Needs redesign with priming of out output buffers.
• patest_sine_channelmaps.c – No sound heard on Mac.
• patest_stop.c – Sounds does not get cut short on Mac. Test relies on 3 second latency. Not possible on Mac.
• patest_write_sine_nonint.c – Test of blocking write with non-interleaved buffers. Broken on Mac.

Pointless Tests

These programs are either redundant or unnecessary as tests. We should delete them instead of maintaining them. This is only a partial list.

• patest_pink.c – Nice demonstration of algorithm but not relevant as a PA test.
• patest_ringmix.c – Cute audio hack but pointless.
• patest_toomanysines.c – Redundant because of patest_out_overflow.c

Platform Specific Tests

Windows

• patest_dsound_surround.c
• patest_jack_wasapi.c
• patest_mono_asio_channel_select.c
• patest_wmme_ac3.c
• patest_wmme_low_level_latency_params.c
• patest_wmme_surround.c

Linux

Macintosh

• patest_sine_channelmaps.c