Testing the Midi Kit

Most of the Haiku source code has unit tests in the current/src/tests directory. I looked into building CppUnit tests for the midi2 kit, but decided that it doesn’t really make much sense. Unit tests work best if you can test something in isolation, but in the case of the midi2 kit this is very hard to achieve. Because the classes from libmidi2.so always need to talk to the midi_server, the tests depend on too many external factors. The available endpoints, for example, will differ from system to system. The spray and hook functions are difficult to test this way, too.

So instead of a CppUnit test suite, here is a list of manual tests that I performed when developing the midi2 kit:

======================================================================

Registering the application

*Required:* Client app that calls BMidiRoster::MidiRoster()

• When a client app starts, it should first receive mNEW notifications for all endpoints in the system (even unregistered remotes), followed by mCON notifications for all connections in the system (even those between two unregistered local endpoints from another app).

• Send invalid Mapp message (without messenger). The midi_server ignores the request, and the client app blocks forever.

• Fake a delivery error for the mNEW notifications and the mAPP reply. (Add a snooze() in the midi_server’s OnRegisterApplication(). While it is snoozing, Ctrl-C the client app. Now the server can’t deliver the message and will unregister the application again.)

• Kill the server. Start the client app. It should realize that the server is not running, and return from MidiRoster(); it does not block forever.

• Note: The server does not protect against sending two or more Mapp messages; it will add a new app_t object to the roster and it will also send out the mNEW and mCON notifications again.

• Verify that when the client app quits, the BMidiRoster instance is destroyed by the BMidiRosterKiller. The BMidiRosterLooper is also destroyed, along with any endpoint objects from its list. We don’t destroy endpoints with a refcount > 0, but print a warning message on stderr instead.

• When the app quits before it has created a BMidiRoster instance, the BMidiRosterKiller should do nothing.

======================================================================

Creating endpoints

*Required:* Client app that creates a new BMidiLocalProducer and/or BMidiLocalConsumer

• Send invalid Mnew message (missing fields). The server will return an error code.

• Don’t send reply from midi_server. The client receives a B_NO_REPLY error.

• If something goes wrong creating a new local endpoint, you still get a new BMidiEndpoint object (but it is not added to BMidiRosterLooper’s internal list of endpoints). Verify that its ID() function returns 0, and IsValid() returns false. Verify that you can Release() it without crashing into the debugger (i.e. the reference count of the new object should be 1).

• Snooze in midi_server’s OnCreateEndpoint() before sending reply to client to simulate heavy processor load. Client should timeout. When done snoozing, server fails to deliver the reply because the client is no longer listening, and it unregisters the app.

• Note: if you kill the client app with Ctrl-C before the server has sent its reply, SendReply() still returns okay, and the midi_server adds the endpoint, even though the corresponding app is dead. There is not much we can do to prevent that (but it is not really a big deal).

• Start the test app from two different Terminals. Verify that the new local endpoint of app1 is added to the BMidiRosterLooper’s list of endpoints, and that its “isLocal” flag is true. Verify that when you start the second app, it immediately receives mNEW notifications for the first app’s endpoints. It should also create BMidiEndpoint proxy objects for these endpoints with “isLocal” set to false, and add them its own list. Vice versa for the endpoints that app2 creates. Verify that the “registered” field in the mNEW notification is false, because newly created endpoints are not registered yet. The “properties” field should contain an empty message.

• Start server. Start client app. The app makes new endpoints and the server adds them to the roster. Ctrl-C the app. Start client app again. The new client first receives mNEW notifications for the old app’s endpoints. When the new app tries to create its own endpoints, the server realizes that the old app is dead, and sends mDEL notifications for the now-defunct endpoints.

• The test app should now create 2 endpoints. Let the midi_server snooze during the second create message, so the app times out. The server now unregisters the app and purges its first endpoint (which was successfully created).

• The test app should now create 3 endpoints. Let the midi_server snooze during the second create message, so the app times out. (It also times out when sending the create request for the 3rd endpoint, because the server is still snoozing.) Because it cannot send a reply for the 2nd create message, the server now unregisters the app and purges its first endpoint (which was successfully created). Then it processes the create request for the 3rd endpoint, but ignores it because the app is now no longer registered with the server.

• Purging endpoints. The test app should now create 2 endpoints. Let the midi_server snooze during the _fourth_ create message. Run the server. Run the test app. Run the test app again in a second Terminal. The server times out, and unregisters the second app. The first app should receive an mDEL notification. Repeat, but now the test app should make 3 endpoints and the server fails on the _sixth_ endpoint. The first app now receives 2 mDEL notifications.

• You should be allowed to pass NULL into the BMidiLocalProducer and BMidiLocalConsumer constructor.

• Let the midi_server assign random IDs to new endpoints; the BMidiRosterLooper should sort the endpoints by their IDs when it adds them to its internal list.

======================================================================

Deleting endpoints

*Required:* client app that creates one or more endpoints and Release()’s them

• Verify that Acquire() increments the endpoint’s refcount and Release() decrements it. When you Release() a local endpoint so its refcount becomes zero, the client sends an Mdel request to the server. When you Release() a local endpoint too many times, your app jumps into the debugger.

• Send an Mdel request with an invalid ID to the server. Examples of invalid IDs: -1, 0, 1000 (or any other large number).

• Start the test app from two different Terminals. Note that when one of the apps Release()’s its endpoints, the other receives corresponding mDEL notifications.

• Snooze in midi_server’s OnCreateEndpoint() before sending reply to “create endpoint” request. The client will timeout and the server will unregister the app. Now have the client Release() the endpoint. This sends a “delete endpoint” request to the server, which ignores the request because the app is no longer registered.

• Override BMidiLocalProducer and BMidiLocalConsumer, and provide a public destructor. Call “delete prod; delete cons;” from your code, instead of using Release(). Your app should drop into the debugger.

• Start the client app and let it make its endpoints. Kill the server. Release() the endpoints. The server doesn’t run, so the Mdel request never arrives, but the BMidiEndpoint objects should be deleted regardless.

• Start the test app from two different Terminals, and let them make their endpoints. Quit the apps (using the Deskbar’s “Quit Application” menu item). Verify that both clean up and exit correctly. App1 removes its own endpoint from the BMidiRosterLooper’s list of endpoints and sends an ‘mDEL’ message to the server, which passes it on to app2. In response, app2 removes the proxy object from its own list and deletes it. Again, vice versa for the endpoint from app2.

• Start both apps again and wait until they have notified each other about the endpoints. Ctrl-C app1, and restart it. Verify that app1 receives the ‘mNEW’ messages and creates proxies for these remote endpoints. Both apps should receive an ‘mDEL’ message for app1’s old endpoint (because the midi_server realizes it no longer exists and purges it), and remove it from their lists accordingly.

======================================================================

Changing attributes

*Required:* Client app that creates an endpoint and calls Register(), Unregister(), SetName(), and SetLatency()

• Send an Mchg request with an invalid ID to the server.

• Register() a local endpoint that is already registered. This does not send a message to the server and always returns B_OK. Likewise for Unregister()ing a local endpoint that is not registered.

• Register() or Unregister() a remote endpoint, or an invalid local endpoint. That should immediately return an error code.

• Verify that BMidiRoster::Register() does the same thing as BMidiEndpoint::Register(). Also for BMidiRoster::Unregister() and BMidiEndpoint::Unregister().

• If you pass NULL into BMidiRoster::Register() or Unregister(), the functions immediately return with an error code.

• SetName() should ignore NULL names. When you call it on a remote endpoint, SetName() should do nothing. SetName() does not send a message if the new name is the same as the current name.

• SetLatency() should ignore negative values. SetLatency() does not send a message if the new latency is the same as the current latency. (Since SetLatency() lives in BMidiLocalConsumer, you can never use it on remote endpoints.)

• Kill the server after making the new endpoint, and call Register(). The client app should return an error code. Also for Unregister(), SetName(), SetLatency(), and SetProperties().

• Snooze in the midi_server’s OnChangeEndpoint() before sending the reply to the client. Both sides will flag an error. No mCHG notifications will be sent. The server unregisters the app and purges its endpoints.

• Verify that other apps will receive mCHG notifications when the test app successfully calls Register(), Unregister(), SetName(), and SetLatency(), and that they modify the corresponding BMidiEndpoint objects accordingly. Since clients are never notified when they change their own endpoints, they should ignore the notifications that concern local endpoints. Latency changes should be ignored if the endpoint is not a consumer.

• Send an Mchg request with only the “midi:id” field, so no “midi:name”, “midi:registered”, “midi:latency”, or “midi:properties”. The server will still notify the other apps, although they will obviously ignore the notification, because it doesn’t contain any useful data.

• The Mchg request is overloaded to change several attributes. Verify that changing one of these attributes, such as the latency, does not overwrite/wipe out the others.

• Start app1. Wait until it has created and registered its endpoint. Start app2. During the initial handshake, app2 should receive an ‘mNEW’ message for app1’s endpoint. Verify that the “refistered” field in this message is already true, and that this is passed on correctly to the new BMidiEndpoint proxy object.

• GetProperties() should return NULL if the message parameter is NULL.

• The properties of new endpoints are empty. Create a new endpoint and call GetProperties(). The BMessage that you receive should contain no fields.

• SetProperties() should return NULL if the message parameter is NULL. It should return an error code if the endpoint is remote or invalid. It should work fine on local endpoints, registered or not. SetProperties() does not compare the contents of the new BMessage to the old, so it will always send out the change request.

• If you Unregister() an endpoint that is connected, the connection should not be broken.

======================================================================

Consulting the roster

*Required:* Client app that creates several endpoints, and registers some of them (not all), and uses the BMidiRoster::FindEndpoint() etc functions to examine the roster.

• Verify that FindEndpoint() returns NULL if you pass it:

• invalid ID (localOnly = false)

• invalid ID (localOnly = true)

• remote non-registered endpoint (localOnly = false)

• remote non-registered endpoint (localOnly = true)

• remote registered endpoint (localOnly = true)

Verify that FindEndpoint() returns a valid BMidiEndpoint object if you pass it:

• local non-registered endpoint (localOnly = false)

• local non-registered endpoint (localOnly = true)

• local registered endpoint (localOnly = false)

• local registered endpoint (localOnly = true)

• remote registered endpoint (localOnly = false)

• Verify that FindConsumer() works just like FindEndpoint(), but that it also returns NULL if the endpoint with the specified ID is not a consumer. Likewise for FindProducer().

• Verify that NextEndpoint() returns NULL if you pass it NULL. It also returns NULL if no more endpoints exist. Otherwise, it returns a BMidiEndpoint object, bumps the endpoint’s reference count, and sets the “id” parameter to the ID of the endpoint. NextEndpoint() should never return local endpoints (registered or not), nor unregistered remote endpoints. Verify that negative “id” values also work.

• Verify that you can safely call the Find and Next functions without having somehow initialized the BMidiRoster first (by making a new endpoint, for example). The functions themselves should call MidiRoster() and do the handshake with the server.

• The Find and Next functions should bump the reference count of the BMidiEndpoint object that they return. However, they should not (inadvertently) modify the refcounts of any other endpoint objects.

• Get a BMidiEndpoint proxy for a remote published endpoint. Release(). Now it should not be removed from the endpoint list or even be deleted, even though its reference count dropped to zero.

• Start app1. Start app2. App2 gets a BMidiEndpoint proxy for a remote endpoint from app1. Ctrl-C app1. Start app1 again. Now app2 receives an mDEL message for app1’s old endpoint. Verify that the endpoint is removed from the endpoint list, but not deleted because its reference count isn’t zero. If app2 now Release()s the endpoint, the BMidiEndpoint object should be deleted. Try again, but now Release() the endpoint before you Ctrl-C; now it should be deleted and removed from the list when you start app1 again.

======================================================================

Making/breaking connections

*Required:* Client app that creates a producer and consumer endpoint, optionally registers them, consults the roster for remote endpoints, and makes various kinds of connections.

• Test the following for BMidiProducer::Connect():

• Connect(NULL)

• Connect(invalid consumer)

• Connect() using an invalid producer

• Send Mcon request with invalid IDs

• Kill the midi_server just before you Connect()

• Let the midi_server snooze, so the connect request times out

• Have the midi_server return an error result code

• On successful connect, verify that the consumer is added to the producer’s list of endpoints

• Verify that you can make connections between 2 local endpoints, a local producer and a remote consumer, a remote producer and a local consumer, and two 2 remote endpoints. Test the local endpoints both registered and unregistered.

• 2x Connect() on same consumer should give an error

• The other applications should receive an mCON notification, and adjust their own local rosters accordingly

• If you are calling Connect() on a local producer, its Connected() hook should be called. If you are calling Connect() on a remote producer, then its own application should call the Connected() hook.

• Test the following for BMidiProducer::Disconnect():

• Disconnect(NULL)

• Disconnect(invalid consumer)

• Disconnect() using an invalid producer

• Send Mdis request with invalid IDs

• Kill the midi_server just before you Disconnect()

• Let the midi_server snooze, so the disconnect request times out

• Have the midi_server return an error result code

• On successful disconnect, verify that the consumer is removed from the producer’s list of endpoints

• Verify that you can break connections between 2 local endpoints, a local producer and a remote consumer, a remote producer and a local consumer, and two 2 remote endpoints. Test the local endpoints both registered and unregistered.

• Disconnecting 2 endpoints that were not connected should give an error

• The other applications should receive an mDIS notification, and adjust their own local rosters accordingly

• If you are calling Disconnect() on a local producer, its Disconnected() hook should be called. If you are calling Disconnect() on a remote producer, then its own application should call the Disconnected() hook.

• Make a connection on a local producer. Release() the producer. The other app should only receive an mDEL notification. Likewise if you have a connection with a local consumer and you Release() that. However, now all apps should throw away this consumer from the connection lists, invoking the Disconnected() hook of local producers. The same thing happens if you Ctrl-C the app and restart it. (Now the old endpoints are purged.)

• BMidiProducer::IsConnected() should return false if you pass NULL or an invalid consumer.

• BMidiProducer::Connections() should return a new BList every time you call it. The objects in this list are the BMidiConsumers that are connected to this producer; verify that their reference counts are bumped for every call to Connections().

======================================================================

Watching

*Required:* Client app that creates local consumer and producer endpoints, and calls Register(), Unregister(), SetName(), SetLatency(), and SetProperties(). It should also make and break connections.

• When you call StartWatching(), you should receive B_MIDI_EVENT notifications for all remote registered endpoints and the connections between them. You will get no notifications for local endpoints, or for any connections that involve unregistered endpoints. The BMidiRosterLooper should make a copy of the BMessenger, so when the client destroys the original messenger, you will still receive notifications. Verify that calling StartWatching() with the same BMessenger twice in a row will also send the initial set of notifications twice. StartWatching(NULL) should be ignored and does not remove the current messenger.

• Run the client app from two different Terminals. Verify that you receive properly formatted B_MIDI_EVENT notifications when the other app changes the attributes of its *registered* endpoints with the various Set() functions. You should also receive notifications if the app Register()s or Unregister()s its endpoints. That app that makes these changes does not receive the notifications.

• Run the client app from two different Terminals. Verify that you receive properly formatted B_MIDI_EVENT notifications when the apps make and break connections. Every app receives these connection notifications, whether the endpoints are published or not. The app that makes and breaks the connections does not receive any notifications.

• StopWatching() should delete BMidiRosterLooper’s BMessenger copy, if any. Verify that you no longer receive B_MIDI_EVENT notifications for remote endpoints after you have called StopWatching().

• If the client is watching, and the BMidiRosterLooper receives an mDEL notification for a registered remote endpoint, it should also send an “unregistered” B_MIDI_EVENT to let the client know that this endpoint is no longer available. If the endpoint was connected to anything, you’ll also receive “disconnected” B_MIDI_EVENTs.

• If you get a “registered” event, and you do FindEndpoint() for that id, you’ll get its BMidiEndpoint object. If you get an “unregistered” event, then FindEndpoint() returns NULL. So the events are send *after* the roster is modified.

======================================================================

Event tests

*Required:* Several client apps that create and register consumer endpoints that override the various MIDI event hook functions, as well as producer endpoints that spray MIDI events. Also useful is a tool that lets you make connections between all these endpoints (PatchBay), and a tool that lets you monitor the MIDI events (MidiMonitor).

• BMidiLocalProducer’s spray functions should only try to send something if there is one or more connected consumer. If the spray functions cannot deliver their events, they simply ignore that consumer until the next spray. (No connections are broken or anything.)

• All spray functions except SprayData() should set the atomic flag to true, even SpraySystemExclusive().

• When you send a sysex message using SpraySystemExclusive(), it should add 0xF0 in front of your data and 0xF7 at the back. When you call SprayData() instead, no bytes are added to the MIDI event data.

• Verify that all events arrive correctly and that the latency is minimal, even when the load is heavy (i.e. many events are being sprayed to many different consumers).

• Verify that the BMidiLocalConsumer destructor properly destroys the corresponding port and event thread before it returns.

• BMidiLocalConsumer should ignore messages that are too small, addressed to another consumer, or otherwise invalid.

• BMidiLocalConsumer’s Data() hook should ignore all non-atomic events. The rest of the events, provided they contain the correct number of bytes for that kind of event, are passed on to the other hooks.

• Hook a producer up to a consumer and call all SprayXXX() functions with a variety of arguments to make sure the correct hooks are being called with the correct values. Call SprayData() and SpraySystemExclusive() with NULL data and/or length 0.

• Call GetProducerID() from one of BMidiLocalConsumer’s hooks to verify that this indeed returns the ID of the producer that sprayed the event.

• To test timeouts, first call SetTimeout(system_time() + 2000000), spray an event to the consumer, and wait 2 seconds. The consumer’s Timeout() hook should now be called. Try again, but now spray multiple events to the consumer. The Timeout() hook should still be called after 2 seconds, measured from the moment the timeout was set. Replace the call to SetTimeout() with SetTimeout(0). After spraying the first event, you should immediately get the Timeout() signal, because the target time was set in the past. Verify that calling SetTimeout() only takes effect after at least one new event has been received.

======================================================================

Other tests

• Kill the server. Now run a client app. It should recognize that the server isn’t running, and return error codes on all operations. Also kill the server while the test app is running. From then on, the client app will return error codes on all operations. Also bring it back up again while the test app is still running. Now the client app’s request messages will be delivered to the server again, but the server will ignore them, because our app did not register with this new instance of the server.

• Start the midi_server and several client apps. Use PatchBay to make and break a whole bunch of connections. Quit PatchBay. Start it again. Now the same connections should show up. Run similar tests with MidiKeyboard. Also install VirtualMidi (and run the old midi_server for the time being) to get a whole bunch of fake MIDI devices.

• *Regression bug:* After you quit one client app, another app fails to send request to the midi_server.

*Required:* Client app that creates a new endpoint and registers it. In the app’s destructor, it unregisters and releases the endpoint.

*How to reproduce:* Run the app from two different Terminals. Ctrl-C app1. Start app1 again. From the Deskbar quit both apps at the same time (that is possible because app1 and app2 both have the same signature). When it tries to send the Unregister() request to the midi_server, app2 gives the error “Cannot send msg to server”. The error code is “Bad Port ID”, which means that the reply port is dead. The Mdel message from Release() is sent without any problems, however, because that expects no reply back. This is not the only way to reproduce the problem, but it seems to be the most reliable one.

The reason this happens is because you kill app1. When app2 sends a synchronous request to the midi_server, the server re-used that same message to notify the other apps. (Because it already contained all the necessary fields.) But app1 is dead, the notification fails, and this (probably) wipes out the reply address in the message. I changed the midi_server to create new BMessages for the notifications, and was no longer able to reproduce the problem.