Product Description

What is your product?

Although many computer music tools exist, they are focused on algorithms, programming effects and loops, and are not for the casual tinkerer or novice. Robot Rock is a fun platform for creating music in real time by conducting simple AI entities, and even someone who knows nothing about music or programming will enjoy it.

Everything centers around a human conductor, who activates AI musicians and changes various global and local musical traits as they perform.

Each AI entity knows how to play one instrument, such as a drum or cymbal or 5-note keyboard. It interprets common information from the user and the other AI performers, and processes this information to determine what it will perform next. These AI will essentially follow a continuous loop of listen, adjust, and perform, improvising a composition through collaboration with each other and guidance from a human conductor, who may control the following aspects of performance: volume, tempo, time signature, key, tonal and rhythmic complexity, and intensity. Further, each instrument may offer specialized controls, i.e. a violin that has the option to perform with or without a bow. Robot Rock will expose even the completely unfamiliar to the world of musical creation and improvisation.

It can also generate revenue: the product can ship with several compelling AI instruments, with more available as simple add-ons.

This project is interesting and challenging because it merges Artifical Intelligence with music, and there are algorithmic challenges for music generation and musical learning. Further, there are design opportunities for creating highly object oriented and reusable code for “plugging in” new instruments on the fly. Finally, since Robot Rock is targeted at novice users, design and interface are critical. In summation, the challenges for Robot Rock are to solve problems related to music generation, artificial intelligence and machine learning, object-oriented design and design patterns, and user experience.

This project may appear ambitious, but since it is highly modular and expandable, and the initial goals are actually fairly basic. A working proof of concept could feature on the order of 3-5 AI bots. They will perform together and still sound musical, if somewhat limited.

The user will have controls to change overall song parameters, as well as the intensity and complexity of specific instrument’s performance. Essentially we are designing a platform for AI music, and the goal is to demonstrate the platform’s appeal and potential. We will offer improvements to the product through future AI instrument plugins.

What alternatives exist?

Currently the only software that fills a similar audience niche is Microsoft’s Songsmith, which appears as a moderate success. Professional tools that are similar are Max/MSP, which retails at $600, and PureData, which is free but has a steep learning curve and is not for novices. GarageBand for the Mac targets amateurs and laymen, but still requires arranging of loops and knowledge of song structure and manual editing. None of these existing products are designed for real time performance.

Although not available as pure software, Reactable and Siftable demonstrate similar concepts by chaining together tone generation and control blocks through an innovative touch interface.

What are its major features? Include at least 4 major features you will provide, along with at least 2 other minor features or aspects you hope to complete.

Major:

• Real-Time Music Generation -- the software must output musical audio while it is running (and not paused).
• Instrument/Musician Manipulation (syncopation, energy, tempo...) -- the user must have controls which alter the characteristics of individual instrument performance. Examples of characteristics which can be modified are intensity, syncopation, pulse, and tonality.
• Appealing, Responsive and Reactive UI -- the UI must be fun to use, and never frustrating.
• Saving and Exporting Generated Music -- the user can save the performance in a standard audio format.
• 3-5 instruments, 2-3 different styles available

Minor:

• AI Producing Quality Music -- the music itself must be complex, harmonious, and interesting
  
• Music Production -- editing and fine-tuning the performance, not in real time
• Sophisticated AI techniques -- AI based on techniques such as machine learning or neural networks
• Library of Instruments/AIs -- A larger variety of instruments which span multiple genres
• Live User-Played Music -- User stands in for an AI instrument by attaching a keyboard
• Hot-Key save states -- Allow the user to assign a hotkey to a particular configuration, then restore that configuration later

What external documentation will you provide that will enable users to understand and use your product? This could take the form of help files, a written manual, integrated help text throughout the UI, etc.

Help text is available to the user in the form of tool tips embedded in the interface. A hint is displayed on startup of how to add an instrument, and tool tips are revealed when the user hovers over an item or widget. The goal is to quickly guide the user into understanding their options and environment. As long as the interface is straightforward, consistent and simplistic, the user will catch on after only a few pop-up tips.

What will the UI look like? Submit diagrams (at least two, possibly more) containing rough sketches of your product's user interface.

Use Cases

Use case: UML

Use case: Add instrument to ensemble

Characteristic Information

• Goal in context: User selects an instrument and includes it in the ensemble.
• Scope: Application
• Preconditions: Maximum number of instruments in ensemble has not been reached.
• Success end condition: Instrument is included in the ensemble.
• Failed end condition: Instrument is not included in the ensemble.
• Primary actor: Robot Rock user
• Trigger: User

Main Success Scenario

1. User exposes available instrument panel.
2. User selects instrument classification.
3. User locates instrument from those in classification.
4. User drags desired instrument from collection to ensemble space.
5. Instrument appears in ensemble space.

Extensions

1. Panel indicates an instrument cannot be added if instrument limit reached.

Variations

1. Instrument panel may already be exposed.
2. Classification may already be selected from previous action.

Related Information

• Priority: Top
• Superordinate use case: UseCase_DanceBeat
• Subordinate use cases: None.
• Secondary actors: Instrument panel, ensemble space

Schedule

• Due date: Beta release

Open issues

• What is the state of newly added instruments? Playing, muted?
• At what point is single-layer instrument classification too simplistic?

Use case: Performing with an instrument

Characteristic Information

• Goal in context: User controls instruments during a session, resulting as changes in the audio output
• Scope: Application
• Preconditions: An ensemble with at least one instrument is setup.
• Success end condition: Changing an instrument parameter augments or modifies the instrument's audio output contribution.
• Failed end condition: Audio output does not change in an expected/logical way.
• Primary actor: Robot Rock user
• Trigger: User

Main Success Scenario

1. User chooses an instrument from ensemble to modify.
2. User changes an instrument parameter:

   • Position upon stage (Use case: Positioning an instrument.)
   • (Un)Muting an instrument. (Use case: Muting an instrument.)
   • Instrument specific trait. (Use case: Setting an instrument trait.)

3. Changing the instruments parameter results in real-time audio feedback.

Variations

1. Instrument is muted, in which case no change in audio feedback is noticed, unless dependent, non-muted instruments make use of changed parameter.

   • Note (alan): Can we assume for now that a mute is implemented by silencing the instrument's published contribution? If so, the other instruments will remain in the dark until it is unmuted.

Related Information

• Priority: Top
• Performance target: High correlation between visual layout and real-time audio
• Frequency: Arbitrary, sequential
• Superordinate use case: Dance beat, Ambient song
• Subordinate use cases:

  • Positioning an instrument
  • Muting an instrument
  • Setting an instrument trait

• Secondary actors: Instruments, Ensemble

Schedule

• Due date: Beta release

Open issues

• In regards to positional parameters, do instruments change as the icon moves or when it is placed at its destination? Or is this an user controllable option?

  Let's aim for instruments which change when the instrument has been in a stable position for a very brief period of time, creating the illusion of fast updating without firing changes on every MoveEvent.

Use case: Set Up a Dance Beat

Characteristic Information

• Goal in context: User creates a dance beat performance
• Scope: Session
• Preconditions: * Robot Rock is freshly loaded and is in its initial state.
• Success end condition: User has "locked in" a dance-y backing beat and begins to manipulate the lead instruments.
• Failed end condition: User cannot produce the sound they are looking for and give up.
• Primary actor: Robot Rock user
• Trigger: User

Main Success Scenario

1. User adds an instrument: Bass Drum.
2. Bass Drum begins performing automatically.
3. User adds an instrument to the ensemble: Snare Drum.
4. Snare Drum begins performing automatically, complimenting the Bass Drum.
5. User drags the instrument to higher energy, medium syncopation.
6. User clicks the lock button on each and they loop on a continuous beat.
7. User adds an instrument to the ensemble: Funk Bass.
8. Funk Bass begins performing, grooving with the Bass Drum.
9. User adds an instrument to the ensemble: Funk Synth, adding melodies to the Funk Bass.
10. The User approves of the beat; the User can now explore it further by moving the Funk Bass and Funk Synth instruments around.

Extensions

1. User adjusts volumes of individual instruments.
2. User increases tempo of master track.
3. User adds additional drum tracks or hand claps.

Variations

1. Instruments may be added in different order.
2. The User may desire a sound that is less "Funk" and more "Techno." The instruments would be "Electronic" versions of the Funk sounds.

Related Information

• Priority: High
• Superordinate use case: None
• Subordinate use cases: UseCase_AddInstrumentToEnsemble.
• Secondary actors: Instrument panel, ensemble space

Schedule

• Due date: Beta release

Open issues

• Can we supply all of the necessary instruments?
• Should we package some of the drums or other instruments together?

Process

Software Toolset

• Language: Python
• UI: Qt with Python bindings
• Version Control/Team Management: Assembla.com
• Audio Generation: FluidSynth

Group Dynamics

Alan is the de facto Project Manager since it was originally his initiative. The rest of the group members are organized so that every major component of the project has a single point of responsibility. This makes life easier for the project manager and helps reduce the inadvertent or subconscious delegation of ownership between two people who share the same responsibilities.

• Project Manager: Alan
• Audio Lead: Travis
• UI Lead: Tim
• AI Lead: Rich
• Graphics Lead: Mike

Schedule

Week 4

Project Manager

• Produce detailed system architecture with help of leads.

Audio Lead

• Finish experimenting with FluidSynth and pyfluidsynth
• Goal: produce programmed sound; determine if audio module explicitly should handle thread or if library does.
• Determine appropriate interface for audio module
• Assist with inter-module design

UI Lead

• Produce semi-complex throwaway interface for familiarity.

AI Lead

• Describe AI interface and internals.
• Help specify AI stream interface.

Graphics Lead

• Understand animation interface of Qt.
• Goal: produce 2-3 simple, throw-away interface animations.

Week 5

Project Manager

• First draft of functional interface specification complete for all parts.

All leads

• Stub out area of ownership.
• Write unit tests (which should all fail).

AI lead

• Initial AI prototyping and testing

Week 6 - 7

Project Manager

• Finalize interfaces, help code.

All leads

• Build out functionality to pass tests.

Week 8

Beta release. User testing.

Week 9 - 10

Apply user feedback. Build more AIs if other features are ready.

Risk Summary

Risk: Musical Artificial Intelligence

Problems: This risk is a major concern because it is the backbone of our project, and also the most complex module. Music is very precise, and getting AI musicians to coordinate with each other in real time will be challenging. Furthermore, because having a large variety of instruments to choose from is a very exciting feature, complexity may increase drastically before a simpler, working model is created.

Evaluation: Will be an ongoing process. Signs that the AI needs to be reworked are increased code complexity, and decrease in performance (both speed and quality of audio) of the application. User tests can give feedback to the quality of the music generated. General AI concepts should be well designed and documented before implementation.

Risk Management: The AI module should start simple, implementing as few varieties and extra features as possible to ensure that a working framework is completed in time. Responses to a complicated AI include reduction of the number of instruments, reduction of the number of styles of music, and decreases in the number of parameters each AI musician uses to determine how to play (each AI musician may rely on a central AI musician to determine its output). Stress quality over variety.

Risk: Usability

Problems: Usability is a concern because our target audience is one that does not have significant experience with musical composition. The goal is to provide powerful music composition tools, and have the application easy to use for a novice, which is a challenge.

Evaluation: User studies will provide the most information on this risk. If a new user can get started quickly and make sense of things, our UI is successful. User studies will also provide information about what features are essential (most used), and what features may be superfluous.

Risk Management: In order to promote simplicity, features that will be used every time a user plays something should be implemented first and be emphasized in terms of screen space. These features include:

• Interface / Menu to add musicians to the band
• Stage, where the musicians can be seen, and be given basic instructions
• Playback Controls: The ability to play, stop, record, or pause

Other features should be added, but should not take up as much screen space until they are selected.

Risk: Performance (Speed/Responsiveness)

Problems: Performance is a concern because the application is intended to perform complex operations in real time. A goal of the software is to provide real time controls for the user to control the AI musicians, and the experience will be less enjoyable and confusing if there is there are delays.

Evaluation: Our own testing and user studies will be valuable to gauge whether the music is responsive to the user's adjustments.

Risk Management: This risk ties in with the AI because the AI module is the most likely spot for delays. If the performance is deemed inadequate, more resources should be assigned to implementing and testing the AI module, to find ways to improve efficiency, or AI features that need to be cut in order for the user experience to be smooth.

User Studies

The optimal time to do a user study for the Robot Rock application is after the beta release. In order for the user to provide feedback about the quality of the music and the performance and responsiveness, the user should be able to interact with a small ensemble of fully functioning instruments and go through a complete session, which will not be possible until the beta version. A secondary user study could be done with a paper UI much earlier than the beta study, to gather feedback about what parts of the UI make sense, and where clarifications need to be made.