But we were convinced we could make our friend happy, and proposed to program some additions to the sequencer ourselves. We showed him a video tape about research done at MIT by Barry Vercoe and his collaborators on computer accompaniment of a real musician. In this project the computer is given a score and several performances of the piece. With that information it can be "trained" to follow and accompany the musician. Not that we were trying to do that, but we could use the idea to annotate each note in the score with its deviation in the performance, in our case in different tempi. Our friend friendly agreed to perform the Beethoven theme at four different tempi that were musically acceptable to him. We saw again that some notes exhibited a large change when tempo is changed, while others were less influenced by the tempo. But we could now use statistical methods to derive the right timing information for each tempo from this data. Our friend, who started to develop a little bit of suspicion, asked: `Will that solve playing at different tempi then?' We were not quite sure. We definitely had more information now, but the representation of the music was still flat; no structural information was provided. It seemed we could not avoid incorporating some organisation above the note level into our program. Our friend agreed with a smile that was almost saying: `are you stupid or am I?' We got a bit nervous. But after some discussion he agreed to concentrate on the timing of simple structural units like beats and bars only, leaving the note by note details aside for the moment.
Then we remembered Max Mathews working at CCRMA, Stanford University, who does important work in conductor systems (sort of the opposite of what Vercoe is doing). He made a system where one can conduct a sequencer on the beat level, which was just what we needed. The idea of a conductor shook our friend up; that sounded a much better approach than all those statistics we tried to explain to him before. We gave our friend an electronic baton, connected to our sequencer, and asked him to conduct the piece. In the score in the sequencer the beats were marked. The program followed the conductor by aligning each conducted beat with the corresponding mark in the score, and it tracked the tempo indicated by the conductor in doing so. At the high tempo, beating the baton very quickly, it seemed all right, but at the moderate tempo it was impossible to steer the timing deviations within the beat. `It sounds too jumpy,' our friend complained. Since the beat level of the system of Mathews is arbitrary (he calls it `generalised'), we annotated the score with marks at a lower metrical level, which alleviated the problem a bit. But, as our friend was still complaining about the controlability, we eventually ended up by marking each note in the score. This gave complete control at last, though our poor pianist, out of breath by the acrobatics needed to draw each note out of the sequencer by means of a single baton, made a cynical remark about the wonderful invention, which we may have heard of, called a keyboard. We became a bit vapid and proposed to help our conductor by connecting three MIDI batons to the computer, the first two used by us to time the bars and the beats, and the third to be used by our friend to fill in the details, using batons inter-connected with a complex mechanism of wires, to keep the timing at all levels consistent. We fantasized for some time about a whole orchestra of conductors, leading one pianist before them. It was clearly time for a tea break.
Glossary on Timing and Tempo, Patterns and Curves.
Over tea our friend told us about a series of programs on BBC radio, presented by the English conductor Denis Vaughan, on the composer's pulse he used in conducting. The pulse is a hierarchical, composer specific way of timing the beats. This pulse was an idea proposed and actually programmed by someone working in Australia. We went to our library and looked for some references that might tell us more on this composer's pulse. We ran into a collection of articles by Manfred Clynes who had invented the notion. This pulse, coincidentally, had precisely the characteristics we were looking for: hierarchical tempo patterns linked to the metrical structure. It basically entailed a system of automated hierarchical batons, and reduced the complexity further by postulating a fixed pattern for each baton. We took a final sip of our tea and hurried back to the lab and added Clynes' Beethoven 6/8 pulse as tempo changes in the tempo track to our sequencer. It divided the time for each bar into two unequal time intervals for the first and second half-bar and divided each half-bar into 3 unequal parts, one for each beat. With some adjustments here and there, we had our program running in no time. We called in our musical friend from the library to provide some professional judgements. He was definitely not unhappy with the result. [play] `This sounds much better than the things I've heard before,' he said.
`Let's do the first variation, and see how our system performs it,' our friend said, far more optimistic now. He was talking about "our" system. This was a good sign. `This variation is written in an ornamental style,' our friend explained, while we loaded the score of the first variation (Figure 3) into our system and created the tempo track containing the Beethoven pulse for this material. `The metrical and harmonic structure is the same for both theme and the first variation. The only difference is that there are more "ornamental" notes added,' he said in a patronizing tone. When everything was set we played him the result. `Well, this is disappointing,' was his short and decisive answer. After seconds of uncomfortable silence he added, `it lacks the general phrasing and detailed subtlety I think is essential to make it an acceptable performance. The rhythmical materials of the theme and the first variation are different. The sixteenth notes of the variation ask for a different kind of timing than the mainly short-long, short-long, short-long rhythm of the theme. This pulse plays only with the metrical structure, but musical structure has far more to offer than that.' So the composer's pulse could not just be mapped onto any rhythmic material. Furthermore, it only linked timing to the meter, and, as our friend made clear, phrasing and other musical structure was ignored.
Figure 3. Score of the first variation of Nel cor più non mi sento.
That rang a bell. We remembered one of the articles by Neil Todd on a model of rubato, linked to phrase structure. His proposal is very similar to Clynes; it explains timing in terms of a hierarchical structure, but now phrase structure is the basic ingredient. The beat is again the lowest level; below that no timing is modelled. The abundance of mathematical notation in Todd's articles did not put off our amateur mathematician. Quite the contrary. `This, on first sight, will give us a solid basis to work with. What he states here is that, if you remove all the constants from the formula, it is actually quite simple,' M said. `Todd proposes to attach a parabola to each level of the hierarchical phrase structure, and sum their values to calculate the beat length.' He simplified a formula, found an error on the way and finally the model became easy to implement. We were quite conscious of the fact that we were the first really to hear Todd's model [play] (he himself had never listened to it). It did not sound very pleasing because this model was expressed in terms of the phrase structure only (based on the idea of systematically lengthening the end of a phrase in a hierarchical way), and because it lacked all expressive timing below the level of beats.
Longing to show our collaborator that the computer could, in principle, also calculate detailed note-by-note timing, we looked for a model that would provide these. Happily we found masses of rules for those subtle nuances in the articles of Johan Sundberg and his colleagues. These rules formulated simple actions, like inserting a small pause in between two notes or shortening a note. The actions had to be performed if the notes matched a certain pattern, such as constituting a pitch leap or forming part of a run of notes of equal duration. In fact there were so many rule sets proposed in his articles that we got a bit lost in the details, but it has to be said that some rule-cocktails really seemed to work for our piece. Especially if their influence was adjusted to effect a subtle change only, the music gained some liveliness. But because these rules are based on the surface structure of the music only we could predict the judgement of our musical expert by now. And indeed he did not even bother to comment on the artificially produced performances. [play] Instead he kindly reminded us that we might give up looking for a system that enabled us to generate a "musically acceptable" performance, given a score (that is what Clynes, Todd and Sundberg are aiming at), for the simple reason that we already had an "acceptable" performance, namely his own. It was true, the initial aim of our endeavour was to find ways of manipulating the timing in a musically and perceptually plausible way, given a score and a performance. Because the simple representations we had used proved unsuccessful, we had been sidetracked by studying even simpler representations that could at most model a small aspect of our friend's performances. We decided to close the session, look for more details in the literature, and give it another try the next day.