SPRING 1998
First of all, I want to compliment the promotion behind Music Technologies' interactive music testing...not only the ads (which are excellent), but also all the interviews in R&R (cancel my subscription...just kidding!). Bill Moyes really does "walk his talk" when it comes to "pouring it on" to establish dominance in a category. He is a brilliant marketer.
But just because the ads are terrific does NOT mean that all they say about auditorium music testing or the telephone-based interactive testing is on target. Let's take a look...
Interactive music test ads contend that going to a hotel to listen to hundreds of song hooks with a bunch of strangers is not a natural listening environment. I agree.
But, let me ask you this...When does the average person listen to radio on the phone? Or, to be more specific, when was the last time you listened to short song snippets and expressed your opinion by hitting numbers on the keypad?
This is natural??? Come on.
The purpose of any music test is determine how familiar your target listeners are with songs, how much they like or dislike songs, whether they're tired of songs. They are quite capable of telling us when they are in a big ballroom with 100 other people.
If you've ever observed a well-run auditorium music test (I'm sure there are other kinds) you know that respondents take their responsibilities seriously, and are focused on doing the "job" we've asked them to do.
When was the last time you observed someone taking a music test at home?
If your life at home is anything like mine, it's filled with taking care of the baby, balancing the checkbook, cleaning the cat box, checking the e-mail, exercising, recycling, mowing the lawn (or blowing snow off the driveway), taking out the garbage, doing the laundry, checking out the latest scandals on CNBC, checking out the latest scores on ESPN, trying to keep up with the reading, packing for the next trip, and being the "ideal husband" (trying to, anyway).
Isn't the life of a research guy glamorous? It kind of makes you long for a night on the road, conducting an AMT!
But seriously, our lives are busier than ever. It's difficult to concentrate on any one thing at home. Can you trust unsupervised respondents talking to a computer? What if they're tired, distracted, even loaded, and just want to "get this thing over with"?
The most overlooked aspect of music testing, from the radio side, is scaling. No prospective client has ever asked me about the scale we use to test music. Yet poor scaling design can skew the results of a music test. A typical approach -- one others have used for years -- is a single scale that might look like this...
| POOR SCALING DESIGN: |
| 5--LOVE IT |
| 4--LIKE IT |
| 3--NEUTRAL |
| 2--LIKED IT, BUT TIRED OF IT |
| 1--HATE IT |
There's a problem with this kind of scale. It combines two issues -- how much I like a song, and whether I'm tired of it.
Research Lesson #1: Do NOT combine two or more issues in one question! In this case, what if I'm tired of it, but never liked it? What if I still like it, but am tired of it? What if I hate it, and I am tired of it? How do I answer???
Judging by the results of our music tests, which use two separate questions -- one that measures familiarity and burnout, the other appeal -- all of these scenarios are quite common.
Is this is just research guy "nit picking"? NO. Burnout, we find, is higher when measured properly -- that is, when respondents are given all possible choices.
I don't know precisely what scale the interactive test uses. Based on the Music Technologies literature, however, respondents hit one key after each song. That makes it a single-scale test, with a single scale's limitations.
(And, by the way, AMT's that use electronic dials provide a single scale that ignores the crucial issue of burnout entirely!)
But, what about the potential problems of auditorium music tests, even if they use a proper scaling technique? Let's address them...
Music Technologies' literature contends that "one hour of 350 or so hooks in a row, a short break, and then another hour of 350 hooks in a row" can be "very fatiguing." And that fatigue "causes song scores in the last half of the test to be lower."
I totally agree that testing 700 songs in a single auditorium music test session is too much. That's why we don't do it!
In an AMT that last less than two hours and tests 500 or fewer songs, there is NO significant "song order bias" -- that is, a song at the end will not test lower than the same song at the beginning. This has been tested and proven.
In the interactive test, on the other hand, songs at the end may test lower than ones at the start. Sure, you can stop and start the test when you get tired. That's a good idea...after holding the phone to your ear for hundreds of songs, you might be tired, too.
I am NOT saying that fatigue is a problem for the interactive tests. I don't know! And neither do you. Because in the interactive music test, "every participant hears the tested songs in a different order." That does not eliminate the possible effect of fatigue..it merely spreads it out to apply equally to each song.
Music Technologies' ad states that auditorium testing "can't focus recruiting into your station's hot zips." Realistically, that's true. However, it matters only if your core listeners in one zip have significantly different song preferences than core listeners in another. That's highly unlikely.
Music Technologies' criticism of local field service practices is the most serious aspect of its argument against AMTs.
It is true that some field services cut corners in their efforts to gather respondents. They recruit from their database of past respondents and allow them to bring along friends, wives, etc. And, unfortunately, some radio researchers permit or even encourage field services to use these practices to cut costs.
But it doesn't have to be that way! AMT respondents can and should be recruited randomly. When we use a local field service for an AMT, our instructions are clear: (1) Respondents must be randomly recruited. (2) We will not accept and will not pay for referred respondents in a session!
Music Technologies' ads rightly point out the ways some field services "cheat." But it wrongly positions these evils as inherent aspects of auditorium testing. Any research methodology can be executed improperly! The key to "doing it right" is to work with people you trust, apply the highest standards to them, and verify that they are delivering.