By
Earlier this week, I outlined my haphazard preparation for what turned out to be nine bewilderingly fun games of Jeopardy! (well, the ninth was less fun). Really, what my preparation amounted to was forty years of turning my omnivore's flypaper outward toward the world, and then spending a couple of weekends cramming in whatever extra facts I thought might be most worth having stuck somewhere in my head. Meanwhile, another Jeopardy! contestant was nearing the end of his its training period: roughly four years of ingesting reams of information, constructing guessing and wagering strategies, and playing thousands of practice rounds, many of them against former Jeopardy! champions, with the backing of a team of dozens of engineers, not to mention 16 terabytes of state-of-the-art hardware.
That contestant is, of course, Watson, the machine built by IBM to win the next generation in a line of John Henry-style challenges, this time battling all-time Jeopardy! champs Ken Jennings and Brad Rutter in a three-day match set to air Feb. 14-16. (One hopes that neither Jennings nor Rutter will "die with a buzzer in his hand, lord, lord.") Stephen Baker, a former technology reporter at BusinessWeek and the author of The Numerati, a well-received book on the brave new world of data mining, got an inside seat for the development of Watson, and he was at the taping of Watson's shows last month. His account of the machine and the match, Final Jeopardy, will be released the day after the shows air (a Kindle ebook is already available, which readers can update for free with the final chapter--about the match--beginning on the 17th).
Of course, I've had Jeopardy! on the brain lately, and I was very eager to read Final Jeopardy and talk to Baker, and he was happy to talk too, although even off the record he declined to divulge anything about the results of the big match. The book is a fascinating glimpse into a high-profile technological sprint, and, for those of us who care, an equally interesting look at how to prepare for the game, if you are made of silicon rather than carbon (although carbon-based forms could likely learn a thing or two from the machine). I came away equally impressed by the brainpower and determination that went into building a machine that can play this very human game as well as any human can, and by the remarkable machines we already have in our damp heads, which can still (for a few months yet at least) hold their own against this closet-sized, parallel-processing juggernaut.
You can get a glimpse of how human Watson seems (especially when Jennings starts beating it to the buzzer, and most especially when it says, "Let's finish 'Chicks Dig Me'") in this advance clip of a practice game, and tonight, PBS's Nova has an hour-long documentary on Watson. And for my conversation with Baker about Final Jeopardy, you can listen to the two-part audio below, or read the transcript after the jump.
Amazon: Well, I should start by saying this is unlike any other author interview I've done. This one hits a little closer to home because, as it happens, if the time line of your story had shifted a few months or more, I might well have made a cameo.
Baker: You could have been in the book. I was definitely watching you on TV while I was writing the last bit of the book.
Amazon: Yeah, so it's fascinating to me, having seen some of the inside things, although the IBM side of things I don't know anything about. Let's talk about your main character, Watson. You actually have a footnote at the end about what pronoun you decided to use for it or him, and you decided on "it."
Baker: I decided on "it" because it is a machine, after all. When you're talking to the IBM computer scientists, they usually refer to it as "it." But, you notice, when it gets in the game, all of a sudden people start talking about it as "him" and "he." And David Ferrucci, the chief scientist on the project, occasionally lapses and is heard calling Watson "I."
Amazon: That's the funniest pronoun of all. So a lot of people are familiar with Deep Blue, the chess computer that IBM built a decade or so ago that ended up beating Gary Kasparov and kind of mastering the game. Chess, at least for me, is not an easier game than Jeopardy!, but it is easier for a computer. Why is Jeopardy! a bigger challenge for a machine?
Baker: Because the most complex thing that a machine can do or, at least, one of the most complex things, is to try to decipher human language. We are experts in human language. We're brought up in it, and we understand the nuances and we immediately understand context. You wouldn't believe how painstaking it is to try to teach a computer how to understand the context in a certain sense.
I'll give you an example. There's one category in Jeopardy! that was called Country Club. And so, if you're thinking like a computer or like a lot of people, you think, OK, it's about golf clubs and tennis clubs and things like that. Well, the Country Clubs was actually about the sticks that are used to beat people in various countries, like baton in French. And so, it's clubs in countries. For Watson to understand that and put it in the right context and answer the clues in three seconds is really a great challenge.
Amazon: And that's certainly not where Watson started. IBM decided to take on this project how long ago?
Baker: In '06. One of the IBM managers had been up in a restaurant in upstate New York in '04 and had noticed that the whole place kind of emptied out at 6:30 or 7:00 when Jeopardy! started. Everybody went to the bar to watch the TV and see if Ken Jennings could win his 50th straight game or whatever it was.
IBM was in the process of looking for the next challenge, and he came back with the idea that, maybe, they could build a computer that could beat a human at Jeopardy!
Amazon: And actually go up against Ken Jennings himself.
Baker: Well, if you're going to build a computer that's going to attract attention, and IBM certainly wants attention out of this deal, you've got to beat the best or, at least, play the best.
Amazon: That brings up one question about the idea of attention, and this is a question I had about Deep Blue as well. Is this a stunt to show what IBM can do with a computer, or are there real useful technologies that they can take from this experience and use in their business?
Baker: Well, I would have to say it's both. I mean, it's what they call a grand challenge. A grand challenge, almost by its nature, is a stunt because it goes off and says, hey, we're going to try something that nobody with a computer can do right now and that is also going to bring the focus of the world to our machinery.
And so it has a real promotional aspect to it. But at the same time, teaching a computer to understand really, really complex English--or any other language, for that matter, because it wouldn't be that hard to get Watson other languages--and then to go and come up with answers that it has enough confidence in to actually bet. That's a technology that could have a lot of use in other areas.
Amazon: You mention medical diagnostics and customer service. Those seem like pretty promising routes.
Baker: I think they have their highest hopes for medical diagnostics. In this case, Watson would be sort of a Dr. Watson, almost like House on the TV show, House, where it would be fed a bunch of the different symptoms, and then would go through exhaustive literature--the kinds of literature that no human has the time or energy to read--and look for various correlations of those symptoms and try to come with hypotheses about what this could be.
And so a lot of people when they see Watson on TV, Watson inevitably will make ridiculous mistakes because sometimes it doesn't understand things. People will say, how could you entrust medicine to a machine that doesn't understand things and can screw up so badly, but it's humans working with the machine. Human will say, well, that's ridiculous, and that one is ridiculous, but this one, maybe, might make some sense.
Amazon: Right. Nobody is really talking about just creating this closed loop in which there is a computer that has all these medicines it can put in the IV and it decides alone. It's suggesting possibilities for a human to actually make the decision.
Baker: In that area, it's suggesting things. When it's answering your questions about the software that doesn't work on a telephone, it might be working by itself.
Amazon: The risk is lower.
Baker: The risk is lower, right.
Amazon: So what kind of mind did they end up building? Actually, I want to get back to the idea of artificial intelligence because as much as technology has changed beyond our dreams, that's an area in which it still hasn't met our dreams. To the point where it's almost a joke how hard it is to build an artificial intelligence. To what extent is Watson an artificial intelligence?
Baker: Well, I would say Watson is an artificial intelligence, but some people have other definitions. Watson is doing things with a type of intelligence that we associate with humans. Now, there are critics of Watson who say, "Oh, this is just one huge statistical machine, and it doesn't think the way we do. And it doesn't come anywhere close to the human mind in things like coming up with original ideas or creativity or all of the other areas where humans excel." And all of that is true.
But Watson does do one job that we do exceptionally well, which is find facts and come up with answers. And so there's almost a theological debate in the field about what is AI. I think for some people AI will always be what might come tomorrow, or next year, or next decade, and they'll never really accept what we have that works today as AI.
Amazon: You divide the camps into the idealists and the statisticians. Is that it?
Baker: The pragmatists.
Amazon: The pragmatists, yeah.
Baker: And Watson is definitely a machine that's built by pragmatists. They had to come up with something that could actually appear on a national television broadcast and acquit itself. And so they could not get too involved in some of these theological debates.
Amazon: Right. Speaking of having that computer on the show, one thing I was struck by was how some of the engineers' emotions about having Watson on the show were very similar to mine and being on the show myself. One of the things they were most worried about was embarrassment, and that's--I didn't even think about winning the game. All I cared about was not making a fool of myself. In a sense, that's one of their biggest concerns as well.
Baker: They were concerned about embarrassment, and they actually have a team of researchers at IBM that was called the "Dumb Team." Their job was to actually look at areas where Watson made utterly foolish mistakes and try to eliminate those. But the other fear was that Watson would maybe come out of Jeopardy with negative money and no qualify for Final Jeopardy. That's another humiliating failure scenario that played in their minds.
Amazon: But they didn't go into this cold. I should just say that we're talking on February 2nd. The shows where Watson plays Ken Jennings and Brad Rutter will be on in mid-February, but they were shot last month. And we cannot discuss the results, which I don't know. Before that, Watson had played many, many games against former Jeopardy! players.
Baker: Right. It played close to a hundred games or maybe a little over. And it did well. In the first round, it played Jeopardy! players who had won no more than two games. So, Jeopardy! wasn't giving them access to its superstars. Against these mere mortals, Watson was winning about two out of three matches. But it was showing signs of real vulnerabilities.
So over the summer, they worked more and tried to fine -tune it, and through the fall, it played against people that qualified for the Tournament of Champions, people like yourself. Again, it did pretty well, winning around two out of three matches. But the thing about Jeopardy! and really about any sport--we're heading into the Super Bowl this weekend, and it's the same thing with football. A bad break can change the results of the game. If Watson screws up a Final Jeopardy, it can lose. If Ken Jennings gets really hot on something, he can win. You can predict how this one double game is going to turn out.
Amazon: Right. And you make the point in the book that for the engineers really they've already had their test. All of the games that they played in the fall--that provides a much better statistical base for papers they might want to write out of this. They've proved their point, but now the two games, as you said, are so dependent on chance that for the public those will be the games that get paid attention to, but for the scientists, the games have already been played.
Baker: That's right. Although if Watson wins, they're less likely to make that point.
Amazon: Right.
Baker: That's the kind of point they're preparing if Watson loses.
Amazon: Right. Well, I read this book, obviously, with great interest for the science, but I also read it as a bit of a strategy book because, as you said, I hope to be in the Tournament of Champions and play again, and I'm trying to study right now. What do you think people who want to play Jeopardy! could learn from the way that Watson learned how to play the game?
Baker: Well, I think there's this trend, and I wrote about it in The Numerati, for people to analyze things statistically. Generally the human by the gut way of working Jeopardy! is you look at a bunch of games and you say, well, I better read up on my Shakespeare and I'd better be pretty familiar with Abraham Lincoln and U.S. presidents, and you kind of look at it that way.
What Roger Craig, who you may be facing in a Tournament of Champions, does is analyze statistically what types of categories are used the most, what types of questions. And then he gets a software program and tries to model his own knowledge and show where his gaps are, and where he should study. This is exactly what the IBM people did with this machine.
Amazon: That was really striking to me that you spend the whole book reading about how they prepare a machine to do this, which is different from how you might think a person would prepare. Then you get to the last chapter where you're talking about Roger Craig's strategies, and this is stuff he did before he was on the show at all. And it was so close--he has a background in computers, actually very similar to David Ferrucci, as kind of biology and computing together.
Baker: That's right.
Amazon: And he went at it in a very similar way. I don't know if I'm more or less terrified of playing him after reading that, because of how much he prepared. Either that gives me a lot of room to catch up because I hardly prepared for the game at all, or I know he's already further ahead of me at this point.
Baker: The one comfort you can take is if you do go face to face with Roger, maybe, because of his intense preparation he would beat you in the majority of the games, but you're only going to play him once. And so even you only have a 36% chance of beating him, that's not bad odds. That's like a batting title.
Amazon: Right.
Baker: Be the guy up with the batting title, like Albert Pujols. One thing that you said--the last chapter, but the last chapter hasn't shown up yet.
Amazon: That's true. I wanted to talk about that as much as we can talk about it. I want to ask you about your process in writing the book. The engineers in the story are working on a very tight time line. They've been in negotiations with Sony and Jeopardy! I was surprised how consistent the air dates stayed throughout the process, that they had this late 2010-2011 time that they were aiming for. And that's what they held to. And you had a bit of that deadline pressure yourself in writing the book. What was that like?
Baker: Well, it was really intense. I'd just come out of the magazine world because I worked for years at BusinessWeek, and so I was used to tight deadlines. And this was really writing a book as if it were a series of maybe 12 cover stories. I had to get the first 11 chapters in by November 15th, and I started researching the book in January. So I had to turn the chapters in in tranches, and we went through and edited those. And that went through the whole editing process, and then I went to the final match and wrote that just two weekends ago. And that was like crashing a cover for BusinessWeek. It was kind of an adrenalin high, I've got to say.
Amazon: Did they shoot in Culver City or New York in the IBM offices?
Baker: They shot it at IBM Research. The Jeopardy! people were concerned about how they would have to change the format and make adjustments to the tried-and-true Jeopardy! format to allow for this anomalous player to participate. Because Watson has some differences. It can't handle audio/video clues, for example--the machine is blind and deaf. And it can't do clues in which Alex Trebek has to explain what the category means because it won't understand it. And so the machine is quite different. To sort of isolate Jeopardy! from this extravaganza, they moved it to IBM Research and had it really be an exhibition match.
Amazon: And it's a home game for Watson.
Baker: It's a home game for Watson. I don't think I'm giving away anything by saying that he had a home crowd--it had a home crowd--because the auditorium was full of engineers and IBM execs.
Amazon: Right. Speaking of the end game, as they're preparing Watson to actually play the game, one of my favorite sequences was they're trying to make Watson more human or more like a Jeopardy! player, and they're negotiating between IBM and Jeopardy!. And they trade a finger for an ear. Could you explain what that means?
Baker: The Jeopardy! execs, what they really want--and there was a lot of tension in this story because the Jeopardy! execs want, they definitely want a fair game, and they want a really good show and great entertainment. And the IBM people want a fantastic showcase for this technology and would love to just cream Jennings as far as they're concerned, and they want science.
The Jeopardy! people said, the way that Watson is buzzing now--at this point Watson was just something, an electronic pulse; it was sending an email instead of pushing the buttons--they said, that's not fair. You've got to build some way for Watson to press the physical button, and the IBM people initially were very upset about this. They said, we've got all this scientific data that's based on Watson's performance with the status quo, and now we've got to change it. We built a brain, and you're asking us to turn it into a robot, and this is like Frankenstein. Ferrucci went on and on about that.
Another thing the Jeopardy! people wanted IBM to change was Watson never buzzes too early, and that's something that you as a player must have had to deal with quite a bit. If you buzz too early in Jeopardy!, then you get penalized a quarter of a second, right?
Amazon: Yeah. Getting the timing of the buzzer is, as everybody' who's played--that's the key to the whole game, and I still don't feel like I've mastered it.
Baker: Well, Watson never buzzes too early because it doesn't buzz until the light goes on, but once that light goes on, its reflexes are incredibly fast. And so they were saying, if Watson never is penalized, we should eliminate the penalty for everybody. But if you eliminate the penalty for everybody, Watson was in big trouble. That was a real problem and a sticking point for IBM. They did not eliminate the penalty, but Watson did have to build a finger.
Amazon: And then there's the humanizing thing that the IBM people wanted: an ear, for Watson to be able to hear or understand what the other people had guessed. So he could take a rebound, which is a big part of the game.
Baker: In that example where you have the Country Club category, Watson was thinking about the Marion Cricket Club and all these other clubs, and the answer, which it doesn't get, a human gives, which is the French word for a stick that police hit people with is baton. If Watson doesn't hear that, then it's totally in the dark about the categories, but what IBM got Jeopardy! to do was to provide an electronic feed of the correct answer after the answer came in which is, in a sense, providing Watson with the equivalent of ears, so that it could orient itself and categorize what it might not understand.
Amazon: It could learn while the game is going on.
Baker: That's right.
Amazon: One concern that I think came up during the book is chess in a way hasn't seemed the same since Deep Blue beat Kasparov. Maybe to a chess aficionado it has. Is there any concern that if the computer gets too good at Jeopardy! that somehow Jeopardy! won't be the same anymore?
Baker: Well, I don't think this would be a concern with Jeopardy! because chess is immutable, and Jeopardy! is not. Jeopardy! is created by humans. And so if a machine got really good and dominant at Jeopardy! the way we know it today, then the writers could move Jeopardy! more toward the human strength, which involve more humor, more nuance, more difficult context and move it away from the computer.
I think this is the challenge that we all face in our lives as we live with computers like Watson, which is they are going to handling this kind of work. The kind of work we see Watson doing, computers are going to be doing. And so we as people living on earth in this decade have to figure out where we move in our careers and in our lives so that we can make a living with the brains that we have and the abilities that we have.
Amazon: I think that's always been actually the appeal of Jeopardy! is it's a very human game. The people who win are kind of everyday people, and they have to give those excruciating anecdotes. And I think that is part of the reason people watch it.
Baker: Oh, yeah. I think it will change regardless of what happens in this match because the computer is going to keep getting smarter and smarter in this domain.
Amazon: Right. This kind of hits the sweet spot where the computer seems just competitive enough with the best players that it makes for an interesting game, but that probably won't last six months.
Baker: Well, yeah. That's one of the things. If Watson doesn't win is IBM going to go for a rematch and put this whole team of engineers on fine-tuning this machine for another 12 months? That's the kind of question that IBM faces.
Amazon: Well, I know you can't talk about what happened in the games that were filmed. But I don't know if you can answer this question. Did Watson have to tell an excruciating anecdote about his own life?
Baker: No. I'll tell you what happened. It's a double game, so it's the equivalent of two half-hour games, and they spread them out over three half-hours. And they give a lot of information about Watson, so the audience will know what this computer does and sort of a little bit of its history and how it thinks.
And so the first game, the first half-hour, you only see the first Jeopardy sequence of the game, the first half of a single game. And so Brad and Ken talk but Watson doesn't, but Watson gets a lot of air time through these videos.
The second half-hour, you see the second half of that game. And then, the third half-hour you see it's just like Jeopardy again. You see the full final game.
Amazon: Well, I'm looking forward to watching it. Thanks for talking, Stephen.
Baker: It's my pleasure, Tom. Good luck on the Tournament of Champions.
Amazon: Thanks!
