The marriage of US Open 2019 to IBM Watson - the Artificial Intelligence engine, that helped conjure the magic of tennis on our screens.

Jottings - Slice of life - 325

The marriage of US Open 2019 to IBM Watson - the Artificial Intelligence engine, that helped conjure the magic of tennis on our screens.

The digital experience of watching the 2019 US open was exemplary. Along with the spectacular tennis demonstrated by top players in the world, the USTA and IBM combination created a technological revolution in collecting, analyzing and presenting the game in all its varied and frenzied splendor that made the event look so intimate and personal to millions of viewers across hundreds of countries. The USTA ( United States Tennis Association), like many other sporting bodies, is a non-profit organization. The only season to rake up big bucks is the two weeks of US open. Broadcasting rights, sale of food and beverages and sporting memorabilia bring in hundreds of millions of dollars; but among the three, It is the undiluted and enhanced user experience on the television sets or mobile devices of tennis fans worldwide that will ensure that this two weeks of tennis extravaganza will remain an event to cherish, bring in more money, and keep coming back to — year on year. The Arthur Ashe or the Louis Armstrong stadium can only accommodate a combined capacity of fewer than forty-thousand people who get to watch their stars close in person; but for the millions watching on their devices, It is the digitized experience that must fill that void. That experience must be close and personal, must recapture the excitement and the aesthetics of the game, and should provide the immersive experience of being in the arena along with others. For years now, IBM, the pioneers of Artificial intelligence and digitization, have been partnering with USTA to transform the Tennis experience for its remote viewers, to enhance the job of editors and commentators by recreating exciting highlights in quick time, and providing interfaces for those present in the stadium to enjoy the sport better. IBM has a separate business unit that works with sporting and the entertainment industry. They have been doing great work collaborating with USTA not only in transforming the visual experience but in helping the spread and nurture of talent in the USA through cutting-edge technological aids. This year, IBM introduced several revolutionary breakthroughs in digitizing the tournament. Powered by IBM Watson engine, built on top of Redhat’s OpenShift, these breakthroughs are examples of how Artificial intelligence, Data collection and analytics, and simple application of sporting rules can combine to create an experience that can be overwhelming for human beings to reproduce in quick time. If we consider the aim of AI is to augment the capacity of Man, and not make Man redundant, then what we see IBM doing in the US Open is indicative of the blessings such synergy between man and technology can bring to the table.

A sporting event faces similar challenges as any other organization to provide great customer experience. For USTA, the product is Tennis, and customer experience for them is engaging their diverse audience of tennis fans with high voltage highlights, answering questions in native languages relating to the tournament, and the ability to interlace past data points to the visual experience of watching the game live or even create artificial simulations. The US open 2019 is powered by Watson - IBM’s pioneering work. In 2012, Watson ( named after the legendary father-son duo who founded and nurtured IBM for the first sixty years of its existence) was designed to beat humans in the game of Jeopardy - A complex linguistic game that was considered beyond the ken of AI till then. The birth pangs of Watson, its eventual victory will be told in another installment, but for this essay it is enough to state that Watson’s ability to “learn” and predict patterns from previous data points is an important feature and one that would play a key role in the visual experience of US open 2019.

In modern days, it is becoming increasingly difficult for people to spend three or four hours watching a sport live. Tennis fans have no time, but even if they do, the patience, understanding, and appreciation required to watch every shot, every strategy unfolding on the court, is dwindling. With television equipped with DVR ( the ability to record) and content producers and sponsors more than willing to offer “Highlights” capturing the key moments of a match, the need to ensure such highlights are true to the game and equally exciting becomes imperative. For a vast majority of viewers, these short capsules are more or less the real game. But the problem is that condensing a fluid work of art or sport is difficult, and more importantly, shouldn’t be biased. When a five hour battle on court between two quality players is condensed to thirty minutes, it better contain the key moments of the game that turn the tide of the match, moments that are not swayed because of the popularity of the players, and it should also be exciting, generate enough adrenaline to keep the viewer riveted, interested and wanting more.

The first task is to have the Watson engine to digest large volumes of data from the previous US Open tournaments. Based on twelve years of data, around 31,000 data points from sources including, audio, videos, text and raw feeds were fed into the system. Watson had to ingest and “learn” about this voluminous data first, through codified rules and weights assigned to different factors, before it can be made to work on real-time data. The Arthur Ashe stadium was specially equipped with hundreds of audio-visual equipment strategically placed around the stadium to capture the game from all possible angles. The visual feeds, the sound of the ball as it leaves the racket, the change of volume in sound made by the ball during the rally, the height of the ball toss, speed and angle of the serves, the intensity of player’s grunting noises, the length of the points, the context of the point — whether it is a game, or set, or a match point, the facial expressions and grimaces of the players before and after a shot, the undulating levels in decibel in the auditory response from the crowds as their loyalties periodically shift from the champion to the underdog and vice versa, the expert insights from the commentators, the climatic conditions on court, the number of medical interventions requested, the gesticulating comments by players on and off games, the controversies — all of this and much more are captured and analyzed based on rules and weights embedded in the AI engine. A human being can process much of this information seamlessly and unconsciously. When asked, we could quickly recount the key moments of the match without much thought. But for a machine to do it, all these symbols, rules and variables that constitute the key moments of play must be explicitly stated, analyzed, derived and extrapolated. IBM AI highlights 2019( for those who watched the live coverage, these highlights appeared frequently as sponsored capsules) used IBM Watson Acoustic Insights to analyze sound, IBM WatsonOpen scale features to iron out known biases, IBM Watson Media to understand the visual feed of points played and the context of it, and IBM Watson’s visual recognition API’s to study Human gestures . It is the exciting combination of these different features that helped Watson engine to curate the highlights of a match within seconds, against the hours it would have taken for Human editors to scan through every frame of a video and audio feed. This year, the curated highlights are further sent to experts for moderation. That’s just the last leg of the race. The bulk of the work has been done by IBM Watson engine. Within seconds, it sifts through millions of video and audio frames to focus on quality moments. The noise produced by millions of data points from the feeds is cleared, and only pristine moments of breathtaking tennis is presented to the experts for their consideration.

Banking on the data collected through its elaborate installations of audio-visual equipment, the second innovation that IBM introduced this year is the IBM Coach advisor.  Top players need to be coached based on factual data pertaining to bodily endurance and personalize training regimens for each player, based on scientifically collected measurements. While the coaches can continue to rely on their experience and intuitive judgments on the potential of a player, they cannot ignore the effect of physical strength and endurance on performance. In a competitive game of tennis that could last several hours, a player runs on an average around six miles back and forth with abrupt starts and stops( which can be even more tiring). Even the most talented of players will succumb to physical fatigue and reach the limits of one's bodily endurance. IBM Coach adviser is designed to work on two measurements - the physiological load and mechanical intensity of the player collectively called the Energy system. The physiological load takes into the account the height, weight, amount of running done by the player to measure the total energy expended during the match; and the Mechanical intensity measures the cumulative weight of acceleration and deceleration on the court and its impact on the body. The product indexes each data point collected during play by factoring in the number of steps run, the number of shots played, the type of shot and its power and so on. With these vital insights, the coaches now have the ability to spot and correlate specific periods of play that correspond to fatigue and lack of physical energy. Though tennis is not an endurance sport, a match like that of Nadal versus Medvedev can turn out to be one. Nearly five hours in court with long rallies, and raw power can test the limits of bodily endurance. As of now, The Coach Advisor is in its trial phase to be used on selected players. However looking ahead at the future of the sport,  players who can survive long points, and can hit the ball with increasing power are likely to dominate the sport. Tools like IBM Coach adviser will become indispensable in training such athletes.

What is interesting in all these innovations is how IBM works closely with domain experts to understand and customize the product for their needs. When IBM Watson shot to fame in 2012, that was IBM’s promise to the world - to implement Watson based systems in healthcare, entertainment, banking, and engineering disciplines. Apart from the features described in the essay, IBM’s natural language processing capability was put to good use by allowing tennis fans to interact with an in-house app in more than two dozen languages. The daily schedule, score-lines, the layout of the tennis courts, the location of shops and restaurants, and much more were available in multiple languages. The USTA has a long term contract with IBM, which gives the software giant to systematically create value for the end-users. Since 1990, IBM has been the USTA partner for AI-based data and security. The recent acquisition of Redhat has only fueled the IBM innovative engine even more. The power of containerization on IBM cloud is now orchestrated by Redhat’s elegant OpenShift offering. In fact, the IBM coach adviser runs on a fully managed Red hat OpenShift instance provided by IBM cloud. It is a stroke of marketing and technical brilliance that within less than a month from the date of formally acquiring Redhat, IBM chose one of their pet platforms - The US Open, to usher and showcase in this important partnership.

On a side note, ESPN won the Emmy award for Outstanding Technical Team Remote for the US Open Tennis Championship. The purpose of the digitization is to allow service providers to consume and present the same stream of data in different formats and for multiple devices. ESPN’s integration with the US Open’s stream of visuals and audio-only enhances the experience for the end-user. ESPN’s ability to tap and package the digital stream, with primly dressed experts and conversationalists zooming in and out of the television screens, sipping coffee and munching nuts and making small talk and sharing predictions — only adds a layer more on top of the sporting wizardry that IBM Watson provides. So, in many ways, it is an end-to-end technology cake with varieties of icing on it.

I hope this essay wasn’t too technical for my general readers, and wherever references were made to technical terms or products, the meaning was clear from the context in which it was written. Sometimes, it is impossible not to use tech jargon, and make the mistake of diluting the subject. I believe, in the tradition of Stephen Jay Gould, one of the most distinguished evolutionary biologists of our generation, that it is possible to write technically without sounding technical. I hope, this essay was enjoyable without proving too hard on the reader.

God bless…

yours in mortality,

Bala