From the well-established Hawk-Eye system in tennis to the Video Assistant Referee in football, we're getting used to technology bringing new perspectives to sports. Now Canon's revolutionary Free Viewpoint Video System is taking tech to the next level, creating a thrilling and immersive new way to view sporting events. It makes it possible to watch the action from any angle, from the highest seat in the stadium to behind the try line or amongst the players, change viewpoint freely while the action is ongoing, and even switch to slow-motion, dramatically transforming the way you experience sports.
Standard stadium broadcast configurations use a combination of fixed cameras, roving operators on the sidelines, cable-suspended units and discreetly placed 'action cameras' to provide video feeds from a number of angles. The Free Viewpoint Video System also makes use of multiple cameras positioned around a stadium, but a much greater number of them, and the high-resolution feeds are combined and converted into 3D spatial data, generating a comprehensive three-dimensional map of the action moment by moment. This is stored on servers and can be subsequently navigated using a 'virtual camera' in order to view the action from any perspective, freely move around as the action is ongoing to get alternative angles on key moments, and put viewers closer to the action than ever before.
Following its debut with a J.LEAGUE Cup Final in 2016, the Free Viewpoint Video System is being put through its paces at Rugby World Cup 2019™. Canon is providing the International Games Broadcast Services (IGBS) with highlight videos of six matches being played at the 72,000-capacity International Stadium Yokohama – the largest stadium in Japan, and the venue for the tournament's semi-finals and final.
The Free Viewpoint Video System has seen numerous enhancements since its introduction. Image quality has increased, while the time it takes to generate the final video has decreased, enabling highlights to be delivered on the same day as the match. Additionally, the area covered by the system has been expanded. Rather than just focusing on the area of in front of the goal, as it did originally, the system now captures the entire pitch, allowing high-definition Free Viewpoint Video content to be generated wherever the action is happening.
To produce such comprehensive coverage of a rugby or football match requires numerous cameras to be placed at a range of angles around the pitch – the number of cameras required varies according to the size of the venue. Despite the amount of equipment involved, the team took just two to three days to install the cameras on pillars all the way around the International Stadium Yokohama.
All the cameras are connected to a network and synchronised, which is no small task – in order for the 3D data to be generated correctly, all of the cameras have to be frame-perfect, which means all of them must be running in sync. Several members of staff are on-site to both monitor the system and produce the finished videos.
As the feeds from all the cameras are captured, the players, ball and other objects are extracted from the image and processed to generate 3D data. Naturally, the huge amount of data that's being created requires some equally prodigious processing, which is achieved with the aid of high-performance servers installed in a production truck located just outside the stadium.
Once the 3D data has been generated, the operator can set or move the position of the virtual camera within the 3D space. The development team has created a controller that not only permits the virtual camera to be positioned at precisely the desired area on the pitch, but also allows match time to be manipulated, speeding up the action or slowing it down.
Being able to move the viewpoint seamlessly while slowing down the action makes it possible to deconstruct and analyse the game-changing points in a match in even greater depth. It's easy to see the potential for the system to be used by coaches and managers to enhance the training of their athletes.
The final stage of the process is to output the video data in a format that can be viewed and shared. During Rugby World Cup 2019™, Canon has provided IGBS with highlight videos from each designated match within an hour of the final whistle.
Currently, the Free Viewpoint Video System is being used to produce curated highlights after a match, but a new phase of development will see the technology being applied to a video replay or a live broadcast. To facilitate this, Canon is ramping up the system to support real-time video generation.
Of course, there is a whole world of sports out there beyond rugby and football. Free Viewpoint Video content could be used to enhance baseball or American football broadcasts, and the system could be applied to sports that take place in venues that are smaller than stadiums, such as basketball and judo. Canon's technology also has an exciting future outside of sport – concerts, commercials, music promos and movies are some of the areas that might benefit from the unique perspectives offered by the Free Viewpoint Video System. Watch this space – from any angle!
• Canon is an official sponsor of Rugby World Cup 2019™, taking place in Japan from 20 September to 2 November.
• TM © Rugby World Cup Limited 2015. All rights reserved.