Barça Innovation Hub (BIHUB) has partnered with Pixellot to develop new systems for automated recording of training sessions and games. This new alliance will allow helping improve technical and tactical analysis by the managers from the different professional sports at Ciutat Esportiva.

The company’s technology allows to film, produce, and distribute sports games automatically and efficiently for fans and the coaching staff. Through the use of AI, it will simulate a camera operator on a high-resolution panoramic image of the pitch.

How does it work?

The panoramic capture uses a wide-angle camera or multiple wide-angle cameras and then stitches these images together. The technology uses advanced auto-tracking algorithms to follow the flow of play within the hi-res panoramic capture. These Artificial Intelligence (AI) decisions need to be made with minimal latency to allow accurate human-like capturing. The latency buffer, in essence, is like a camera operator who looks into the future to understand the flow of the play and then makes a conscious decision about how to shoot it.

Is this a human or a robot? 

To provide the right viewing experience, the technology needs to simulate the human camera operator capture with smooth movements, preferably simulating the movement of a video tripod with a fluid head.

The fact that the same game needs to be filmed differently for the fans and for the coaching staff also allows automation to prove its efficiency. For the coaching staff, it is more important to focus on the tactical view of the game. An analyst looks at the team formation i.e. all the field players excluding the goalkeepers. For the fans watching at home, this angle might be too far to enjoy the game.

It’s all about the algorithms. 

Most automatic capture technologies share the following principles. However, because each sport has its own game logic each game requires completely different algorithms.

• Automatic ball detection – in ball-based sports (e.g. football, basketball, etc.), usually the ball is located at the heart of the action. To capture the action, the algorithm tries to detect and follow the ball.
• Player(s) detection – in more advanced technologies automatic ball detection is complemented by the detection of players. This allows a better understanding of the action and serves as a basis for game state detection. Both the ball and player detections are based on the ability to analyze the images and distinguish between the background and the objects of interest (i.e. the ball and the players).

Why is he just standing there?

One of the challenges in player detection is when a player is standing still for a few seconds. For example, in football, during a free kick, some of the players may not move for a period of about 30 seconds. When this occurs, the algorithm must ensure that the player will not blend into the background.

In addition, the algorithm should be able to distinguish between players who are not active (e.g. waiting in the sidelines) from the active players, even when some of the active players are far from the ball. The algorithm should also be able to identify the referee, who is not part of the gameplay.

Game state detection

Based on the ball and the player detection, the algorithm needs to identify the game state. The game state is the type of play currently happening. For example in football, a corner kick or a penalty kick. In basketball a free throw or a counter-attack, etc. Each game state has its own visual characteristics and by understanding the game state, the algorithm can predict certain behaviours and make smarter decisions about how to best capture the action. Every sport has a long list of different game states, making this a challenging task for an automated system.

Deep learning 

To overcome this challenge, the game state detection may be based on Deep Learning algorithms, which can automatically learn how to identify a corner kick, based on a data set of examples. In the case of deep learning, the system trainer doesn’t have to come up with rules about how to identify the corner kick. The system will automatically generate its own rules and select its own characteristics to identify this specific game state. By taking into account all these parameters, the system can make a decision on how to capture each frame.

Looking ahead to stay ahead

Today’s extraordinary advancement in computing enables the placement of extensive AI and deep learning capabilities in the field. The combination of computer vision, AI, and deep learning provides, for the first time, a viewing experience that is comparable to a human camera operator and, in some respects, supersedes the abilities of a single camera operator. These algorithms will get better and will be able to cover more types of sporting events and isolate different scenarios in the practice sessions.

The core capabilities have been already proved and tested in the pitch. With the help of professional coaches, trainers and analysts at FC Barcelona, the research and development of Pixellot’s features in close collaboration with the BIHUB will add new automation processes to streamline the current workflows, as well as catering the advanced requirements the club has.

The Barça Innovation Hub team