It seems that we are on the verge of exploring fairer rules and judging methods for the game of football. As various industries incorporate artificial intelligence into their daily operations, football, as a historical sport, is seeing more excitement in a digital age.

Whether in the past or now, referees always play a vital and unique role in games. Their whistles and cards can affect a game's outcome, especially if the two teams are locked in a stalemate. There is nothing more embarrassing in a game than a referee turning a blind eye to an apparent foul or blowing the whistle when everyone believes it is safe and sound.

Despite the game's advancement, it is continually seeking more reasonable and fair rules, and the introduction of AI injects new life into it.

In this article, we invited Mr. Ren Bin, co-founder and chief scientist of artificial intelligence startup CoCoPIE, to share his thoughts on the development and controversies associated with artificial intelligence and how it will affect the experience of football games.

A need born out of controversy

Football games are fast-paced with relatively complex rules, where the referee's authority cannot be provoked, and the penalty often varies from person to person. Thus, controversial decisions are inevitable, and it is common for fans to express their dissatisfaction and question the decisions in public.

We have seen so many cases in the history of football:

Maradona's "Hand of God" helped Argentina win the World Cup in 1986;

Gamal Al-Ghandour's whistle in the 2002 World Cup seemed to send South Korea to the quarterfinals;

Stamford Bridge witnessed the classic scene of Chelsea's hated exit and Michael Ballack's wild chase after the referee in the 2008–09 Champions League season...

Due to various aims and subjective factors, it isn't easy to make accurate decisions under the conditions of only artificial referees. AI referees can better handle common controversial issues, such as goals, out-of-bounds balls, or a player being offside.

Ren Bin believes that although AI football referees are still in the early stages, the development of core technologies is almost done. Typical AI football refereeing technologies may include:

YOLACT-based image segmentation, which can automatically determine whether the ball is in the net or out of bounds;

OpenPose-based human posture recognition to determine whether the player is offside or foul;

YOLO-based target detection to achieve ball tracking;

C3D-based action recognition for providing an accurate basis for handball offense and other penalties.

Mr. Ren mentioned that artificial intelligence has a lot of in-depth research in academia and industry. Algorithms and models are becoming more accurate, so from a technical standpoint alone, this presents good implementation prospects.

As he pointed out, however, industrialization still faces many challenges. We would need a large amount of data and accurate labeled information to improve model training and explore the possibility of real-time sentencing. This will enhance the efficiency of the rulings and simplify the deployment model to reduce the cost of the system. It is essential that more football professionals and technological talents participate in the development process and work closely together to advance the process.

Another possibility of technology empowerment

There have been numerous AI modes incorporated into the game in the past few years, primarily focusing on goal-line technology, Hawk-Eye, and VAR.

Mr. Ren introduced most of the principles of these technologies, which are composed of similar cameras and computer processors. First, multiple high-definition (or even panoramic) cameras capture the game video in real-time and then send the data to the computer. Later, the computer uses AI software to centralize the analysis of the captured data.

While these technologies have undeniably helped referees better handle the game, there are still some limitations. The high cost will stop it from being used in all competitions, but only in major ones such as tournaments. Meanwhile, the data needs to be transferred to the computer for centralized processing. The referee must watch the video and judge with more explicit images, angles, and references, which is time-consuming and less efficient.

CoCoPIE explored another feasible route to address these pain points - extending AI capabilities to the end by carrying AI referees on smartphones and wearable devices to achieve quicker deployment and real-time adjudication.

According to Ren Bin, the goal of CoCoPIE is to free the AI referees that are currently deployed on the server-side and centralize the shooting features and AI processing capabilities on mobile and wearable devices, which would allow AI models to run in real-time on the terminal side, thus reducing hardware and computing costs. The core advantage of this solution is its "ease of use and low cost," which makes it widely used in "a large number of low-level events and daily training sessions for athletes at all levels."

From servers to mobile and wearable devices

The key to making this happen is having powerful mobile AI capabilities. But this is where the challenge lies. Generally, the larger the model, the higher the accuracy rate, but this will also significantly increase the computational volume of the model, which requires more demanding computing power. According to Ren Bin, "it may be possible to achieve real-time computing on a powerful server, but the phone's computing power is limited, so it will be difficult to achieve real-time computing on the phone." 

So what is the definition of "real-time"? According to Ren Bin, "Assuming the video has 30 frames per second, and the AI must complete the analysis within 33 milliseconds for each frame." The processing speed cannot be less than the camera frame rate.

CoCoPIE uses its unique compression and compilation co-processing technology to optimize, compress, and speed up the model while effectively reducing the model and its computation volume to realize real-time processing of a high-precision AI model on a mobile phone without losing accuracy. Ren Bin introduced that the current core technology models, such as OpenPose, YOLO, and C3D, have been able to complete real-time phone analysis functions to obtain information helpful for penalty and training.

With the technology boost, AI functions are liberated from servers to mobile and wearable devices, which gives a broader imagination to the application of AI in football games. In CoCoPIE's mind, there are four categories of people who may get a whole new experience.

Referees: For existing systems, AI capabilities can be optimized and upgraded; for coming systems, three on-field referees can use wearable devices, and the fourth official will use a mobile phone to more easily use AI to improve the accuracy of penalties.

Coaches: They could collect game and training data with official consent and adjust training programs. Through AI data analysis, coaches could quickly access information and optimize tactical cooperation.

Players: AI could assist players in daily training and game review. It could improve the individual's technical and tactical level, optimize training plans, and avoid injuries.

Audiences: Implementing terminal-side AI will bring a qualitative leap to enhancing the viewing experience. Through the AI capabilities on mobile devices, even non-professionals can get increased viewing participation and immersion through scene recognition, intelligent labeling, assisted commentary, instant access to player information, etc.

In Ren Bin's view, CoCoPIE has two goals for the development of AI referees: "On the one hand, it contributes to the popularity of football through low cost and convenience; on the other hand, with the authorization of users, we can also collect game and player movement data, thus contributing to the training of athletes, the selection of outstanding athletes, and the improvement of football."

A clear boundary: AI should never take over the game

Major or minor doubts and criticism always accompany the promotion of new technology. Especially in such a fierce rivalry sport as football, how to grasp the "boundary" of AI referees' intervention will be a vast topic. Is it possible for AI referees to completely replace human referees in future matches? Ren Bin also expressed his opinion about this.

During the early promotion of VAR technology, some suggested that such intervention would interrupt the smoothness of the game and even reduce the spectacle because it was "too precise." "The accepted principle is 'minimum interference, maximum benefit.' We suggest that the same principle be applied to AI referees, i.e., no obvious errors or omissions on the one hand, and accurate references in case of disputes on the other.

The original purpose of AI referees was to provide reference information and assist human referees in making decisions, not to replace them entirely. If AI becomes the game's protagonist, it will make fans and practitioners distrustful. Looking at competitive sports, we would know that the specific functions of AI referees need to be positioned according to the nature of the actual sport.

"If it is only for scoring, AI referees may take the lead in some sports. But in reality, many referees are not just scoring. A good football referee can control the rhythm and promote the atmosphere, and there are sports like figure skating, which involve aesthetics and whose judging is with a subjective appreciation component, so it would be unlikely to be completely done by AI, " said Mr. Ren.

Conclusion

It has been over a decade since the technology of AI was first developed, but as we look back on our lives, we can see how AI has permeated virtually every aspect of our lives, including sports. At the 2022 Winter Olympics, AI referees and coaching systems were in the spotlight more frequently than ever before. Even though there are still controversies about artificial intelligence, we could be at the beginning of an era where this technology will permeate every aspect of our daily lives as it becomes increasingly accepted by the public.

Guest Introduction

Ren Bin is an Assistant Professor in the Department of Computer Science at William & Mary College, the Chief Scientist of CoCoPIE, and the recipient of the NSF CAREER Award and Jeffress Trust Award. His research interests include compilers, parallel computing, software systems, real-time machine learning, and machine learning systems.

In addition, he has accomplished much in parallel graph computing, parallel irregular computing, heterogeneous system parallelism, SIMD compilation optimization, and code generation, with a number of patents on compiler technology. He has produced more than fifty peer-reviewed articles in relevant domains, including ASPLOS, PLDI, PPoPP, SC, FSE, ASE, ICML, CVPR, ECCV, AAAI, DAC, PACT, CGO, ICS, TPAMI, and TPDS. His CGO'13 article was awarded the Best Paper Award, chosen as a SIGPLAN Research Highlight, and nominated for a CACM Research Highlight.

About CoCoPIE

CoCoPIE is a Boston, United States-based AI technology startup founded in 2020. The founding team consists of leading AI professors from Northeastern University, North Carolina State University, and the College of William and Mary, as well as senior architects from Google, Alibaba DAMO Academy, Intel, and other worldwide technology corporations. The company has developed products such as CoCo XGen, CoCo XServices, and CoCo XStore in an effort to popularize real-time AI in IoT and mobile. Currently, tech businesses such as Tencent, Didi, and CoCoPIE are clients of CoCoPIE and utilize their products.