The paper presents a novel method for an end-to-end AI analytics application in sports coaching, including both collecting analytic data and deriving coaching insights. First is a ‘proof-of-concept’ for an automated data collection system using AI-based video analytics that builds a sports dataset of observable player and referee game actions relevant for analysis. This data is called ‘tagged data.’ Then, we demonstrate how ‘tagged data’ can derive sports insights relevant for coaching purposes. These models and methods use Rugby Canada data. The insights provided will provide benefits to Rugby Canada in the upcoming Olympics. However, the methods presented here can apply to many other sports.
The automated video collection methodology will demonstrate how a broadcast video feed alone can be sufficient to generate datasets automatically. Automated video dataset collection is an alternate approach to the manual tagging process currently used by sports teams to collect tactical data. The manual process involves an analyst using software with hotkeys to encode actions while watching game videos. Our automated video tagging process will free up an enormous amount of time for an analyst to collect more data and deliver better actionable insights.
The sport analytics models built offer great insights for in-game strategy. Specifically, the models developed for Rugby Canada includes an in-game win/loss model, game win/loss model, player clustering model, and player action combination model. These models provide valuable insights for Rugby Canada;the models can also be transferred and re-trained for application in other sports.
Rugby Canada Partnership
We have partnered with Rugby Canada to build our automated video tagging and advanced sports models. Rugby sevens is a highly unpredictable sport. Games last just 15 minutes and are played in a two or three-day tournament format with seven players from each team on a full-size rugby pitch. The vast amount of space means that, unlike traditional rugby union, sevens are hugely upset-prone, higher scoring, and more fan-friendly. With games being of such limited duration, teams will on average, touch the ball just ten times per match, and the difference between a win and loss often comes down to crucial tiny moments of skill-execution or decision-making. To further complicate matters, there is a high degree of variation in styles of gameplay between competing nations, meaning that there’s no such thing as an ‘average’ sevens match. As such, it is challenging for Rugby Canada to understand which key factors influence game outcomes.
Although sports analytics offers many benefits, there are still challenges holding back sports organizations from realizing the full benefit of a data-driven approach to sports strategy. Often many teams cannot afford a team of analysts. Also, data collection occupies a large amount of time for analysts. As a result, data collection is limited. For example, because data collection for sports analytics is time-consuming, many teams only gather a limited number of key statistics on their players. With advanced deep learning algorithms and computer vision processing techniques, it is possible to train a machine learning model that will automatically tag all actions provided in a video data feed. Automated video tagging frees up time for analysts and provides even more data by collecting more actions for all players. Automation enables collecting new features such as detailed position information of each player, running speed, and other measurements that are not practical for human extraction from a video feed.
Automated Data Collection
(1) Identifying critical features from broadcast video.
YOLOv3, a well-known pre-trained machine learning model used for object detection, was used to identify key features from the broadcast video. This machine learning model can detect players on the field. YOLOv3 was select for its processing speed and high accuracy. YOLOv3 processes each frame of the video to capture each identifiable pre-trained class for ‘person’ and ‘sports ball’. OpenCV image processing records the X-Y coordinates of each ‘person’ and ‘sports ball’ class identified by YOLOv3, as illustrated in the figure above. The X-Y coordinate is translated onto the orthogonal rugby view (as seen in the figure below). However, translating the points from the ‘broadcast frame’ to the orthogonal view requires understanding the x,y coordinates relative to the entire rugby field, i.e. not just the camera frame of reference. Step 2 will explain how to determine the location on the field that the camera is looking at.
(2) Deciphering current zoom and pan angle of the stationary broadcast feed.
We implemented a novel approach to determine the field of view of the stationary camera. A trained custom classifier, using a deep learning neural network, was deployed to predict the field of view of the stationary camera independent of the zoom and pan angle of the broadcast camera. To train the classifier to recognize the location the camera is looking at, requires generating a series of image clips labeled with the x,y coordinates on the field.
To generate a dataset for training a panoramic image covering all angles of the broadcast camera is needed. This panoramic photo is constructed manually by stitching individual frames from the broadcast video together to create a complete panoramic view of the field. The image is then processed manually using ‘photo-shop’ techniques to remove all the players from the field. The result is a 1280x720-pixel panoramic image used to generate image clips for training purposes. We generated 48,343 image clips for training/validation and 4,790 in the test dataset.
(3) Translating broadcast feed into 3D space, applying geometric projection and translation to map the points on orthographic field projection.
Once we understand the bounds that the camera is looking at, we need to map the coordinate on the panorama to the rugby field using a mathematical operation. The mathematical formula reprojects the points from the panoramic view to the orthogonal view, as illustrated below.
(4) Extracting actions from the points on the map orthographic view (Tagging Model)
With points, players, and ball tracking in each frame, rule-based logic determines potential actions to tag. Our implementation included a single rule to tag ‘try automatically.’ This rule used the x,y coordinates of the players to detect crossing the try line and automatically record a ‘try’. Because we had difficulty tracking the ball, we assumed any player crossing the ‘try-line’ was doing so to score a try.
Additional rules can be implemented, and also other classification algorithms could be developed to tag all potential actions; these are areas for future development. Because we were only able to extract a single action from the dataset, we used Rugby Canada’s manually collected dataset as a substitute for automatically collected data to build sports AI models.
Machine Learning Models
In-Game Win-loss model
An in-game win-loss model was created to predict a win or loss at any point during gameplay. The model can predict the win probabilities of team Canada throughout the game and assist in coaching strategies to effect team wins. To train this model, we used play-by-play data from Rugby Canada. The input features included: scoring timeline, number of seconds elapsed in the game, home team score, away team score, and opposing team name. The trained Random-forest model accuracy was 85%.
The model was operationalized as an interactive widget in Jupyter Notebooks that could be used by Rugby Canada analysts.
Game win-loss models
The game win-loss probability models were the primary tool for us to extract KPIs, understand trends, and draw insights on what helps the Canadian Rugby team win games. We started with understanding the features by visualizing game stats based on collective team summary statistics (such as line-break involvement, tackles, and conversions) and results. We used logistics regression and xgboost to predict the probable outcome of each game. We performed hyperparameter tuning with 5-fold cross-validation, using gridsearch and random search to extract the best predictions. To interpret the results and understand feature importance, we used logistic regression as our primary model output. We were able to achieve 85% accuracy, an 86% f1 score.
Based on the model coefficients, we identified six main features that affected team Canada’s game outcome.
We explored odd ratios and log-odds (evidence) to interpret the results. We decided to use evidence to calculate results as we can add up all the variables and understand how much evidence we have. For example, if the action X coefficient is +5.8, this will equal 25.1 of evidence towards action X effecting team wins.
As an expanded step, we added the players and created a new model that combined all player-action features into one dataset. To ensure we give higher weighted importance to players playing against higher-ranked teams. We created an adjusted game stats dataset that multiplied each game stat by the opposition team’s degree of difficulty. The degree of difficulty is based on the opposition teams’ overall win% in rugby world seven tournaments. For example, if an opposition win percentage is 82%; all game statistics against this team will be multiplied by 1.82.
Using this dataset and through feature engineering, we created over 600 variables, but through correlation checks, elimination of low variance features, outliers, high missing value features, and excluding any limited minutes players. We were able to reduce the dataset to 282 features.
The same approach was used to extract predictions and results, as mentioned above. We were able to achieve accuracy and an f1 score of 76%. Again, the logistics regression model was the chosen model to interpret results, and evidence was used to calculate the player-action effect on team wins. Summary statistics by player, visualizations of player-action evidence were extracted to understand each player-action’s overall importance on effecting the game outcome. Based on the total combined evidence, we were able to rank the Canadian rugby players based on their overall results combining positive and negative impacts. These sports analytics models helped understand the role and impact of having a player on the field beyond his number of involvements (tackles, tries scored, etc.).
Player Clustering Model
To understand the similarities between players. We used average per game adjusted action stats, combined with individual players’ roster metrics such as height, weight, and speed. We used unsupervised machine learning to cluster players into groups. Using A Centroid model (K-Means), we checked the manual elbow method to get the lowest inertia score (how far data points within a cluster are far away from each other), equal to 51.4. The highest silhouette score (how far the data points of one cluster are from datapoints of another cluster), equal to 0.2. We were able to cluster the Canadian Rugby team into five main groups. Based on a small sample size of 20 players, the differences in data points between clusters are 20–50%. We were able to derive key insights that can segment players into groups based on speed, defensive, and offensive actions. This clustering model will also help identify where new players are most similar to veteran players and where they fit within the team.
Player Action Combination Model
Although we looked at individual player action impact from our win-loss probability model, we want to understand how players interact with each other in different combinations on the field. Finding out who operates better as pods in matches against varying opposition; will assist the coaching staff in recognizing how interactions between players’ actions affect the game outcome. The player action combination model was built using the same adjusted data used for the win-loss probability model. To fit the data into a market basket analysis approach, we transformed the player-action stats per game into four categories, no action, low, medium, and high. The transformed stats are based on the 40th and 80th percentiles of each action variable. For example, any player action stat higher than zero and lower than the 40th percentile is considered low. The new dataset was divided into a team win and team loss datasets. Each dataset was encoded to fit into the association rules learning algorithm (Apriori) and a minimum support threshold of 6%, we were able to generate a 5000+ set of interactions. We analyzed the players’ action interaction based on how likely a players’ action occurs based on another player’s action (Confidence) , how frequent players’ actions occur together (Support), as well as the degree of association between actions (Lift).
Automated video tagging
The results showed as a ‘proof-of-concept’ that it is possible to use machine learning AI techniques to build a system to extract the location of player positions on the field using broadcast feed without the use of any complex sensors or camera arrays.
We had limited success in extracting tagged actions from the field. More work is needed to refine the AI models, implement a ball tracking model, and improve object tracking between frames.
In-Game Win-loss model
This In-Game win-loss model was found to be of limited use for Rugby Canada because the games are only 15 minutes long. However, this methodology could be applied to longer duration sports to adjust strategy throughout the game. The model can advise coaching staff based on the time remaining in the game to apply a defensive or offensive strategy. An offensive strategy is suggested when the model predicts a loss given the remaining in-game time, scores, and opposing team; otherwise, the model suggests a defensive strategy.
Game win-loss models
Although these models could predict a game outcome, the models were most beneficial as a tool to drive actionable insights for team Rugby Canada. Including, delivering an analysis on the areas and player actions that Rugby Canada needs to focus its efforts on improving. Also, our analysis aided in the ranking process of players and understanding their impact on team performance and hence game outcome. We faced some challenges around the limited number of games we had to feed into the models; the more data we have, the better our analysis would be. Implementing an automated video tagging process will allow us to apply the same analysis to any opposition team with broadcast video available. This automation will also help analyze the game of rugby as a whole and not only from one team’s perspective.
A new methodology that could also be implemented is to include a time series component to analyze sequential actions on team performance.
Player Clustering Model
Clustering players into groups gave Rugby Canada a new perspective on how similar some players are to each other in areas they didn’t think of before. When recruiting new players to join the team, the coaching staff can look at areas of similarities between the potential recruits and the current top players. Recruiting players based on key attributes that can help improve the overall team dynamic and performance. Again, using an automated video tagging process to collect data on opposition teams will expand our 20 players’ analysis. Including players from all over the world will give a new perspective on what makes top players perform more effectively. We will no longer be only assessing the clusters based on one team but a more diverse dataset. To improve the clustering process, we can also use a density model (DBSCAN) to deal better with some noise in our data.
Player Action Combination Model
This model proved beneficial in understanding how players interact with each other in different combinations during the game; and how certain combinations affect games’ outcomes positively or negatively. For example, a negative combination of high confidence actions will alert the coaching team to look at new strategies to correct this negative outcome. This model includes a high-level analysis, without containing opposition teams’ names. We could expand the process by creating separate models by the opposition team to have a more robust view of player action outcomes.
Using an automated tagging process and adding opposition players’ data will tremendously improve this model’s impact. This automation will open a new opportunity to analyze how different players’ combination affects each other from both teams during the game. Further analysis will help team Canada start a diverse variety of sevens players depending on the team they play against and improve the game outcome. This model could also be expanded to examine how different clusters interact with each other during the game and simplify the coaching strategy process.
Sports organizations rely on hard stats, game tapes, and reruns to understand performance, opponents’ strategies, and evaluating the next steps. However, machine learning and analytics advancements have revolutionized the industry in ways that were not possible before. AI is essential for both inside and outside the arena, and any team or major sports league not taking advantage of this will only play inside the box. The system presented in this paper demonstrates that it is possible to construct an advanced sports analytics platform. A platform that uses advanced machine learning techniques to provide actionable insights and recommendations, can influence the Rugby Canada team’s performance and coaching strategies.