Capstone Two

Capstone Two - AUDL Win Model(s)

Predict game score margin from basic summary statistics

Contents

• Submissions
• Python Environment
• Project Overview

Submissions

1. Project Proposal
2. Data Wrangling Notebook
3. EDA Notebook | Outlier Detection actual assignment is "11.5"
4. Preprocessing Notebook
5. Modeling Notebook | Classification Notebook
6. Project Documentation
   • slides
   • report
   • model metrics

| Data Folder | Graphs Folder |

Python Environment | 3.10, 3.11

show

Packages

• pandas
• numpy
• requests
• scikit-learn
• catboost
• xgboost
• lightgbm
• scipy
• matplotlib
• seaborn
• plotly
• pyarraow
• tqdm
• separate environment used for PyCaret
  • pycaret_requirements for list of packages installed in my venv for Pycaret
  • fresh installation without requirements file recommended

Overview

1. Background, Problem Statement
2. Data Collection
3. Data Cleaning
4. EDA and Feature selection, engineering
5. Modeling
6. Future Work
7. Win Classification alternate problem statement

Background, Problem Statement

• American Ultimate Disc League (AUDL) is a men’s professional Ultimate Frisbee league established 2013 in North America, currently consisting of 24 teams across four regional divisions
• use the most basic game summary statistics to predict the outcome of the game: the winning team and the difference in the two teams’ scores
  • final models should provide a basis for understanding the remaining summary statistics
  • data pipeline should exemplify collecting, cleaning, and exploring AUDL data for future studies.
  • this page focuses on regression for home_margin, see win classification work below

Data Collection

• data collected via AUDL API docs
  • official endpoint for game lists, all games 2011-2023
  • unofficial endpoint for game summaries, example game

Data Cleaning

• some 0 values are real, some indicate missing data
• newer statistics (redzone, hucks) not recorded until midway through 2021
• see report appendix for detailed data checks
• still unsure of: blocks vs turnovers vs incompletions

Data Cleaning graphs

Feature Distributions after data collection

Feature Distributions after data cleaning

Features

• only basic stats used for modeling: throws, completions, blocks, turnovers
• these were used to engineer additional features
  • completion rate, completion rate difference, block-turnover difference
• target features used the final score for definition
  • home margin = home_score - away_score
  • home win = if home_score > away_score

EDA graphs

Feature Distributions, relation to Home Margin

Feature Distributions, relation to Home Win

Feature+Target Correlations

Automated Outlier Detection
see more thresholds and outlier detection based on PCA components in folder

Isolation Forest

Local Outlier Factor

Feature Engineering graphs

completion rate example

completion rate = completions / throws

completion rate difference = home completion rate - away completion rate

Completion Rate Distributions

Away vs Home completion rate, colored by Home Margin

Completion Rate Difference vs Home Margin

see also: block turnover difference

• distributions
  • home/away blocks are added to away/home turnovers to get a total for a given team (block turnover's), rates are the total / number of points played.
• individual vs margin, difference vs margin
  • as shown above, the difference between the two teams correlates more strongly to home margin than either itself

Modeling

• separate studies for each target, separated into two notebooks
• PyCaret used to streamline initial studies, scikit-learn + manual loops were then used to evaluate and train final models
  • final hyperparameter tuning with RandomizedSearch, help with distributions
• Regression for Margin
  • CatBoost, GBR, XGB (tree based boosting), kNN all performed well. Voting Regressor blend of CatBoost+GBR selected for final model
    • residuals analyzed and shown below. two worst errors were games with faulty records that should have been removed during cleaning.
• Binary Classification for Home win
  • kNN, Extra Trees, SGD, CatBoost classifiers all performed well. kNN selected for final model
    • kNN better recall, slightly worse precision. not as good at identifying narrow lossess?
    • results from one more false positive than others, possible faulty record?

Regression Model graphs

Tuned Model Selection

Predicted vs Actual Home Margin

Feature Importance

Future Work

• repeat efforts with team and matchup specific models
  • predict current model's inputs --> predict winning team and score difference?
• data expansion: game events + player data + position data
  • established detailed game record collection + persistence
  • frame many other studies from this collection

Win Classification

see Classification Notebook for details

• results and residuals not discussed here, in general, high accuracy was achieved
• same process for data preparation as used for home_margin regression, different target variable: home_win
  • home_win is True if the home score is greater than the away score at the end of the game
    • slight home-field advantage noted, see report
    • ties should not occur by game design, but two existed in the dataset. they were treated as losses: home_win=False
• tested a wide variety of classifcation models, progressed kNN, SGD, and ExtraTrees classifiers for hyperparameter tuning and final comparison
  • determined best feature selection (SelectKBest) and normalization conditions for each model
    • conditions listed with tuned hyperparameters for top 3 models
  • training and testing scores similar for all models, SGD may be overfitting slightly
• kNN better recall, but worse precision than ExtraTrees and SGD classifiers. kNN higher in aggregate scores: F1, Balanaced Accuracy, ROC-AUC
  • in more words, kNN better at home wins correctly but worse at predicting home losses
• Voting/Stacking used to blend final three models (tuned) and CatBoost classifier (untuned)
  • kNN + CatBoost + ExtraTrees blend (VotingClassifier) may provide improvement over kNN or ExtraTrees alone
  • all blends shown on table are VotingClassifiers with equal model weights

Model	precision	recall	F1	Balanced Accuracy	ROC-AUC
KNeighborsClassifier	0.949	0.943	0.946	0.937	0.937
kNN+Cat+ET	0.954	0.938	0.946	0.938	0.938
ExtraTreesClassifier	0.953	0.926	0.939	0.932	0.932
SGDClassifier	0.953	0.926	0.939	0.932	0.932
kNN+Cat	0.959	0.920	0.939	0.933	0.933
CatBoostClassifier	0.948	0.926	0.937	0.928	0.972
kNN+Cat+SGD	0.948	0.926	0.937	0.928	0.928

Classification graphs

Model Selection 1

Model Selection 2

Top 3 classification models: SGD, kNN, ExtraTrees

tuned model hyperparameters, note model-specific normalization and feature selection

see more: confusion matrices, etc. . .