This is Part One of a three-part series in which we analyze the College Football Playoff (CFP) selection process.  In Part One, we address what we believe to be the CFP Selection Committee’s biggest challenge, which is a lack of data to compare teams in proper context.  In Part Two, we identify data that we believe is instructive, and in Part Three, we propose an alternative framework to address the challenge.

Part One  – The Data Problem

Being a member of the CFP selection committee is an enviable job with an unenviable task.  Who wouldn’t enjoy a job where the description includes having a working knowledge of college football and ability to engage in discussion with other passionate observers of the sport? That said, having to select and, thereby, exclude seemingly very accomplished teams is arguably an unenviable task. 

A simple Google search of “CFP committee best vs. most deserving” reveals a plethora of articles which dissect the CFP’s methodology for selecting playoff participants year-to-year.  While the committee’s stated intention has always been “the four best,” it is difficult to prove that standard has been upheld, as its rationale seems to have shifted year-to-year.  In some years, it seems they have selected the four best teams and seeded them in a manner that creates the most compelling bracket.  In other years, it seems they have picked four teams that match the consensus expectation of the media and fans, even if that consensus opinion might not result in picking the four best teams.  Further, it selects teams against a backdrop of competing constituencies with differing priorities ranging from financial to geographic to access and exposure. Many of those factors combined to catalyze playoff expansion.  As we enter an era with twelve teams, and a new television contract on the horizon, we wonder if teams with proven track records of generating compelling ratings for their TV partners will have an unwritten advantage.  We don’t believe this issue has played a major role in the era of the four-team playoff. However, the 12-team CFP format is coming into existence at a time where the sports’ two largest television partners have each aligned themselves with a specific conference. 

Most importantly, even if there was clarity on whether the criteria utilized was “best” or “most deserving”, the methodology of determining such is unclear. The committee has referenced many different measures which include strength of schedule metrics, quality wins, the value of winning a conference championship, and other measures of statistical dominance like game control.  As fans of advanced analytics and statistical analysis, we are encouraged by the fact that the committee almost certainly has access to advanced metrics.  Unfortunately, the committee doesn’t provide transparency nor open-source access to the specific data it employs and how they weigh that data.

That concerns us. Without proper context, data can be reduced to supporting narratives and preconceived notions.

The CFP’s data problem

The CFP has an issue even larger than our concerns about transparency and selection methodology.  The committee has a significant data problem. There is an insufficient sample set of non-conference games between P5 teams.  As a result, much of the objective statistical data that exists about P5 teams lacks proper context. Comparisons of teams from different leagues is based upon a perceived understanding of teams’ strength of schedule, quality wins, and other subjective/qualitative criteria (e.g., “eye” test, program perception, overall talent ranking).  In contrast, other major sports have longer regular season schedules that feature head-to-head matchups and provide teams with an opportunity to affect their relative standings and overall winning percentages.  As a result, teams have more chances to rank order themselves in an objective manner.  The CFP committee has no such luxury. Objective context is a highly elusive concept in the current era of college football because the number of intersectional conference data points is small. 

How small is the data set?

If we use 2022 as a proxy, on a per team basis, teams in P5 conferences averaged less than one out-of-conference regular season game against another P5 team.  If we include Notre Dame’s games against ACC opponents, that number creeps above one.  Obviously, that isn’t a large enough data set to be able to draw meaningful conclusions about the different conferences and how their playoff contending team compare to each other.

The current data is limited…

• Of the 34 regular season OOC games played between P5 conferences/ND in 2022, only seven were between teams who were ranked in the final CFP Top 25.
  • Three of those seven involved Notre Dame.
  • The six games were as follows:  UGA-Oregon, Texas-Alabama, FSU-LSU, USC-Notre Dame, Notre Dame-Clemson, Clemson-South Carolina, and Ohio State-Notre Dame.
• Nine of the thirty P5 matchups were between teams from the SEC and ACC.  On a combined basis, the two conferences accounted for 16 of the 30 games. 
• The two “leading” P5 conferences (SEC and Big 10) had one head-to-head matchup (Auburn vs. Penn State).

…Things have been this way for 25 years

Over the past 25 years, the members of Power 5/BCS Automatic Qualifier conferences have only averaged roughly one non-conference game a season versus other P5/BCS AQ opponents. Those numbers are reflected in the table below.  There are numerous reasons why this has occurred.  The most obvious is that securing a P5 vs P5 non-conference game usually requires a home and home agreement or a willingness to play games at neutral sites. The second is that most programs prefer to have seven home games. Thirdly, three of the P5/BCS AQ conferences added a ninth game to their conference schedule (Big Ten, PAC-12 & Big-12). This removed a slot in schedules that could have been used to create more marquee non-conference games. None of those things were the fault of the CFP Selection Committee.

Where the committee has failed is by creating an imbalanced risk/reward ratio for programs considering scheduling marquee non-conference games. The asymmetric risk of obtaining a quality win compared to a disqualifying loss created incentives for teams’ to schedule non P5 opponents.

Though the SEC and Big 10 have emerged as presumptive leaders of the sport in the expanded CFP-era, there isn't a rich data-set of regular season games between the two conferences. That will improve a bit in the coming years as many of the traditional powers of both conferences have scheduled series with another. Unfortunately, the match-ups are not series of games between similar opponents like the conferences challenges that take place in college basketball.

• In the 25 years of the BCS and CFP era, the SEC and Big 10 have played only 25 regular-season games. 
  • Only eight of those games involved one of the six SEC programs that have won a national title in the last 25 years (Elite SEC program = Alabama, Auburn, Florida, Georgia, LSU & Tennessee). 
  • Over that time, the Big 10 “power” programs (Michigan, OSU, and PSU) faced the SEC seven times.
  • Somewhat miraculously, six of those seven/eight games matched a Big Ten “power” with an SEC “elite” program.
    • Four of those games were instances where both teams were ranked at the time.
      • Including “Power-adjacent” Wisconsin adds two more games between a ranked Elite SEC Program and a ranked B10 “power” to the total.
  • Among elite SEC programs, neither UGA nor Tennessee has played a Big 10 regular season opponent, and among Big 10 powers, Ohio State hasn’t played an SEC team in the regular season.
  • None of the other “Power-adjacent” Big 10 teams (Iowa, Michigan State, and Nebraska) has faced an SEC team in the regular season.

To put these numbers in context…

• On average, elite teams from the Big 10 and SEC face each other once every three years, but only involved two ranked teams once every six years.
• Indiana and Kentucky have faced each other eight times, which is more than all of the SEC “Elite” and B10 “Power” programs combined.

The total number of games between teams from the ACC and SEC is highest because of the four annual rivalry games that take place between the conferences (UGA-GT, UK-UL, UF-FSU, Clemson-SoCar).

Without proper context, what has the committee done?

Simply put, they revert to focusing on the loss column. In the nine-year history of the CFP, there is not a single instance of a P5 team with more losses being selected to play in the CFP (i.e. top 4 final ranking) over another P5 team with less losses…  NOT ONCE!

The CFP Committee is just following tradition.

For several decades, the teams that have played for the national title were the ones from the “major” conferences who had the fewest losses. Only twice since 1975 has a team won a national title with two blemishes on its record (1990 Colorado and 2007 LSU). The table below presents the loss data of national title game participants prior to the CFP starting in 2014. 

In the 22 years prior to the CFP, the title games featured teams with a total of seventeen combined losses. In the sixteen years of the BCS, the two title game participants had a combined thirteen losses. Additionally, the title game featured an unbeaten team in 20 of the 22 seasons, and two unbeaten teams in 8 of those 22 years. Put differently, in roughly one of every three seasons prior to the CFP, having a single loss eliminated a team from national championship consideration. While the CFP has increased teams’ margins of error, we contend that the committee’s attitude about losses hasn’t materially changed. 

2017 was the only time the committee has definitively placed greater consideration on a team’s wins over its losses. That treatment stands as an extreme outlier because Auburn, having defeated two undefeated top-ranked teams in November, made their accomplishments impossible to ignore. Auburn’s case was also buttressed by its competitive performance in a losing effort in September against the number one ranked team at the end of the regular season (Clemson).  Ironically, Clemson “benefitted” from Auburn’s credibility by being the number one overall seed in the CFP, which matched them against Alabama. 

In the week prior to the conference championship games, the committee deemed undefeated Wisconsin as more deserving of a CFP berth than one-loss Alabama, but only as long as it remained undefeated. After Wisconsin’s close defeat to a two-loss Ohio State team, the committee deemed Alabama, which did not play in its conference championship game, as more deserving than one-loss Wisconsin. Further, the fact that Ohio State beat Wisconsin didn’t change the fact that it had lost by 31 points to unranked Iowa and 15 points to third-ranked Oklahoma. Put differently, beating Ohio State shouldn’t have been a huge credibility boost to Wisconsin’s resume. Our point is that if the committee thought Alabama was better than a one-loss Wisconsin team, they could have also reasoned they were better than an undefeated Wisconsin team, which was validated a month later when Alabama won the title.   

As the CFP expands from four to twelve teams, relying on the loss column as the primary selection criteria is unlikely to produce the best twelve teams.  Sorting through that is going to need additional context.  We are encouraged by the fact that some of the premier programs of the SEC and Big Ten are scheduled to meet beginning in 2024, but the volume of games is nowhere near enough. There are still programs among the “elite” tier of both conferences that have failed to schedule bravely.  That said, between the competition among major broadcast partners and the SEC's potential migration to nine conference games, there is risk that many of these games could be canceled. Based on the precedent set by the CFP Selection Committee, cancelling these games would likely reward teams that didn’t make any attempt at scheduling aggressively in the first place.

One of the ways that administrators sold CFP expansion to college football fans was by pitching it as a way to create more marquee non-conference games. The public has been told that a 12-team field means teams will not be penalized as much for suffering a non-conference loss. Many in the media have assumed that the sport’s best teams will have ample incentive to schedule compelling non-conference contests. As SEC Spring Meetings have shown, coaches and athletic directors don’t see it that way. They don’t know what the Playoff Committee is going to do. They have been given no assurances that, given more data, the committee is going to weigh tougher schedules instead of focusing almost solely on the loss column.

The result so far is that administrators are reluctant to schedule more big time games that could result in a loss. So far, the football watching public isn’t getting what it was sold. The committee has only itself to blame… 

In Part Two, we will focus on the data that we think the data that we do have and why the committee should pay attention to it.