The Window Question: Why Our Draft Analysis Runs 2012 to 2022

A reader wrote in last week to ask whether the 2012 to 2022 hit-rate window we use for every team scorecard on this site could be widened. Why not run it back to 1980, where our pick database starts? Why not push it forward to capture last year’s first round, or this April’s? More years equals more sample, and bigger samples are usually an analytical virtue.

The answer is that the window is in the right place at both ends. The 2012 floor exists because that is the year publicly available snap counts begin, and snap-weighted retention is the moat that separates this work from the major published retention rankings. The 2022 ceiling exists because draft picks need three full seasons before the 500-snap hit threshold can fairly evaluate them. Move either edge and the methodology breaks.

Both cuts will move on their own as the league plays more football. Neither should move now.

The 2012 floor

Common-sense football journalism reaches back into the 1990s and early 2000s when it ranks general managers. Bill Polian’s Colts run started in 1998. Ozzie Newsome built the Ravens from 1996. Pittsburgh has been anchored by the same general manager since 2010. Common reference for “drafting well” is a multi-decade story, and writers reach for it because it feels accessible.

The problem is what data exists for those years. Pro-Football-Reference does not publish per-game snap counts before 2012. Before 2012, all that exists publicly is roster presence (player on the active roster, on injured reserve, or off the team) and games played. Those measures are useful, but they collapse the entire range between “rotational starter at 1,100 snaps” and “special-teams call-up at 220” into one binary roster bit.

That collapse matters. Josh Dubow’s AP retention piece, which we admire and reference often, used a binary “on the active roster or on IR” snapshot of the 2021 to 2024 classes. He did the best he could with the data he had access to under his deadline. We can do better because we run the window back to 2012 and weight retention by snap share, so a 4,500-snap left tackle who stayed nine years is not equivalent to a special-teams contributor who suited up for eleven games of his fourth-year tenure. Pull the floor back to 2008 and we lose snap-weighting on the four added years. Pull it back to 2002 and most of the dataset becomes roster-bit retention.

If we did that, we would still have a longer window than Dubow. We would not have a different methodology than Dubow. The methodology is the moat.

The 2022 ceiling

The other end of the window is the one fans push back on more, because the player names that should be in the conversation feel obvious. Caleb Williams went first overall in 2024. Drake Maye went third. Marvin Harrison Jr. went fourth. Why would a draft analysis site that calls itself The 53 Report leave them out?

The answer is the 500-snap line. Our hit threshold is 500 regular-season snaps in any single career season, which is roughly one season of meaningful rotational playing time at about 30 snaps per game across a 17-game schedule. League-wide hit rates run 55 to 60 percent against that line. Some rookies clear it. Most do not. Rookie wide receivers struggle to hit 500 offensive snaps. Rookie defensive linemen rotate. Rookie offensive linemen drafted in the third round generally do not start in September. The most common outcome for a player drafted in April is some special teams, some package work, and a longer ramp into year two.

Including 2023 picks in the window means evaluating them after roughly two seasons. Including 2024 picks means evaluating them after one. Both options penalize teams whose recent draft classes are real but young. The Lions’ 2022 class, with Aidan Hutchinson and Amon-Ra St. Brown, looks great today; with one season of evidence, only some picks looked like the eventual contributors they became. A methodology that judged a draft cohort at one season would have systematically overrated teams whose rookies happened to crack the lineup early and underrated teams whose picks were on the development arc football actually requires.

Three completed seasons is the line we treat as fair. Pick a 2022 player, give him 2022, 2023, and 2024 to clear 500 snaps once, and there is a defensible read on whether the pick produced. By that standard, 2022 is the most recent class we can evaluate now. The picks who needed three years finally got their three.

Two cuts of the same data

The reader also asked why we run two windows at all. The hit-rate analysis covers eleven classes from 2012 through 2022. The cohort retention analysis covers five classes from 2017 through 2021. It feels like inconsistency. It is the opposite.

The hit rate asks “did the pick produce.” That is a question about the player, and players need their full development window to answer it. Eleven classes is a wide net, and it lets us read team-level hit rates against a stable league baseline.

The cohort retention asks “did the team keep the player through the second-contract decision.” That is a question about the team, and the team’s answer arrives at a specific calendar moment: the end of the rookie scale contract. Under the current CBA, first-rounders have a fifth-year option; non-first-rounders hit unrestricted free agency at the end of year four. You cannot evaluate a team’s retention of a draft cohort until the cohort has had the chance to leave.

The 2017 class is the right starting point because every 2017 pick is now well past the fifth-year option deadline; some are on their third NFL contract. The 2021 class is the right endpoint because by the 2024 season, every 2021 pick was past the standard four-year rookie window. Their second-contract decisions had been made, one way or another. Drafts after 2021 have not.

We could collapse the two windows into one. Dubow does. Sports Info Solutions does. The result either gives up the development window on hit rate (and inflates teams whose recent rookies happened to start early) or gives up the second-contract window on retention (and inflates teams whose recent classes are still on rookie deals because the team has not had a chance to release them). We hold the two separate because the questions are separate.

How this extends prior work

Two pieces of public-facing draft retention analysis predate ours. We extend both.

Dubow’s piece used the 2021 to 2024 classes and a binary roster-or-IR snapshot. The window is four years. The retention measure is roster presence. The data source is licensed and proprietary. It is the most comprehensive published retention ranking that has appeared in mainstream sports media in the last twelve months. It is also limited by what binary roster snapshots can show. A team that keeps eight nominal backups from a draft class scores higher than a team that keeps four starters and lets four contributors leave for second contracts elsewhere.

Sports Info Solutions published a positional value study in April 2026 that focused on retention of first-round picks across a four-year endpoint starting in 2016. SIS used positional value weighting, which is more sophisticated than Dubow’s flat measure. They also limited the sample to round one. Most NFL drafting happens in rounds two through seven, where the retention story for teams like the Eagles (twelve-for-twelve in round two from 2012 through 2022, by our measure) sits.

Our work runs eleven complete classes for hit rate (2012 through 2022) and five for cohort retention (2017 through 2021). Snap-weighted, all rounds, draft-capital-adjusted using the Jimmy Johnson chart. The full comparison table is at /methodology.

The counterpoint

The strongest pushback on this window is the one the reader pushed: bigger samples are better, and we have more years available than we use. The pick database goes back to 1980. Roster data goes back to 2002. By restricting the canonical analysis to 2012 forward, we are throwing away a lot of football.

That is true. It is also the correct trade. The methodology breaks if we extend it past where the data supports it. We do use the older data for context. Howie Roseman’s Eagles tenure starts before our window opens, and we cite Fletcher Cox (drafted 2012, our window’s first year) and Lane Johnson (2013) when they bear on a piece. Comparable cases exist around the league for general managers whose careers predate our snap data. We do not pretend those drafts did not happen. We do say that comparing them against teams whose entire window we can snap-weight is a different kind of comparison, and that mixing them risks losing the apples-to-apples that holds the work together.

The secondary critique is the dual cut. Two windows are harder to read than one. The honest answer is that one window loses the question. Hit rate and retention are different decisions made at different points in a player’s career, and forcing them onto one window flattens the answer to whichever question gets the wider net. The Bills look like a flat-A team if you only run one window. They look like an A in retention and a B in drafting if you run two. The two-window read tells you what kind of A team Buffalo is. The one-window read does not.

What’s next

The window will widen on its own. By the 2027 season the canonical hit-rate window becomes 2012 to 2023, because the 2023 class will have had three full seasons. By 2028 it becomes 2012 to 2024, and the class with Williams, Maye, Harrison Jr., and the rest of last year’s first round will be in the window for the first time. The cohort retention window rolls on the same schedule: 2018 to 2022 next year, 2019 to 2023 the year after.

Calendar time does the work. We do not need to widen the window intentionally. The dataset gets richer; the methodology stays consistent. Every team we cover today will be re-readable next year against a slightly larger sample, and the year after against one larger again. The Bills’ 64.4 percent hit rate against a league baseline of 58.4 percent is a 2017-to-2022 number; in two years it becomes a 2017-to-2023 number, and the comparison stays like-for-like because the meaning of “hit” does not change.

Until then, the window stays where it is. The next team scorecard ships next week, graded against the same window every team before it has been graded against. That is the point of having a methodology you can defend. It moves when the football is done, not when fans want it to.