what's good about stats

From "CMMI: Guidelines for Process Integration and Product Improvement" by Chrissis, Konrad, and Shrum (btw, I would not recommend to ANYONE that they read this unless they have to):
"The purpose of Measurement and Analysis (MA) is to develop and sustain a measurement capability that is used to support management information needs....
Specifying the objectives of M&A such that they are aligned with identified information needs and objectives...
Providing objective results that can be used in making informed decisions, and taking appropriate corrective actions."

Translation: take measurements of something useful that you can do something about.

The best parts about stats are that they're objective and can be used to aid in decision-making, both individual (making better choices with the disc) and team (allocating playing time or determining strategies).

The easiest way to improve with stats is to identify strong negatives (or to use them to prove their existence to doubters). Player A is 1 for 10 on his hucks. Player B throws away 2 break marks a game. The team has not forced a break when playing zone D all year. These things will stand out on their own without any additional analysis needed. The corrective action will require some analysis, but you know that you have a problem. For Player A, you need to decide whether to improve his throws, tweak his decisions, or get him to stop hucking.

The next thing to do is to establish baselines, which won't be the same for everyone, owing to skill level, role, and reward-level of their throws. 88% completion might be borderline acceptable for an aggressive thrower who racks up a lot of goals and big yards, but cause for action for someone who only dumps it or who takes risky but not useful throws.

Now, at this point I'm not expecting anyone to actually establish numerical baselines. Besides being too much work, there are probably too many other confounders that your limits wouldn't be legitimate. But you can, as a captain or coach or even as an individual, establish goals for each player. Be aware, though, that a player might overcompensate on the risk/reward decision in order to improve that completion percentage. (In the earlier stat entry, gambler asked whether anyone played to the stats by making suboptimal decisions. I commented at the time that stat-padding (going for fantasy league stats instead of smart, solid play) was limited to blowout games, but I neglected the other side of suboptimal play.)

Maybe the largest benefit of stat-keeping as we know it is that it forces introspection. Although a player may deceive himself, he is also the one who is most aware of each of his plays. Stats can force a player to look at each of his miscues and reevaluate whether he made a good decision.

Finally, recognize that a Holy Grail of stats is not going to be available for a long time. There will not be any reliable way to roll up all of a player's offensive contributions into a single number. Accept this, and accept that any stat you keep is going to be just one aspect of production that is linked to the other aspects, and you might just find that you can learn something.

The outside world

We’re all Joe Durso, to some extent. In a strong contender for “Article of the Year”, Kenny wrote of him (and his ilk) “in the absence of mainstream acceptance, they cling to scraps of fleeting fame, seek out the company and comfort of those who recognize the significance of their accomplishments, and dream of a day when the world at large will recognize their greatness.” This is me, although I’m going to quibble over parts. I know that my accomplishments are possible only because we’re such a small and minor sport that the Div I athletes don’t bother with it, so I’m not looking for the world to acknowledge my greatness, any more than I want them to acknowledge my Div C modified fast pitch softball championship (yeah, Cougars!). But I do want the world to understand that it is an accomplishment worthy of my time and effort.

I enjoy being a minor celebrity (or, as Corey said on the podcast, “add a few more ‘minor’s in front of that”) in the world of ultimate. I appreciate being around people who feel as strongly about the game as I do. And yes, I’m a little bit afraid of the day where ultimate is not part of who I am but simply part of my past (a very special part, perhaps, but still in the past).

And as for clinging to scraps of fleeting fame, well, yes, there is that aspect, but the larger part is the thrill of the chase. You put in all the work in order to have a shot at, well, greatness. But even for those who seek the limelight, what makes for greatness is not the acknowledgment of the outside world, but the acknowledgment of your inner self that what you are doing is important to you. But the thing is that almost all of us need some confirmation of this from the outside world. It is a rare man who doesn’t care in the least what other people think, and kudos to Mr. Dobyns for having that in him and for being able to move on to other important things in life. (Check back with me five or six years after I retire; maybe things will look differently then.)

Also to the point is that “the outside world” is not the 6 billion others who inhabit the planet, but those who we come into contact with, and those close to us. Kenny’s mom would load up her wagon and drive 10 hours to feed Kenny’s teammates, even when Kenny wasn’t there. His older brother played, married a frisbee chick, and still coaches. Do you think he’d be so dismissive of the outside world if this part of the outside world did not acknowledge the importance of the game to him? My parents and my personal outside world have for the most part embraced my commitment, and that has made all the difference.

Here’s a guess that Kenny or Artie can check on sometime: when Joe Durso won his first national championship, his mom probably told him, “I guess this means you can give up that silly game.”

The perils of stats

I love stats, and always have. When I was 5, I would quiz my mom about stats from the backs of baseball cards. I kept track of Little League stats, wrestling records, golf scores, everything. And in ultimate, I have a record of every tournament since 1992 (and the game scores for those first few years, too). Whenever I have a chance, I've taken individual stats from videos, and was part of a team that recorded every pass back in 1991.

But sometimes I wonder what it's all for. There are a few problems with stats that might just overwhelm their usefulness.

1. Small sample sizes.
2. Other things being equal, ....
3. Field position.
4. False normalization.

The first one is a bane for all sports. You'll hear "so and so is 2 for 18 against this pitcher." This has almost no value in predicting the outcome of their next encounter. Even if you have 100 passes, there is still some chance that a 90% passer will outperform a 95% passer. A lot of studies have shown that the "hot hand" is mostly just a figment of the beholder's imagination.

The problem with the second one is that they're usually not. 3 for 3 is better than 2 for 3, if the passes are the same length, but if the first passer completes dumps that put the disc on a trapped line while the second passer attempts upwind hucks, the second one is more useful. Over time, some of these things wash out, but there will still be outstanding differences resulting from usage patterns. The ideal solution is to compartmentalize the stats as much as possible, so that you separate hucks from dumps, O points from D points, zone from man, strong opponents from weak opponents, but then you get back into problem #1.

Field position (and wind) is just a subset. 40% scoring efficiency might be great if you're starting from your own goal line going upwind, but horrible when starting in the red zone.

False normalization results from the zero-sum aspect of the game. Baseball fielding has a similar problem, in that no matter how bad your defense, you still produce 27 outs in a game. You'll get 15 goals if you win no matter how bad your opponent or how bad you play. Sure, you'll have turnovers, just as you have errors or "defensive efficiency" (percentage of batted balls turned into outs) in baseball, but it makes it impossible to compare players on different teams. Having good teammates will actually hurt your raw stats in either ultimate or baseball (for fielding; batting stats will improve) because you'll have fewer opportunities for yourself.

What to do about this? I'm not sure. It's a bit of a problem, since we need to have a large sample from all levels in order to come up with a good model of how the game works, but no one wants to put in all the effort if they won't be able to do any actionable analysis immediately. But if nobody does anything, then all we're left with is conventional wisdom about what the smart move is in a given situation.

I'd like to see more situational analysis, measuring tradeoffs. For instance, if you have a 20% chance at pulling it OB, is it worth the risk of trying to land it within 10 yards of the sideline? How effective are set plays? What's the difference between having the disc on the line vs in the middle of the field? &c.

More O/D crap

Over at the PuPs, I've been spewing stats about O and D value. Here are a few points I want to make about value:

• Value comes from being better than average or replacement.
  
• If the difference in a particular skill between good and average (or best and worst) doesn't translate into a substantial difference in points or turnovers or runs, then that skill isn't that valuable to the game.
  
• However, it may be true that the skill is a threshold skill, where you have to achieve a certain level just to play the game. For instance, weight for an offensive lineman in the NFL is largely uncorrelated to value, but you have to be at least 270 or 300 lbs to be able to play. I guess we could also call this a fundamental skill in which simple competence gets you 95% of the value of mastery.
  
• We've all been commingling two discussions about value, one being whether an individual offensive player is worth more than an individual defensive player, and the other being whether team offense is more important than team defense. The first issue, I think, is pretty clear for most sports and most players. A top offensive player is involved in a higher percentage of the plays than a top defender. A "shutdown" in ultimate just results in a few seconds wasted (which can of course help create a turnover), while a successful cut clearly increases the odds of scoring. A 20 yard reception from 40 yards out might increase your scoring efficiency from 70% to 80%, but if that cut is shut down by good defense, the efficiency might drop only from 70% to 66.5%. (Here's how I got these numbers. I assume a team is 90% from the goal line and that they lose 5 percentage points every 10 yards. At 20 yards out, they're 80%, and at 40 yards out, they're 70%. For the shutdown, what completion percentage would you expect on a dump where you turn at stall 5? Multiply that by 70% to get your new efficiency. Say it's 95%, then 70% * 95% = 66.5%. You might also need to factor in the yardage loss, pushing it down to maybe 64%.) And you can only cover one player. (Exceptions to this would be a great middle middle in the zone or a great defensive center in basketball who can influence every drive into the lane.)
  
• This is almost an aside, but while there may be an expectation that the offense should score every time, the reality is that offenses score only about half the time, which is also about the rate of scoring in the NBA (fluctuations in points per game are as much a function of pace (possessions per game) as efficiency). I maintain that even for the top of the game, a large percentage of the turnovers have little to do with what is considered "hot D" but is just a matter of being close enough to remove some of the margin of error.
  
• While I don't play on the D team anymore and I follow the disc when I'm on the sideline, I still see an awful lot on the field when I'm playing O.
  

Oh, yeah. Offense rulez!!!