Monday, August 26, 2013

Stat post for The Huddle

With all the stat posts at ultiworld.com, I thought I'd post an article I wrote for a stats issue of The Huddle which never got published.

Stats

In the early 1990s, my teams (Earth Atomizer and Big Brother) recorded every pass of the season, entering them in a notebook using a shorthand notation during games, and some friends and I would compile them afterwards. Among other things, we found that forehands were thrown away about 50% more frequently than backhands, about 60% of hucks were complete (except for a certain anti-stat hothead who went 4 for 16), and 1.5% of passes were dropped. We did use that first piece of knowledge (coupled with “scouting” observations) about forehands to decide to force forehand most of the time. But what else did we gain for all that time spent?

I have come around to believe that for the time being, the concrete value of tracking individual statistics to predict or to evaluate is doomed by two things, context and sample size. We tried to make one adjustment for context, namely, separating out “tough games” from “chump games”. But then that fed into the second issue, sample size, since we now had fewer games to draw from. And was the line between “tough” and “chump” in the right place? Some of the games were tough because of bad playing conditions, others because we just played badly or exceptionally well in a game that would normally be a blowout, and still others because we had a skeleton crew. Oh, and some teams played zone and forced the handlers to pile up twice as many throws as usual (without, I hope you realize, playing twice as well). But we counted them all equally.

(I should also add that “opportunities” is something that must be accounted for when trying to analyze. But it’s not always as simple as dividing by the number of touches. In the seminal basketball analytics book “Basketball on Paper”, Dean Oliver (now in charge of stats at ESPN) highlighted that player efficiency decreases with increased usage as the players who bear the brunt of the offensive load have to make plays that are closer to the margin. Furthermore, these players will also draw the toughest defenders.)

But we could still tell who was good at completing passes, right? Well, as I like to say at my job where I analyze my company’s engineering performance, it depends. They recorded individual stats on last year’s NexGen tour. I was very excited to get this dataset, because every game was against a quality opponent, almost everyone played almost every game, and each game was a showcase and not just one of many in a long weekend. As it turns out, this dataset too suffers from some confounders such as the first half of the tour beings spent figuring out how to play together and what roles to settle into, but it was still the purest dataset I know of. For the complete tour, turnover percentage* of the players ranged from 3.4% to 12.4%. But for the most part, the guys at the higher end of the turnover range also threw a higher percentage of their passes for goals, while the low-turnover guys didn’t throw as many goals. Here’s the graph for all of them, split out by how often they touched the disc per point: *They didn’t separate out drops from throwaways so we’ll have to use this instead of incompletion rate.

Note also that the high touch players were in the lower left corner. I can think of two explanations for this besides them being conservative handlers. One, an in-bounds pull almost always results in an uncontested completed pass. Two, passes in general in that half of the field are typically easier to complete because the defense has to respect the threat of the long pass, and I think that handlers have a higher percentage of the touches there than they do closer to the endzone, where pass frequency is more evenly distributed. At the other end of the graph, deeps are going to be catching more of their passes near the endzone, resulting in relatively more opportunities for goal throws but also with each completion a little more difficult because of the reduced space. The risk/benefit of a few extra yards changes near the goal line as well. I wrote in “Ultimate Techniques and Tactics”, a book co-authored with Eric Zaslow and published by Human Kinetics and still available through your favorite Internet reseller, that being in the endzone instead of just on the goal line increases your chance of scoring as much as being 10 yards closer elsewhere on the field.

I once set up a simulation of an offense where the players were equally talented (i.e., had the same incompletion rate per yard of throw) but had different roles in the offense and different throw choices. The first thing I noticed is that a particular player sometimes had MVP-level tournaments and sometimes had tournaments where he would have been benched. The more important point, though, was that the players’ stat lines resembled those of real teams such as NexGen, with some players racking up the goals and turnovers while others had lots of touches but few fantasy league stats. This leads me to conclude that much of the difference between the stat lines of any two players is not a difference in effectiveness but simply a matter of taste. (Note that there are still some players who stand out, either good or bad, but you generally don’t need a calculator to know that.) Two equally-efficient players can have drastically different stat lines due not to any difference in skill or on-field decision-making but to the difference in their roles.

So what do these detailed individual stats (at this stage in our history, where we have only the stats of our own team against a wide range of opponents in vastly different environments) bring to the table? Accountability and self-awareness. Lord Kelvin wrote, “If you cannot measure it, you cannot improve it.” Simply being aware of your actual completion percentage on hucks should force you to contemplate whether you are making good choices. I remember going over each of my turnovers in a weekend (with the aid of the stat pad) and being shocked to learn how many of them were simply poor risk/reward decisions, and I was able to eliminate some of those.

Lest you think I’ve given up on stats, I haven’t. But I think the payoff for now would come on analyzing team decisions. The first priority would be to get realistic baselines for performance. I routinely see people write that five turnovers in a game is typical or that drops never happen or that hucks are completed 75% of the time. While there are certainly examples of these happening, I would guess that they aren’t the typical performance. The other area I would like to quantify the value of particular scenarios. For instance, how much harder is it for a team to score off a deep, high pull vs a low pull vs a brick, and how consistent are good pullers at achieving good pulls? Could someone who is an otherwise bad defender still be a good D player simply by virtue of his pulls? On the offensive side, exactly how costly is it to rest one of your top players? How deadly is it to turn it over in your own half of the field? Might the Huck-‘n’-Hope offensive style actually be a reasonable strategy due to the long field left after a turnover? We might have opinions about those now, but until we measure these, we don’t know.

7 comments:

Dennis said...

Ultimate is not really the only sports game in which statistics are of little utility. Even your NexGen graph shows that the upper-left mid-triangle guy (4% tp,10% gp) was rocking -- and your handler under 4% is also remarkably proficient. Yards-per-turnover is truly the important statistic and allows one to compare efficiency across positions. It is very simple: a handler throwing similar yards as another handler but with half the turnovers is significantly more valuable. Stats let you know who are the turnover machines -- and who truly value the disc. And it is NOT the case that without stats people will still notice extreme proficiency. Personality, popularity, attitude, reputation, style, etc., affect perceptions. I’ve seen guys with five turnovers in a game hailed as a hero afterward because of a phenomenal layout catch. Finally, you are likely partly correct about why the handlers are in the lower-left hand corner – but also part of the reason they are throwing a lower percentage of turnovers is that, well, they are handlers. And so almost certainly the best on the team at throwing, breaking-the-mark and valuing the disc.

Steve said...

You might be interested in some team-wide stats I took during a training camp for the Australian Junior Open Team in the lead up to WJUC last year. I kept track of:

- Passes originating in the middle of the field travelling to the middle of the field
- Passes originating in the middle of the field travelling to the side of the field
- Passes originating at the side of the field and travelling to the middle of the field
- Passes originating at the side of the field travelling to the same side of the field.

Middle-middle completion rate was about 98%. Both of the next two were about 95%. The up-the-sideline completion rate was about 50%.

Obviously as players increase in experience the completion rate will go up from 50%, but I think this is still valuable information. Especially when you consider what happens if you want to attempt to string passes together along a sideline.

Martin said...

Yay! Jim is back!

I would completely agree that we aren't at a point where we understand what stats are useful and what stats aren't. It is exciting (at least to me) that we are at the experimental phase with this statistical analysis.

As a coach I think one thing that is overlooked is how to use statistics to impact change on the field. While the raw number of turnovers a person has might have some use, I think it is in the reflection on those turnovers that the utility lies.

A particular number of turnovers may be too many or just fine depending on the conditions, which alters the numbers too much to measure effectively. But by looking at the quality of each turnover (errant throw vs stall 9 bomb) you add in subjectivity to the raw number.

The dependence of the statistics on so many variables makes it difficult to utilize well at an individual level. At a team level it seems more manageable.

Hopefully as we continue this experiment people will find statistics that have more meaning. But at this point I still trust my eyes more than I trust the numbers.

*Note: I would say that stats during scouting has been very useful to me. But then you aren't looking to use stats to change something, only to inform about something.

parinella said...

Steve, that's interesting. What was the force and the stack? Perhaps it's not middle per se but rather than those cuts are with the force while others are breaks. But either way, it highlights that counting only north-south yards and ignoring all lateral movement (and its effect on completion rate) is inadequate.

I plan to do another post on the ultiword stats heat map and its simplified linear version.

Brady said...

We asked the 'better puller vs. better defender' question (and similar questions) a lot when I was helping run Chicago Machine. I was hoping to do some team stats work then can help give us best guesses at what on field situations are best for offenses and defenses. It seems like many others are thinking in a similar fashion, so hopefully we can create some useful theory in the near future.

Steve said...

In this case it was a vertical stack, the force varied between BH and FH. Obviously the vast majority of passes going up-the-sideline are to the open side. This is clearly going to be lower percentage when there is less space to throw and cut into.
The swing from this position will of course often also result in a couple of easy passes.

It is interesting information to have though, and especially powerful to tell a less experienced team to reinforce the need to move the disc horizontally too; rather than completing one low percentage pass, you can complete a higher percentage one which will lead to a string of high percentage ones.

Perhaps a little unrelated, but these sorts of stats can also lead towards D tactics. At least at this less experienced level, given these stats I'd be tempted to funnel the disc towards the sideline when it is in the middle of the field using a straight up force, and once it is there force towards that sideline.

randy said...
This comment has been removed by a blog administrator.