Richard Billingsley's Computer Is Beyond Broken

by PB @ BON on Oct 28, 2007 10:13 PM CDT in Football Comment 40 comments

Back in 2003, when USC wound up #1 in the human polls but didn't get a chance to play for the national title because computers didn't rank them highly enough, the BCS formula was changed. More weight was given to the humans ranking these things; the computers were skeptically pushed to the side a bit because they didn't spit out results that people watching these teams were comfortable with.

Not only were computer rankings' weight in the BCS formula devalued, the computer rankers were disallowed from taking margin of victory into account. This struck me as absurd, given that human voters invariably were taking margin of victory into account in their own rankings.

Well, today, in one of the most bizarre years of college football in recent memory, I took a long, hard look at the current BCS Standings. And among all the oddities of the 2007 season - of which there have been many - one thing stood out...

This:

That #16 circled in red? Means Richard Billingsley's computer ranks USC 10th in the country.

Honestly, USC's way, way, totally, without-question out of the national title hunt, so my gripe has nothing to do with wanting to see USC suffer. I could give a shit about the Trojans' fate this year (and truth be told, I think it'd be neat if Texas and USC matched up in the Holday Bowl). My outrage is based upon the fact that one of the computers which counts in this mess of a formula has the Trojans ranked #10 in the country. NUMBER 10!

Based on what, pray tell? USC's schedule to date:

vs Idaho (W, 38-10)
at Nebraska (W, 49-31)
vs Washington State (W, 47-24)
at Washington (W, 27-24)
vs Stanford (L, 23-24)
at Notre Dame (W, 38-0)
at Oregon (L, 17-24)

Strong wins: None
Good wins: None
Bad losses: Home vs Stanford
Billingsley computer whirls, computes: #10

Based. On. What???

As someone who loves the idea of a computer taking a huge input of data and spitting out a reasonably objective ranking of teams, this makes me throw up. More to the point, it makes me wonder what algorythm Richard Billingsley could possibly come up with to spit out this result. Whatever it is? F-cking blows.

I'm honestly more disgusted with this than I am with anything else in this whole sordid process.

--PB--

0 recs | Comment 40 comments

Story-email Email Printer Print

Comments

hell, theres all sorts of crazy shit in there

Michigan is ranked 7th by billingsley as well. The same Michigan that lost to a div 1aa school and the same michigan that got blown the f out by oregon.

by the other Andrew on Oct 28, 2007 10:26 PM CDT reply actions 0 recs

I dont get it...

How in the hell is Michigan ranked above Texas??? Yeah we had a tougher-than-expected time with Nebraska but we still won. They didnt exactly have a blowout game against Minnesota... and they lost to Appalachian State earlier in the season for God's sake!!!
someone explain this to me...

by deysmak on Oct 28, 2007 10:42 PM CDT reply actions 0 recs

billingsly!

Billingsley! Party at Taylor's house NOW Billingsley! Gonna get wasted! Yeah! Billingsley! BILLINGSLEY!

Peter Ullman is my adopted Longhorn. He's a big guy.

by whoopspat on Oct 28, 2007 10:47 PM CDT reply actions 0 recs

Here's an interview

with Billingsly from last year:

http://sports.espn.go.com/espn/page2...

His website doesn't say much about his system (some proprietary license is assumed, I guess; although who would want to steal his shit?). However, I did notice he is from Oklahoma, which may explain his flawed algorithm.

by limnonectes on Oct 28, 2007 10:59 PM CDT reply actions 0 recs

If I remember right...

Billingsley wrote into is formula a restriction that prevents drastic shifts in ranking from week to week, and since USC was #4 in his rankings coming in, they couldn't really be expected to drop lower than 10. That, of course, leaves the obvious question of why they were #4 coming in.

My boy Ryan Palmer is better than you think he is

by aorist9 on Oct 28, 2007 11:01 PM CDT reply actions 0 recs

ESPN Interview

Based on that ESPN interview, different preseason inputs would seem to give different results. This is complete and utter crap. I think that a convergent solution, independent of the initial conditions should be a criterion for selecting computers to be used by the BCS. What a joke.

by Off Tackle Tom on Oct 29, 2007 12:07 AM CDT reply actions 0 recs

Notre Dame

Notre Dame's weekly rank in the Billingsley system:

10, 21, 23, 29, 50, 53, 48, 55

Going into the USC game, he had ND as the 55th best team in the country. That is why USC is so high, they have played a number of teams that were decent last year (ND, Nebraska, etc.)

There are some other pretty bad ones. Oregon comes to mind as does Northwestern, Maryland and Kansas. All teams are here.

by Off Tackle Tom on Oct 29, 2007 12:18 AM CDT reply actions 0 recs

My Favorite Part

is that somehow ND is ranked higher than Mississippi State.

by Vol Navy on Oct 29, 2007 9:03 AM CDT up reply actions 0 recs

Spechless, befuddled,

shoked, awed, eyes glazing over. I think I'll have that drink with lunch after all

marshalld

by duras on Oct 29, 2007 12:02 PM CDT up reply actions 0 recs

I agree that his rating system is

jacked up. On countless occasions, a team's rating will increase after losing to a higher ranked team, assumably regardless of scoring margin. Makes no sense that Central Michigan would increase their rating and move up 2 spots after losing 70-14 to Clemson. I know none of these systems are perfect, which is why they incorporate multiple models, but jeez.

by limnonectes on Oct 29, 2007 12:33 AM CDT reply actions 0 recs

Texas/OU

As a general matter its certainly acceptable to elevate a team based on a loss to highly regarded rival. I certainly thought better of the Horns after OU and the cannonization of Charlie Weis (perhpas a smidgen premature) began after the nailbiter loss to USC.

However, margin of victory has obviouly got to be taken into account.

FWIW, I don't think that Notre Dame actually is the 119th best team in 1-A. Given the strength of schedule they have a strong claim to a top 110 ranking and if they beat the academies, Duke, and Stanford (no gimmes those) the top 85 could beckon.

marshalld

by duras on Oct 29, 2007 12:07 PM CDT up reply actions 0 recs

Doesn't really matter

The BCS formula gets rid of the highest and lowest computer ranking, so this ranking isn't even part of USC's ranking. aka, doesn't really matter.

by ryanlionrah on Oct 29, 2007 1:13 AM CDT reply actions 0 recs

Even if it doesn't count

It is obvious that there's something wrong with it, and it shouldn't be included.

Ill tell you the things that bug me the most:

- I think all computer algorithms should be public, and resources should be provided to athletic departments to evaluate scenarios. Why? because it makes it to where a team understands why the rankings are what they are, and what they reward and punish. Also, it prevents rankings from being adjusted to cater to the usual powerhouses.

- No ranking should be based on preseason polls, because preseason polls are the biggest crock of shit since crocks of shit first came out in the 1600s

- Margin of victory should be included, but capped at some amount, maybe 2 TDs, maybe less. If you win by more than that amount, it doesn't matter by how much.

- If you win a ball game by 3 points or less, chances are, you aren't significantly better than the team you beat, specially in high scoring games. Make it so that if the score difference relative to the total score is particularly low, the game is for all purposes a tie.

I know this last one seems to imply that winning by less than 3 isn't really winning, but the fact of the matter is that if you're trying to find a consistent ranking, it is perfectly valid to assume that teams that had close enough scores really are equally good.

by fathead on Oct 29, 2007 1:40 AM CDT reply actions 0 recs

15 seconds of laughter

I literally sat here laughing out loud for 15 seconds upon reading, "Because preseason polls are the biggest crock of shit since crocks of shit first came out in the 1600s."

--PB--

by PB @ BON on Oct 29, 2007 5:19 AM CDT up reply actions 0 recs

Bring Back the Tied Game

I wouldn't go so far as to count a 3 ~~point win as a "tie," but do think we should do away with "overtime." Yeah, I know how exciting six overtimes can be, but it has four drawbacks: (1) a 1~~point loss in quadruple over-time shouldn't count the same as a 50-point loss in regulation, (2) win or lose, multiple overtimes take a physical and psychological toll on the players' next game, (3) they often run into the next televised game, and (4) it changes the late game dynamics and decision-making process (e.g., if you are down by 7 late in the game and score a TD, you can just go for the 1 PAT, and force OT rather than having to decide whether to go for the tie or the win).

"Only angry people win football games." --DKR

by OBdoc on Oct 29, 2007 8:34 AM CDT up reply actions 0 recs

What are those lines?

I don't know where those horizontal lines came from; I was not trying to delete anything. Please ignore them.

"Only angry people win football games." --DKR

by OBdoc on Oct 29, 2007 8:37 AM CDT up reply actions 0 recs

What is the rationale for the computer rankings?

One might expect some differences in polls done by coaches versus sports writers versus fans because they have different perspectives and biases. It seems to me that the only reason for having computer ratings is to remove subjectivity and give all teams a fair chance regardless of pedigree, area of the country, league, or fan base. If you remove the subjectivity, there should be less variability among the various computer polls, not more variability than in the human polls. Any computer poll significantly discrepant from the other computer polls, should be dropped at the end of the season. I also agree with the suggestion that the computer geeks should be forced to "open their books" if they wish to participate in the BCS annual disaster.

"Only angry people win football games." --DKR

by OBdoc on Oct 29, 2007 8:21 AM CDT reply actions 0 recs

Christ

Your a fountain of good suggestion OBdoc. You can be sure the BCS will assidously ignore them.

marshalld

by duras on Oct 29, 2007 12:11 PM CDT up reply actions 0 recs

I'm going to have to disagree

USC v. Virginia Tech
USC v. Michigan
USC v. Connecticut
USC v. Texas
USC v. Auburn
USC v. Alabama
USC v. South Florida
USC v. Florida
USC v. Wisconsin
USC v. Boisie St.
USC v. Virginia
USC v. Wake Forest
USC v. Clemson

Exactly who else ranked 11-25 do you think USC wouldn't beat? I think the SEC Schools are the only ones with a chance. They are getting screwed in the rankings because of their record. If anything Oklahoma, Kansas, and Missouri are overrated due to their record.

by afat on Oct 29, 2007 8:36 AM CDT reply actions 0 recs

Not how a computer works

With a computer formula - you give it input (data of some sort) and it gives you output (the rankings). I'm asking what sort of input, given USC's accomplishments, could possibly result in an output of USC #10?

The input would have to be something like what you're suggesting - a subjective "I think so and so is a better team." Which makes it just a human power poll.

--PB--

by PB @ BON on Oct 29, 2007 9:08 AM CDT up reply actions 0 recs

Rankings initially and when the game is played...

It's interesting to look at the results over time that Off Tackle Tom linked to above. You can see that the moves in USC's ranking came as a result of wins against Nebraska and Washington.

USC starts season at #6
Win over (then) #18 Nebraska moves USC from #8 to #4
Win over (then) #30 Washington moves USC from #4 to #2
Loss to (then) #70 Stanford moves USC from #2 to #8

Timing has interested me with regards to computer models. When you factor in the rank of the teams or wins-losses of opponents.
--Do you consider the rank/records at the time of the game?
OR
--Do you always factor in rank/records at the time of running the computer model? (Or at week N-1 to get things started...)
(Meaning, does USC get the points for beating a #18 Nebraska in week 4 and those points never get revised if Nebraska is ranked higher/lower later in the season. (It's possible this could create a convergence problem for the model...but I digress.))

Billingsley's model clearly:
-starting rank matters a lot
-the books are closed after each week, meaning the rankings at that time are what matters, not rankings as of the time the model is run

It would be cool to reverse engineer what is going on here and then feed in different initial rankings and see where things end up...

Clearly USC wasn't punished much for losing to the Ducks... For the record he has the Ducks at #18 at the end of week 9, after starting at #48.

Notre Dame
Good to know you can be ranked #55 after a 1-7 start.

Since we care about Texas:
Starts at #21, moves to #13, before dropping to #33 after a loss to then #61 Kansas State.
Losing to then #24 OU didn't affect things at all.

So if KSU is ranked higher than #58 to start the season (loss to Auburn actually moved them up to #55), then Texas has a better rank today.

I'm just glad that all of the computer polls only count for a third.

by horn_in_france on Oct 29, 2007 10:56 AM CDT up reply actions 0 recs

Moving Averages

You do this kind of stuff with stock picking all the time. You use a short exponential moving average and compare it to a longer term exponential moving average:

http://clearstation.etrade.com/educa...

The computers probably use some sort of moving average that stresses back a season or two. They use them because more times then not they are consistant. Have faith that the next 4 games will count exponential more the the 4 he's using from last season as they progress

by afat on Oct 29, 2007 9:25 PM CDT up reply actions 0 recs

What about...

USC v. Stanford at USC? A no-doubter, right?

Thats the great thing about college football(this season anyway) you cant say, "really, well team x vs. any other team and team x wins."

I hate the "parity" and by parity I mean mediocrity in college football right now, as much as anyone, but you really cant bring that argument to the table this year. You cant, with a straight face, say that USC would undoubtably beat anyone of those teams. Thats not even a factor.

My adopted son, Lamarr Houston, once beat up Chuck Norris. Now thats a bad ass!

by kirk1005 on Oct 29, 2007 10:07 AM CDT up reply actions 0 recs

Fighting Crime with Crime

First I took the standard deviation of each computer poll from their collective average ranking.

Results:
Poll Std. Dev. Delta
A&H 2.939387691 0.010293372
RB 5.083306011 2.154211692
CM 2.029778313 -0.899316006
KM 1.811077028 -1.118017292
JS 2.529822128 -0.399272191
PW 3.181194744 0.252100425
Avg. 2.929094319

Delta is difference between a poll's stddev and the average. Negative numbers are good. Zero is average, obviously.

Objective Conclusion:
Either Billingsley is an idiot, or everyone else is.

I've also made a list of how many times each computer's ballot was thrown out for being highest/lowest of the six. This does not include a wash, where someone else agrees with you.

Sorry my numbers are out of line, BZ should show me how to do tables.

Results:
Throwouts
Poll Number %
A&H 7 28
RB 11 44
CM 1 4
KM 5 20
JS 5 20
PW 3 12
Avg. 5.333333333 21.33333333

Percentages are the percentage of times your poll was thrown out, out of the twenty five ranked teams.

Objective Conclusion:
Either Billingsley is an idiot, or everyone else is.

Granted, this is just for this week, but I could go back (and maybe I will, if this catches on) and crunch the numbers for every season with the available data. I think it's pretty obvious, though, that Billingsley does little but make the other five polls worth more, since he gets thrown out nearly half the time.

Growing up, I only fed Jared Norton paper. That's why he eats plays.

by Horn Brain on Oct 29, 2007 11:36 AM CDT reply actions 0 recs

Also

Some of the polls don't count D 1 AA team games at all, so maybe that's why Michigan gets love.

Growing up, I only fed Jared Norton paper. That's why he eats plays.

by Horn Brain on Oct 29, 2007 11:38 AM CDT up reply actions 0 recs

Also, Also

The Colley Matrix is where it's at, ya'll. Check it out. No gimmicky adjustments, no piddling, just a raw mathematical approach that kicks ass and makes pudding. Look at the throwout percentage. Awesome.

Growing up, I only fed Jared Norton paper. That's why he eats plays.

by Horn Brain on Oct 29, 2007 12:16 PM CDT up reply actions 0 recs

Yes, but

the exceedingly high ranking for BC and tOSU. I agree that the methodoligy seems very sensible and far superior to the Billingsley clusterfuck but it still yields these flawed/human results for these two schools.

marshalld

by duras on Oct 29, 2007 12:24 PM CDT up reply actions 0 recs

All but two

computers have BC #1, the others have them at #2. BC just looks good this year. Computers aren't allowed (by humans, mind you) to take into account margin of victory, and Colley actually makes a pretty good argument for excluding it himself. That said, LSU might as well have lost to Kentucky by thirty, because the computers see it the same way. With these rules, being undefeated counts a lot. LSU is still #3 in the CM (ahead of Ohio St.). The other thing about it is that it gets more accurate with every game played, so expect to see these "flawed" results dissipate.

Growing up, I only fed Jared Norton paper. That's why he eats plays.

by Horn Brain on Oct 29, 2007 12:35 PM CDT up reply actions 0 recs

Excellent points.

BTW, your boy's my hero.

marshalld

by duras on Oct 29, 2007 12:40 PM CDT up reply actions 0 recs

stop stealing column ideas from me

i have enough trouble coming up with them as it is...jerk!

by billyzane on Oct 29, 2007 12:47 PM CDT up reply actions 0 recs

How bout dis?

I send you my spreadsheet as a template, then you can just copy paste the data from ESPN, add one column of data and reproduce the results for each week since ESPN started posting the full breakdown of BCS rankings. We split up the years, then we can crunch all the numbers and send it in to the BCS and get Billingsley's sorry ass kicked out of the system. Sound nice?

Growing up, I only fed Jared Norton paper. That's why he eats plays.

by Horn Brain on Oct 29, 2007 12:52 PM CDT up reply actions 0 recs

that's a good idea

i don't have time to geek out about all this stuff right now (it's an elaborate, time-consuming process involving multiple costume changes), but here's what i'm thinking: you're in college right? so you do all the work, write a first draft of a column, I edit it, put my name on it, and then maybe mention you if i remember....just like a real college professor does! sound good?

seriosuly though, get my e-mail from PB and we'll talk about it.

also, i don't want to alarm anyone, but a column's coming later today.

by billyzane on Oct 29, 2007 1:00 PM CDT up reply actions 0 recs

kthxbai

Growing up, I only fed Jared Norton paper. That's why he eats plays.

by Horn Brain on Oct 29, 2007 1:05 PM CDT up reply actions 0 recs

Just for a haha

Check out the stddev deltas. Billingsley is so bad, he pulls the average up to where everyone else is at or below it, in terms of stddev. Snort.

Growing up, I only fed Jared Norton paper. That's why he eats plays.

by Horn Brain on Oct 29, 2007 1:29 PM CDT up reply actions 0 recs

The only thing wrong....

...with the BCS is the C. it's really just BS. The only postitive thing that may come out of Billingsley's deranged rankings is that maybe someone may look at it and say "maybe we could do one of those thingys where we get a lot of teams together and have them play and see which one is the best" (other BCS administrator): "you mean like in March Madness?" yes exactly. "well you think the gym can hold all them people?". seriously though, the BcS has gone on long enough. college football needs a playoff system like OJ needs Johnny Cochran. And I think time is running out. I will ALWAYS watch UT football and I ALWAYS love it, but as far as the rankings go and "post season bowl games" i think it's all a bunch of BcS.

by UTHomeSearch on Oct 29, 2007 11:38 AM CDT reply actions 0 recs

Um, No

PB's flex suggestion is quite excellent and would tend to yield justifiable results in the various situations.

marshalld

by duras on Oct 29, 2007 12:19 PM CDT reply actions 0 recs

for the record...

the Flex System was a proposal I made in a guest column on BON.

intellectual property! proprietary rights! other legal-sounding words! loud noises!

by billyzane on Oct 29, 2007 12:41 PM CDT up reply actions 0 recs

BZ

I'm hurt that you didn't geek out on my standard deviation/throwout percentage comment. Or is that what you're doing right now?

Growing up, I only fed Jared Norton paper. That's why he eats plays.

by Horn Brain on Oct 29, 2007 12:42 PM CDT up reply actions 0 recs

Many compliments

and profound apologies. I've done a good deal of due diligence and have drafted any number of clauses relating to variaous IP rights that a certain party holds, might conceivably hold in the future, could contemplate holding, or might thinking about possibly claiming in the future and so forth. Thus I am fully aware of the magnitutde of my transgression.

The flex idea itself is terrific. I've generally been opposed to a playoff but also, as are most cf fans deeply dissatisfied with the BCS as currently configured. After we won the mnc some friends and I discussed how perfectly the system worked out in that case (incidentally if would have worked as "perefectly" even if USC had won, as pefectly that it is but not nearly so magnficently - from out point of view) but also considered that it had not and was not likely to in the furure function quite as splendidly. We came up with some very crude and impractical variations on diffrent systems could be used in various alternative scenarios. For the record the word "flex" was never part of our conversation and your concept is certainly infinately more sophisticated and refined than anything we considered. Again, many compliments.

marshalld

by duras on Oct 29, 2007 3:30 PM CDT up reply actions 0 recs

Playoffs

Create 16 conferences out of the current D 1-A framework and the champions of each conference participate in a month-long playoff. Would make conference play much more important. Would crown an actual (not mythical) champion.

Smash the computers and their human brothers, the pollsters!

by bfaut86 on Oct 29, 2007 11:07 PM CDT reply actions 0 recs