It’s time for our annual re-tuning of the algorithm(s) and a quick look back at how we’ve done for the past year.  By far, the most feedback we’ve received has been focused on our newly released ranking systems – most specifically Europe.  I take the fact that the rest are uncontroversial as a good thing.  Primarily the comments have been related to our decision to include all teams in our rankings (A-teams, B-teams, C-team, and even “home” teams). I admit that ranking home teams has been a challenge and that, as one commentator put it, they can “...look like junk data that messes up the…usefulness of the ordinal [rankings]”.  However, we were motivated by a desire to be as inclusive and non-judgemental as possible.  In these open ratings systems, there’s no sanctioning body to define the teams and the bouts that should count.  And we certainly don’t want to presume that sort of authority for ourselves.  So in the spirit of derby, we just include anyone and everyone who plays the game.  As the regions mature, we’ve found that enough home teams take on the occasional outsider that we are better able to place them within the global context.  So while it does make our rankings a little more difficult sometimes, we hope you’ll bear with us while we continue to refine them.

Speaking of updates, lets talk about what changes have taken place for the 5 different systems that we now have.  Details for each system are below, but I think the main thing to say is that I think we’ve arrived at a relatively stable place.  There will always be disruptions and anomalies, but the algorithm, as implemented, appears to be pretty robust now.  The new unsanctioned systems still have some teething issues mainly due to bubbles of disconnected teams.  But, I think they are progressing and it’s my hope that as the various regions mature, the quality of the ranking system will improve along with them.

 

WFTDA

I’m pleased to say that the WFTDA ranking system has become very stable now.  It looks like the scaling modifications for new teams that I put in last year are working relatively well.  The biggest update was to the initial ratings for the European teams.  Since 2013 saw a lot more transoceanic bouts, we’ve been able to better align the different regions and we see Europe as a whole moving up 30-50 ratings points.

 

MRDA

The initial MRDA system has always been a little different.  It started with a much smaller data set, but the teams have always been very well connected.  Finally, after this year, we’re seeing the main adjustment parameter (the K-factor) move closer to what we’re seeing in all the other systems.  This makes me feel like the system is settling in.  In addition, the introduction of MRD seemed to cause a lot of disruption in the top 10 rankings due to a big blowout in their first sanctioned game.  This update presented an opportunity to better gauge the appropriate starting point for MRD.  As a result, there’s a fair amount of shuffling of the 7-15 placed teams, but I think it’s more robust overall.

 

Europe

The base parameters of the European system remain relatively stable from those we put in last year.  The main adjustments have had to do with the starting point for new teams.  Now that a few more of the home teams have started to play outside, we’ve had a better time placing them within the broader community.  There’s now only 1 home team that is rated above it’s travel team, so it seems like we’re getting closer to a fully coherent system.

 

North America

There’s really not much to say here.  A few clusters of teams that were previously not connected to the main ranking system have now played out and allowed themselves to be better placed.  There will always be some of these, so we do the best we can.  Other than that, the base parameters really haven’t changed much at all, which means to me that the system on the whole is very stable and robust.

 

Pacific

By far the Pacific system is the least well connected and has had the most problems with credibility – for example, many home teams were higher rated than their travel teams.  Given the relative regional separations, I think we’re going to be struggling with this system for a while.  That said, this recent update did seem to move closer to the parameters that we see in the other systems, so I think we’re moving in the right direction.  In addition, some of the most egregious home team mis-rankings appear to have been corrected.  My hope is that this system is at least somewhat useful, even if only within regional subsets.  I’ll keep working on it every year and hopefully as the region matures, we’ll move towards the consistency that we see with the North American system.


Comments

Could you go into some depth about the "scaling modifications for new teams"?

I was just referring to the discussion in last year's algorithm post about the "VRDL effect" and the response I implemented to minimize the impact of similar situations.