Fort matchmaking (for pickup) and how it works

General Stuff about Armagetron, That doesn't belong anywhere else...
Post Reply
User avatar
raph
Average Program
Posts: 82
Joined: Sun Feb 07, 2010 11:43 pm

Fort matchmaking (for pickup) and how it works

Post by raph »

Hi folks. I recently created a first pass system for automatic fortress matchmaking. Since I hope to share this for us to use within discord for fort pickup, I thought I would shed some light on the current matchmaking process. Just to reiterate, this is a first pass! Feel free to share your thoughts and feedback, but don't expect this to be final or perfect.

To start, I'll explain the current process of matchmaking. Given that we have a list of players (like people added in discord)...

Using a rating system
We need a way of figuring out relative skill level for creating balanced teams. To do this, I'm using armarankings, and for the first pass of fort matchmaking in particular, I'm using the ratings from pickup season 1: https://armarankings.com/rankings?match ... -fortress1. This dataset currently has the richest set of fortress match results (with 583 matches). In the future, we'll migrate to using something more current (since these ratings aren't currently being updated from results) with hopefully just as much rating accuracy.

Bucketing into tiers
Now that we have a rating system to use, we can bucket players into categories for their relative skill level. The reason for doing this rather than strictly using ratings and sorting by that is to add an element of randomness and hopefully more fun to the matches. Players signed up are bucketed into the following four categories based on the percentile of their rating:
s-tier: top 10% of ratings
a-tier: 10-25%
b-tier: 25-60%
c-tier: 60-100%

Now that we've bucketed players, we choose two captains. If we have enough players signed up in s to b-tier, captains are randomly selected from those tiers. Captains should have no more than one tier difference between them (unless for example there's a single s-tier signed up and then only b-tier players).

Generating teams
With players and their associated tiers, we can now generate a list of players ordered by tier (like [s-tier players..., a-tier players..., ...]) which should roughly match the order in which a captain might pick players for their team. I'll quickly add a note that players within tiers are shuffled randomly; the list is not ordered any more granularly than by tier. Now we start going through the list player by player, following a similar picking system than we do in "manual" fort pickup: first player to team A, second and third player to team B, etc (ABBAABB...). The last step here is to associate captains with the appropriate teams, and voila, we have our (hopefully) balanced teams!

That's pretty much it. If you're curious to look at the python code that does this, feel free to have a look here. Shout out to Kronkleberry for doing most of the algorithm design as well as a proof of concept.

Random notes
  • For this system to work from discord, we need your tron auth username when you add since there's currently no association between peoples' discord accounts and their tron usernames.
  • When you !add fortauto <tron auth username>, you could technically pass in any username because of the bullet above. Please don't do this and just use yours :) If you use someone elses, it will mess up the matchmaking. I'll leave it to the community and mods to moderate this until there's a better system in place.
  • If you want to mess around with this by creating sample matches or whatever, you can. The REST API used for matchmaking is publicly accessible and just accepts a JSON list of player usernames. Let me know if you're interested.
User avatar
sinewav
Graphic Artist
Posts: 6413
Joined: Wed Jan 23, 2008 3:37 am
Contact:

Re: Fort matchmaking (for pickup) and how it works

Post by sinewav »

:) Thanks for all the hard work, I'm glad to see this long overdue feature added to games. This is a good start and hopefully it will have an affect on some of the problems we have.
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Re: Fort matchmaking (for pickup) and how it works

Post by Z-Man »

Sounds good to me!
User avatar
sinewav
Graphic Artist
Posts: 6413
Joined: Wed Jan 23, 2008 3:37 am
Contact:

Re: Fort matchmaking (for pickup) and how it works

Post by sinewav »

Just want to say that, in my opinion, this matchmaking has worked amazingly well and has dramatically improved the our setup time, resulting in more matches. The balance feels good and we rarely need to intervene. There is still an issue of new players not being able to sign up and being forced to use other players logins, but hopefully that can be addressed soon. Thanks for continuing to improve our tools! :star:
syllabear
Shutout Match Winner
Posts: 1030
Joined: Fri Oct 13, 2006 1:37 pm
Location: UK/HK

Re: Fort matchmaking (for pickup) and how it works

Post by syllabear »

Can we manually add a new player (or multiple player IDs) to the armarankings for new players?

For instance, can set a custom new GID (such as newplayer@forums) to a low rating and never have it change, or would we need to have a bunch of accounts (newplayer1@forums, newplayer2@forums etc.) and then we can direct people to use these for GID?
The Halley's comet of Armagetron.
ps I'm not tokoyami
User avatar
sinewav
Graphic Artist
Posts: 6413
Joined: Wed Jan 23, 2008 3:37 am
Contact:

Re: Fort matchmaking (for pickup) and how it works

Post by sinewav »

raph123 wrote: Wed Dec 30, 2020 11:15 pm Bucketing into tiers
Now that we have a rating system to use, we can bucket players into categories for their relative skill level:
s-tier: top 10% of ratings
a-tier: 10-25%
b-tier: 25-60%
c-tier: 60-100%
I feel like we've tested the system thoroughly and we should consider tweaking it. The bot works really good when it comes to keeping the top players from winding up on the same team. However, it does a bad job evenly distributing the lowest level players. The four tiers are fun and clever, but probably don't reflect the actual population of players. I feel like it would make more sense if we did a little math using the current rankings as a guide and based the tiers on standard deviations of a bell curve or something. Or, somehow weight the lowest and highest tiers. Or divide them 15/35/35/15%.

Just like having s-tier players on your team makes you OP, having a too many c-tier players is a huge handicap. I feel like we can do better, though I want to emphasize my general happiness with the current system.
User avatar
Kronkleberry
On Lightcycle Grid
Posts: 19
Joined: Fri Apr 16, 2010 3:29 pm

Re: Fort matchmaking (for pickup) and how it works

Post by Kronkleberry »

The original concept I made for the matchmaking algorithm used made buckets based on a bell curve, ideally this would be better and suits our given skill distribution. However, if I remember correctly, raph went with the system we have now because the buckets are less skewed by an influx of new auths into the matchmaking pool. Since we have few players, the buckets can shift with just a few new players, whereas having the bottom percentiles being a larger partition creates more tolerance for that effect.

I think the real limitation in the way of consistently fair matchmaking is the quality of the ratings. If you look through the fort all-time ratings, and scroll down to where the hypothetical B/C tiers would be, you'll see that it's not so cut and dry. Since trueskill partially rewards activity, you'll notice that the players in the 45-100 rank range are from various different skill levels. Separating the true C tier players from the (perhaps less active, but underachieving) A or B tier players who have a low rating, is now pretty hard to do without over-complicating the system. For example Titanoboa and Orly have lower ratings that would put them in the lowest bucket, despite both being strong players/ladle-winners.

When I was first drafting this concept I was using buckets with no data whatsoever, just placing people in tiers of where I thought they should be. The teams, on paper, were consistently balanced (to my perception) and easily to manipulate. So perhaps the easiest fix to balance check teams would require some dubious hardcoding, which opens up a whole can of worms which I'm not sure is worth it.
User avatar
G5
Average Program
Posts: 64
Joined: Mon May 01, 2006 2:24 pm
Location: Stuttgart, Germany

Re: Fort matchmaking (for pickup) and how it works

Post by G5 »

Kronkleberry wrote: Sat Apr 24, 2021 6:36 pm I think the real limitation in the way of consistently fair matchmaking is the quality of the ratings.
Correct. The sample rate (number of matches played) is too low and the algorithm (for calculating the rating) is unsuitable.
User avatar
sinewav
Graphic Artist
Posts: 6413
Joined: Wed Jan 23, 2008 3:37 am
Contact:

Re: Fort matchmaking (for pickup) and how it works

Post by sinewav »

G5 wrote: Wed Apr 28, 2021 10:26 am...the algorithm (for calculating the rating) is unsuitable.
It's such a difficult problem. We're trying to derive individual scores from a team game. And because there is randomness built into the system we'll see fluctuations on top of uncertainty. Still, the system does a fairly good job of making approximate rankings. When I look at them, I feel like they are in order coarsely (the top 10 players are clearly better than the bottom 10), but not finely (you may have about 20 peers of varying ability). Anecdotally, I feel like my ranking is where I'd expect it to be. I don't look at it too often and it doesn't seem to change much, but I'm satisfied with the result.

Because of inherent limitations I think only slight changes can be made and we will have to accept that this is probably the best we can do without a major effort toward a comprehensive system. G5, were you ever in Crazy Tronners? I can't remember. Their internal ranking system was really great, but required effort by the players. Armarankings doesn't require players to do anything (well, except the armarankings team). Maybe in the future we can create a system that better measures individual score? But then we would need an incentive, and there isn't much to gain from it.
User avatar
G5
Average Program
Posts: 64
Joined: Mon May 01, 2006 2:24 pm
Location: Stuttgart, Germany

Re: Fort matchmaking (for pickup) and how it works

Post by G5 »

sinewav wrote: Wed Apr 28, 2021 7:53 pm It's such a difficult problem. We're trying to derive individual scores from a team game.
Yes. It is not possible. Too much luck involved. I see the same thing in StarCraft II where I play a mostly 2v2. But Blizzard does not really care because serious StarCraft II is all about 1v1.
sinewav wrote: Wed Apr 28, 2021 7:53 pm G5, were you ever in Crazy Tronners? I can't remember.
LOL, no.
Post Reply