Player Ranking System Feasibility

General Stuff about Armagetron, That doesn't belong anywhere else...
Post Reply
User avatar
sinewav
Graphic Artist
Posts: 6413
Joined: Wed Jan 23, 2008 3:37 am
Contact:

Player Ranking System Feasibility

Post by sinewav »

With the exception of the Kings of the Grid leader-board, there has never been a usable player ranking system in Armagetron. Would player rankings benefit Arma like other games? Is such a thing possible, and if so, what would it look like? I propose that player ranking in Arma is possible and would have benefit. I also have an idea about what it might look like and welcome everyone to explore this idea with me.

The benefits of ranking

A lot has changed in online gaming since the first lines of Armagetron's were written. It goes without saying that eSports is a huge industry and professional gamers make more money in a couple than most of us ever will in our lifetimes. Just like traditional sports, gamers have to worry about their stats, and those stats vary wildly depending on the game. Kill/Death ratio is a common one, as are Win/Loss (or ELO) and various time-based metrics. It's assumed every modern game released with a massive online multiplayer component will collect some kind of in-game stats and use them for match-making. Obviously, these systems exist because players demand games that are fair and challenging.

Tronners also demand fair and challenging games. #pickup culture exists to have fair and challenging games. Ladle went through years of small and large changes on the road to improvement. However, most of those changes were at the team level and did not affect individual players. Today there are few teams. Clans have mostly gone extinct (for better or worse) and players are largely independent. And yet, we still have room for improvement. Ladle matches are more lopsided than ever and even #pickup games are becoming hard to balance with our limited pool of players. If there were ever a time to experiment with player rankings, this is it.

Player rankings could be used to make fair teams, not just in Ladle, but in all Armagetron tournaments. I am not suggesting that matchmaking is done completely by algorithm, but having ranks could inform team-making decisions. One possible scenario is to break players into tiers, then have team captains pick equally from every tier. Another scenario could use a point system where each team gets a budget to spend. Do you want to have three top level players on your Ladle team? Sure! But you either play with fewer players on your team, or you're forced to spend the rest of your budget on low level players to make a team of six. Of course, there could be any combination of manual picking, automated picking, and randomness. We decide -- but we can't do any of it without rankings.

How to rank players?

Armagetron doesn't collect stats in a meaningful way. Anything we currently use gets parsed from log files. Arma is also notorious for giving points away to other players, so even if we did track K/D it probably wouldn't be useful. The problem with in-game stats is that they don't create an accurate picture of a player's skill. This is more true of Armagetron than other games like shooters and racing. The ability to read your environment and make smart/skillful moves cannot be understood by the game engine. What I'm getting at is, player value is subjective.

We can recognize good and bad players. We can recognize ourselves in relation to other players. We all have an internal raking system of the players we encounter in Armagetron. So, let's come up with a way to pool our internal ranking systems into a tangible one we can use as a game-wide reference!

My suggestion is to create a website that allows us to rank our peers. In my mind I imagine a domain that requires a login and could also be used for authentication. After logging in, users are presented with a choice to make between two players:

Is player 1 worse|equal|better than player 2?

The system continually generates random match-ups and records the scores. If enough people sign up and rank their peers, over time we would have a pretty good picture of who the best and worst players are in the community. Notice that I said "best" and not "skillful." Since these rankings are subjective, it leaves room for evaluation on our own personal criteria. I might recognize someone as having incredible skill, but maybe their situational awareness isn't so great and maybe they are kind of unpleasant to play with. I might want to give a high rank to a player with exceptional wisdom who doesn't have the best reaction time. Accuracy isn't the goal, but rather we should focus on creating a useful tool that has a nice distribution of player scores.

Some considerations

Ideally, the system should stay intentionally vague. What I don't want is players reduced to baseball card stats. Ideally, the actual ranking score should remain a mystery, even to the database owner (if possible, I haven't spent much time thinking about how to do this programmatically). I think ranks should be displayed in low resolution, just a few tiers. I think players would be happy to confirm they share a tier with a dozen other players since they probably know this intuitively. There are enough egos in this game that we don't need someone parading around a badge with a number.

I have not thought deeply about how the math would work in a system like this. Math isn't my strong point. I also have not thought much about how to prevent abuse, though with a community this small I don't think abuse would be too much of a problem. Tournaments and #pickups have been pretty much free of drama for a while -- isn't that nice!?

:stubble: Thanks for reading and I welcome your thoughts.
Rnoodles
On Lightcycle Grid
Posts: 21
Joined: Thu May 30, 2019 10:39 pm

Re: Player Ranking System Feasibility

Post by Rnoodles »

I love where this is going, i'm sure there are many tronners out there who play in ranked play lists on other games and they would all agree having at least a semi accurate ranking system would be better then non at all. This is made obvious when i see players wanting to go to Tweety's Sumo for example because it logs matches won, i think the leader board in yellow sub servers has also helped keep it active during the tron dark ages haha.

You mentioned that the client may not be accurate in recording data, i think i know what you mean. Like when someone in sumo for example dies on your wall and another player gets the kill? This is a bit annoying but for the most part it appears to get it right more often then not. If we were to log a bunch of stats that may not be perfect, we would at least get a much more detailed ranking system then we currently have.

If we could log average points per game, total matches won, total matches played, kills/death ratio as a base line this would be super cool. Then we could make it more accurate by adding in scores from ranked matches. These matches could include pickups, sumo bar league, ranked fort matches, tournaments ect. If this ranking sytem could be visible to other players and one of the ways you can get a higher overall ranking is do well in ranked matches such as tournaments, this would encourage newer players to get more involved.

More specific data could be logged (if possible) for certain game types like fort and sumo? Like kills in enemy fortress, kills in home team fortress, fortress zone captured ect.
Then to make it even more accurate we could also imply what you mentioned, a community player rank.
If we solely base it on a community ranking, we could have issues with players not really knowing the game in depth, ranking players. Or players who dont get along/want to troll, down ranking players out of spite. At the moment the community is pretty legit and i dont think that would happen too often but if we can somehow combine all that together, it would be the closest thing to a proper elo system that other competitive games out there have. Having your rank linked to your login would be best?
Last edited by Rnoodles on Thu Sep 10, 2020 7:43 am, edited 1 time in total.
User avatar
Kronkleberry
On Lightcycle Grid
Posts: 19
Joined: Fri Apr 16, 2010 3:29 pm

Re: Player Ranking System Feasibility

Post by Kronkleberry »

I've had pretty similar ideas for a while now. I even wrote an algorithm that splits a 12-person queue into two (mostly) balanced fort teams. Deso, raph, and I have been in talks lately about implementing this into the pickup bot on the discord. I think it'd be a popular addition in that it would save the 5-15 mins it usually takes to pick teams as well as avoid games that are totally unbalanced and end in something like 200-80.

My system is also based off of 'low resolution' tiering of the current playerbase. I have 4 tiers (S, A, B, C) with a 15-35-35-15 distribution. After a lot of testing different numbers of tiers and distributions, having it like this proved to be the best compromise between giving consistently balanced teams while also being able to randomize the queue within the tiers, so it won't output teams with the same players every time.

As of right now it's completely subjective as I manually sorted everyone based off of my own perception of skill, with some input from others. I was mostly basing it off of where I see people picked regularly with our captain system. This is was done mostly to test my code and is obviously not ideal. I'd like to populate the tiers through more data driven means. I have some ideas for this but it makes the implementation much more complex, so I'll let the discussion develop for this.

I kinda like your survey idea, having players rank each other based on random head-to-head matchups under the premise of 'who would you chose first on your fort6 team?' This would give us accurate populist data very quickly, and kind of build on the philosophy that this algorithm is to emulate/automate the captain drafting process.

Ranking could also serve a purpose beyond matchmaking. It would be cool to see leaderboards for most matches won, best winrate, best K/D, etc. Maybe we could have a few-month-long splits where we post results, reward winners, then reset the data and give a fresh start for the next split? could incentivize more activity.
User avatar
delinquent
Match Winner
Posts: 759
Joined: Sat Jul 07, 2012 3:07 am

Re: Player Ranking System Feasibility

Post by delinquent »

I think it may be prudent to look at performance during specific events at first. I say this because it's often difficult to determine whether someone is playing seriously or not - when someone is playing just for fun, or whilst under the influence, their performance is invariably changed from the norm, so perhaps it might be a good idea to look initially at ranking players based on their performances in, say, Ladle. It's a bit of an undertaking, though, since like Sine mentioned it cannot be based solely on kill/death ratios or scores. If we did that, effective measures like a zone save would effectively coung against a player, when in fact it is a useful and valid tactic in fortress. The same is true of holing and no-point holing.

Maybe it would be useful to have spectators in a ladle match vote on player performances. It's still not ideal, but it allows extenuating circumstances to be taken into account.
User avatar
Nanu Nanu
Core Dumper
Posts: 189
Joined: Wed Jul 27, 2011 3:20 am

Re: Player Ranking System Feasibility

Post by Nanu Nanu »

I think you could populate a pretty accurate data set of player opinion by using the current pick order in pickup for active players, though this maybe leaves out some less active players who only show for ladle (liz for example). Some captains don't pick based on skill much, intentionally or not, but there's not too many outliers like this
Prema wrote:The second match starts, a new beginning,
Nanu and Prema, Sui and Ninja,
versus those same old hoes grinning.
User avatar
Kronkleberry
On Lightcycle Grid
Posts: 19
Joined: Fri Apr 16, 2010 3:29 pm

Re: Player Ranking System Feasibility

Post by Kronkleberry »

delinquent wrote: Thu Sep 10, 2020 3:06 pm I think it may be prudent to look at performance during specific events at first. I say this because it's often difficult to determine whether someone is playing seriously or not - when someone is playing just for fun, or whilst under the influence, their performance is invariably changed from the norm....
Maybe it would be useful to have spectators in a ladle match vote on player performances. It's still not ideal, but it allows extenuating circumstances to be taken into account.
hard disagree here. I think I, among most veterans, could probably accurately place someone in a tier list after a few seconds/minutes of watching them play. But instead of seconds/minutes, we've literally been playing for decades. We have over 100 ladles, thousands of pickup games, and years of gameplay with each other to base this judgement off of. Extenuating circumstances will come out in the wash if we have more volume to whichever statistical approach we go with. If you're under the influence or not playing seriously in a pickup, maybe that should also count against you? because you're likely to do that again? I've been known to be both those things so IDC if that shows up in my ranking.
Nanu Nanu wrote: Thu Sep 10, 2020 3:38 pmI think you could populate a pretty accurate data set of player opinion by using the current pick order in pickup for active players, though this maybe leaves out some less active players who only show for ladle (liz for example). Some captains don't pick based on skill much, intentionally or not, but there's not too many outliers like this
this would entail someone logging the pick order for every pickup draft no? Kind of a lot of work to achieve similar data that sine's voting system approach would yield. I do like how your idea takes into account how some captains don't necessary pick on skill, because picking to fill a position is just as valid. I suggested using sine's idea but instead of voting on 'who do you think is more skilled?' have it under the pretense of 'who would you pick before the other in a fort6 game?'. This might add a little context to the data, but would keep it equally as subjective.
User avatar
sinewav
Graphic Artist
Posts: 6413
Joined: Wed Jan 23, 2008 3:37 am
Contact:

Re: Player Ranking System Feasibility

Post by sinewav »

Kronkleberry wrote: Thu Sep 10, 2020 7:39 amI have 4 tiers (S, A, B, C) with a 15-35-35-15 distribution.
It's hilarious for me to read this because I was also imagining tier naming based on OPM, heh. I'm def a B-class. Maybe bottom of A.

I think collecting stats from server logs is too much work for too little benefit. Even if ranking only happens before and after Ladle, with participants ranking a few rounds of their peers, we would have a pretty good dataset within a couple months.

This subjective ranking system I propose is basically a hack. The best solution would be to build something into new versions of Armagetron that allow players to opt-in to rankings and have those stats sent to a master server.

Does anyone have time to build a ranking tool as I described? I'm a decent programmer but I don't have a lot of time these days. I could host it, or maybe Kira would be interested in hosting it at Lightron.org?

Thanks for all the good feedback so far.
User avatar
Kronkleberry
On Lightcycle Grid
Posts: 19
Joined: Fri Apr 16, 2010 3:29 pm

Re: Player Ranking System Feasibility

Post by Kronkleberry »

sinewav wrote: Thu Sep 10, 2020 8:54 pmDoes anyone have time to build a ranking tool as I described? I'm a decent programmer but I don't have a lot of time these days. I could host it, or maybe Kira would be interested in hosting it at Lightron.org?
I just wrote a script that does exactly what you said, asks user for input on a random pairing of two pickup players and stores accordingly. I'm assuming raph could host it on armarankings.com, where we can make a page for a GUI for people to cast their votes and store in a DB. From there we can have it feed to deso's pickup bot so we can use the other script I made to sort the teams based on who's !added on the discord.

This could all be done within a week or so if people really wanted it.
User avatar
sinewav
Graphic Artist
Posts: 6413
Joined: Wed Jan 23, 2008 3:37 am
Contact:

Re: Player Ranking System Feasibility

Post by sinewav »

Kronkleberry wrote: Thu Sep 10, 2020 9:39 pmarmarankings.com
Wow, TIL this exists. Yeah maybe we can promote the idea on Discord a little bit and see if anyone comes up with a reason why it wouldn't work or could be abused, that would be great. Also, we might want to consider weighting the ranks a little based on account creation date and number of ranks given. For example, someone who has been ranking lots of people for months would have more weight than someone who just signed up. This might help prevent abuse. I admit I did not think about the implementation too much, just the concept.

Very cool, I'm excited to try this!
blondie
Core Dumper
Posts: 160
Joined: Sat Jun 21, 2014 2:57 pm

Re: Player Ranking System Feasibility

Post by blondie »

I think having it implemented with pickup bot is a no-brainer, at least as a beginning.

In terms of server-based ranking system, I briefly tried a while back but it never went anywhere partly for my not having the financial resources to execute the idea and for people objecting to a server that required a gid to participate and for the overall game dying around that time. Some of the stats collected in that project were really cool, unfortunately I have none of the files and my knowledge of php and server scripting has similarly disappeared.

Basically, I think a good ranking system works when all players have to do is login to a GID, and the server does the rest.
User avatar
delinquent
Match Winner
Posts: 759
Joined: Sat Jul 07, 2012 3:07 am

Re: Player Ranking System Feasibility

Post by delinquent »

OK, I have the infra right here, and the programming ability to build the ranking system. I actually specialise in data visualisation and management, so this is right up my alley.

If you like, Kronk, you come up with the formula and I can then build it in one language or another. I can do it in Mono, since it supports nice data to a degree, and I'll write out an endpoint that Raph can embed in his site - unless I just host the lot. We can perhaps start with the survey system?

Alternatively, if you just want to write the algorithm yourself, let me know what resources it needs and I'll get them bedded in. I can put it under the dollhaus umbrella, since that's the higgledy-piggledy side of things with a lot of different projects.
User avatar
sinewav
Graphic Artist
Posts: 6413
Joined: Wed Jan 23, 2008 3:37 am
Contact:

Re: Player Ranking System Feasibility

Post by sinewav »

Cool. If you guys are going to do this, make sure the system is extensible. blondie's attempt shows that concrete stat collection is possible, and in the future it would be nice to integrate that with an existing ranking system. In fact, right now we could be building the groundwork for something officially supported by a future Armagetron client. Can you imagine an in-game menu that lets you rank the people in the server and sends that data somewhere? That would be interesting, haha!
User avatar
sinewav
Graphic Artist
Posts: 6413
Joined: Wed Jan 23, 2008 3:37 am
Contact:

Re: Player Ranking System Feasibility

Post by sinewav »

RELATED IDEA

While playing #pickup tonight I had idea about using IRC/Discord bots to create balanced teams in fort6. Disclaimer: I don't think there is anything wrong with #pickup. It's super fun. This is just an idea to talk about, not necessarily something to try.

This follows along the same idea as player rankings that involve personal opinion rather than hard stats. What if we created a #pickup bot that asked players to rate other players?

Suppose it is Sunday and there are lots of people playing #pickup games (and by "lots" I mean the entire Fort community of three dozen people, heh). As people sign up for fort6 matches, the bot looks at the pickup queue, randomly selects a player and a position to rate, then sends a private message different player in the queue:

Code: Select all

On a scale of 1-4 with 4 being the best, how is <player 1> at center|wing|sweep|defense?
The bot slowly collects feedback over time then uses that feedback to make "suggested" teams that are balanced based on the rankings. We don't have to take the suggestions, but it would be neat to see if this helps captains who aren't so great at picking teams -- for example, me. I don't play as much as the rest of you can I can't keep up with all the name changes. I would love a tool that let me know who is considered "good" these days. :stubble:

Of course, the rankings should remain secret if possible and throwing a tiny bit of randomness in the team picking algorithm to further obscure them would be helpful too.
User avatar
delinquent
Match Winner
Posts: 759
Joined: Sat Jul 07, 2012 3:07 am

Re: Player Ranking System Feasibility

Post by delinquent »

If we were to approach such an idea, I might suggest that we do so completely separately from regular pickup - a so-called "ranked" pickup if you will. Assuming we score players based on their performances solely in those games, we might see some slightly different results.
User avatar
sinewav
Graphic Artist
Posts: 6413
Joined: Wed Jan 23, 2008 3:37 am
Contact:

Re: Player Ranking System Feasibility

Post by sinewav »

delinquent wrote: Wed Sep 23, 2020 10:09 amI might suggest that we do so completely separately from regular pickup - a so-called "ranked" pickup if you will.
I don't think that's necessary. The point is to make using the suggestion optional. It would be cool to see if a ranking system actually spit out teams that match what we expect. The suggestions would probably be useless at first, but after several weeks of collecting data they should get better. The good part is, it's doesn't rank players from best to worst, but rather highlights the best player for each position and makes fair teams.
Post Reply