Double Binding is the crutch of the weak

[Lucifer]

server side lag can't be solved by getting a better somputer, only a better server/more bandwidth for the server's connection.

Respectfully disagree.

Server side lag can be caused by any of the following problems (and more that I haven't mentioned):

1. Bad scheduler. Running the server under Linux 2.6 or a Windows NT kernel is probably your best bet, I don't know if it'll run on Mac OS X, but Darwin should have a decent scheduler too. This only affects servers that run multiple processes/threads.

2. Bad threading model. As above, so it is here.

3. Slow processor. This isn't the most likely cause, most of the time.

4. Too little RAM. Depending on how much memory each connected client requires, but keep in mind that each client requires some memory just for upkeep of the connection, and then some memory for their movements, colors, name, and so forth. (name is probably under upkeep)

5. Bad IP stack. This shouldn't be a problem either, since all of the OSs in use right now use BSD-derived IP stacks and are pretty bulletproof in that regard.

6. Inadequate connection. Not necessarily too slow, just inadequate. A cable connection is bad for this application because of the disparity between upstream/downstream speeds. A slower downstream DSL connection will probably host a server much better than a cable connection. Still, many of us don't have the choice on connection.

7. Too many running processes. No matter how good your scheduler is, too many other running processes will still affect the load significantly.

8. Bad programming. Not necessarily bad, but in certain server applications it's definitely better to run a separate process for each client. In others it's better to just run threads instead (also known as lightweight processes). In still others a single thread is best. So the "bad programming" I'm referring to is just choosing the wrong model for the application, and in many cases it's not necessarily "bad", it's more "I chose this model and after hundreds of people started playing discovered it was the wrong model". "Good programming" in this case would allow the programmer to switch models without having to rewrite the whole application.

9. Latency. Latency in general, but I'm specifically thinking about kernel latency. How long it takes for someone's UDP datagram to get through the layers from the NIC to the application and then back again will certainly affect lag. General latency in other areas will as well, such as how many hops the datagram has to go through to get to the server (hence why some users have high pings while others have low, but we already knew that). Client machines will contribute to lag in this fashion because if your computer has to in translate the incoming data from the server and render it to the screen, which introduces some latency in every action you take. This latency can't be avoided either, but can be reduced.

How to cut down on lag?

First, does AA use a single thread, multiple threads, or multiple processes to manage each client connection? Some of these solutions won't make a difference, depending on this one single fact.

1. Get the best available kernel for your OS. If you're running Linux, upgrade to 2.6 if you haven't already. You might want to apply the low latency patches, I think they're badged Alan Cox but I could be wrong.

2. Same. If you're running any NT kernel, any Linux post 2.2 (possibly 2.2 has a good threading model too), and Mac OS X (or any BSD, for that matter), then this isn't a problem.

3. As the most expensive solution, hold off on this one until absolutely necessary. Some things to consider, though. If AA uses multiple threads, you might have some luck with a hyperthreading processor such as the Xeon. If AA uses multiple processes, then a dual-proc setup will help a *lot*. A beowulf cluster might help, but then you have added latency you don't really need, so I suspect building a supercomputer to host an AA server would be firing a shotgun at a mouse. Also, more L# cache is better, so if you're running a Duron, getting an Athlon of comparable speed will improve performance noticeably. Same if you're running a Celeron.

4. More RAM is the cheapest solution that requires money. Also look at the speed of the RAM and make sure you're running RAM that is the fastest your motherboard/processor can support. Faster swap space is an option, but usually more expensive than RAM, and I just can't see AA using all that much memory.

5. You can probably ignore this one unless you're interested in kernel hacking. If you are interested in kernel hacking, then you don't need anything *I* might be able to tell you, since I dont' know jack about kernel hacking.

6. You probably don't have a lot of choice in the matter, but I understand that SpeakEasy has one of the best-reputed DSL services for gaming.

7. If you're running an actual dedicated server, you solve this problem by running a very light weight Linux/BSD installation, where the only running processes are things you absolutely must have (iptables, syslog, a couple of others). If not, then you probably don't have an easy solution to this, other than to shut down all unnecessary services. If you're running a display manager because you use the computer for regular work, that's going to create a lot of lag, but you don't have much choice, do you? On a dedicated box, Windows is the absolute wrong OS for this, since it basically runs a display manager no matter what. There's a reason Linux and BSD servers routinely outperform Windows servers.

8. You have to ignore this one unless you want to hack on AA. Even then you might discover that AA already has the best model for its application, but there's almost certainly room for optimization. Until you've embedded assembly code everywhere, there's always room for optimization. Of course, assembly code isn't portable, but some projects have found ways around it (MPlayer being the most visible that I know of). But some performance gain might be found in rewriting the client connection code in C, if it's in C++ now. I'm not certain about that, but I understand that C++ bindings did turn out to affect performance, but there's almost always a bigger fish to fry than that.

9. The best known way to deal with latency is to compile your kernel, building in your NIC drivers and so forth rather than having them built as modules. The old performance hit of running modules is rumoured to have been fixed in linux 2.4, and is almost certainly fixed in Linux 2.6, but I don't know it for a fact. Switching the server to Gentoo Linux is the easiest way I know of to deal with this, since Gentoo optimizes itself for your system while you install it. If you're running Debian, switching to Mandrake, SuSE, or whatever, will also help a bit, since Debian is generally built for 386 processors where the others build for Pentium and above.

When you've done all of htis, you might find you have gained the ability to run lag-less with two or three more connected clients, or you might not. Good luck.

[Tank Program]

That is a good post. And lucky me... Tigers Network is on a dual xeon .

[philippeqc]

Your quite right lucifer,

"bad programming", latency and pumping asemblor code
Humm, yes, your right. A bit extreme, but right. It's the "or we could nuke them" solution of programing in my opinion. To go and get that last drop of performances. But before I touch asemblor, I'll see if just changing a few algos wouldnt help the case.

Yep, you have a neat list. Very neat. But I think you forgot ...

9) Distance to the server.

From Stockholm, Sweden to Cali USA, time required for traveling packets is no negligable. We are talking speed of light limitation here (actually more like 2/3 of speed of light for electrical signal transmision). And delays by every router you hit on the way. All resulting in a ping in the order of 200.

I've always beleived google to be located somewhere in the west coast of the US. And I can ping them in 60 ms. But its a fake value, because I'm sure aa server in cali arent connected on such a fast pipe as google, ICMP packet are processed differently than TCP and UDP.

I still beleive there must be some resonnable optimisation left to be done, at least when many players are involved. Saturday I had a nice little game versus guru on his server (about 1000 km south of stockholm) with a ping of 4. If my memory serve me right, I get pings of 45 when more than 5 players are involved.

But overall, lucifer, you had a very nice analysis.

Now the question is, could the bright mind who compiled the list take the job to fix a few of the items. Lets say the ones within his reach, cause even the devil his bound by the speed of light and other pestive nuisances....

-ph

[Lucifer]

Now the question is, could the bright mind who compiled the list take the job to fix a few of the items. Lets say the ones within his reach, cause even the devil his bound by the speed of light and other pestive nuisances....
-ph

Would love to, with the minor problem that AA is already beyond my C++ skills. ;( Sorry. I'd be willing to beat on it at some point, but my plate is pretty full right now. I was hoping to list stuff that people can do without coding using current versions to deal with lag rather than suggest long-term coding solutions. (Also, having tried to compile AA before, it was having trouble with some libraries I have installed. Maybe I'll beat on it just to work out these problems, 'cause I'd *really* prefer to build it myself than install an rpm, but that's work I kinda owe to Crimson Fields right now)

OTOH, it would be nice to get my feet wet with C++ again, but right now I owe any C++ time I come up with to Jazz++...

Heh, as you can see I've got a lot on my plate. ;(

Edit: forgot to point out that optimizing to assembler is actually a bad idea, in my not-so-humble opinion. The advantage to not going lower-level than C is that you give the compiler the option of optimizing for the architecture, and by going to assembler you could defeat that. MPlayer gets around it by the amount of testing they can do, thanks to the amount of people using the thing, so they use assembly to optimize for specific architectures in places where they *can* do better than the compiler.

And also, of course, if AA had been written in Python I'd be able to start hacking on it anytime, since Python's what I work with the most.

[Lucifer]

I've always beleived google to be located somewhere in the west coast of the US. And I can ping them in 60 ms. But its a fake value, because I'm sure aa server in cali arent connected on such a fast pipe as google, ICMP packet are processed differently than TCP and UDP.
-ph

Mm, forgot to mention. Google servers *used* to be located on the west coast, but now they've got servers all over Europe and the US. I think they've also got a round-robin setup on www.google.com, so your pings will be going to different physical servers, as is your web browser, each time you hit it. I don't actually know if google checks IP addresses and routes you to a server nearby, because that's a fairly well-known bad way to do it, at least at the state level, but it should be pretty reliable identifying connection requests that come from europe vs the US and so forth. When IPv6 finally hits (if it ever hits), I think they're planning on distributing the IP addresses better so that your IP address will almost always accurately reflect the region you're in, and I'd expect DNS to take that into account somehow.

And last I heard, Google was hooking up either at Tier-1, Tier-2, or straight into the backbone, and I forgot to mention that just getting your server closer to the backbone will help with ping times, as is getting your client closer to the backbone. And it'll probably make a significant difference.

There are some things on the list I can hack at, now that I think about it. If you guys are willing to throw money at it, anyway, there are dedicated hosted servers plugged into awesome internet connections (usually tier-1) that go for $100/month, running Xeons and so forth. I'd be happy to admin such a box, but I'd have to charge something for membership, in order to pay for the server. No promises that lag will be better, but the rented servers usually have better equipment than most of us are likely to have in our homes, and there are a number of places offering varieties of Linux and BSD servers.

[philippeqc]

lucifer,

I think we are at quite the same technical understanding. Noting to aregue with what you say.

About getting a box closer to a backbone, great, but who will pay? and In what country. US located box are not interesting for European because of the trans-atlantic lines. (and great distance)

So we would need at least 2 of theses. Europe and North America (north america. I'm canadian and will always stay, even if I live in Swedent) and I guess few ppl are willing to pay for aa.

Now as you mentionned, solution come at one of 2 cost. Money or Time. Seems few ppl have either, we will have to support the endless whine of lag comment for a while. (Even your idea of finding better libs take time. its a sad sad thing) Unless you have a good idea that I might be able to fix?

[Lucifer]

About getting a box closer to a backbone, great, but who will pay? and In what country. US located box are not interesting for European because of the trans-atlantic lines. (and great distance)

That's the hard part. Any idea how much a European box would cost these days? Just rented, possibly, or a cost analysis of high-speed tier-1 or tier-2 connection + good server vs rented?

As for who will pay, the simple answer doesn't exist. People want things that we (as in "everyone with a server", and my server's brand new in the list) don't have the time to provide. Tourneys, mostly, but other stuff. As the program itself develops, we'll (hopefully) have the option to have special graphics/music and stuff for the servers we host. Those two things are value a commercial service could offer. These forums are value, but there's almost certainly more that can be done for community, although nothing's coming to mind right now.

There are three possible sources of revenue. Membership fees are one. Advertising on community websites are another. A third, the most annoying one, is advertising directly in the game (when custom by server graphics are supported). If there's enough business, there may appear corporate sponsors of tournaments and stuff, with the possibility of adding prizes.

Expansion would involve getting other good open source games and hosting servers for those, possibly other things that come to mind.

If there are enough North Americans willing to commit to enough membership fees to pay for a server in North America, that's enough to get started. I can't front money for it right now, but in a few months that'll hopefully be different. Same with Europe, for that matter.

Two important things here, though. One is that online gaming in general doesn't make money, although specific networks do (everquest?). The other is that if the thing can start at break-even off of membership fees, I should be able to market it in such a way that it'll grow, although I can't guess at how fast it would grow. Every business I've started so far has grown, although I've shut each one down (except for one, long story, but they haven't grown since I left) for reasons generally not related to how much money they made.

In the long run, advertising will likely be the larger source of revenue, but membership fees need to pay a certain minimum amount of hardware/bandwidth costs or the thing won't have staying power in the market.

I should warn you, before you start thinking something else about me, that every oss project I've gotten involved with I've raised a fuss about how to make money off it and not been in a position to actually do so. Here, though, if enough people are willing to commit to enough money to break even, there's room for me to work on it. I've just started doing independent contract writing (OSTL is interested in my stuff, and I've been published by LWN before), and if that goes well I'll actually have time to pursue something like this. But I'm also kicking tires on some other business models I've got in mind, and when it comes time for me to pick one, I'll the pick the one that's most likely to make money the quickest. Anybody interested in investing in my next get-rich plan?

Now as you mentionned, solution come at one of 2 cost. Money or Time. Seems few ppl have either, we will have to support the endless whine of lag comment for a while. (Even your idea of finding better libs take time. its a sad sad thing) Unless you have a good idea that I might be able to fix?

Without cracking open code, I've no ideas on what specifically in the code can be fixed or changed to deal with lag. I'm not exactly hot on network programming in general. How does AA already deal with it? Does it use one thread for everything, one thread for each connection, or what? I know it doesn't start new processes for each client, just by watching my process listing when multiple clients are connected. I would generally think that you need to be multi-threaded at least, with a master thread keeping track of everything and one thread connected to each client. Multi-process isn't a bad idea, but the overhead is quite large. Still, we know that UNIX domain sockets are plenty fast since that's how X talks to itself, and X is plenty fast (could be better, but several prominent open source hackers have pointed out that that's not the bottleneck on X), and the benefits of having one process per client are substantial, making the thing very much scalable. Imagine having one server connecting a hundred clients into one arena.

I would think that internal latency is the best and possibly only place to address lag directly in the code.

Besides coding, we could start up a topic that specifically discusses how you can improve individual server performance, and individual client performance. Hopefully the server is already coded in such a way that individual client performance doesn't actually lag the server. There we could discuss such things as renicing the server process, or just running it with root privileges (a *bad* idea), or whatever else comes to mind. Actually run semi-scientific tests and see what works and what doesn't.

[ishAdmin]

I'll chime in here. I'm an old school programmer, am not familiar with C, nor am I familiar with programming for network communication. The last *big thing* I wrote were a couple of applications for Macs around 8 years ago. Since then I've been more of a script hacker, and mostly for linux, using Bash, php primarily. (I've dropped perl for the bloated pig it is.)

Now ish is on a dedicated server with about the best connection to the net one can get. It's in the U.S., in a data centre with multiple backbone connections. The last time I checked, it had around 14 Gigabits of bandwidth into cogent, verio, time, etc. Most of the time the traffic will be through a single 1Gigabit connection. Switching based on load will go on, and I've wondered if this might result in a temporary lag showing up. (no idea on that, just speculation) The server box itself has a 100 Mb/s connection to the data centre internal network as that is the speed of it's network card. This will be the case for pretty much any server anywhere btw, you're just not likely to have anything faster than 100 Mb/s as a theoretical maximum between you and the backbone.

I've done some experimenting over the months with things like renicing the process to see how it improves performance. I've never observed anything significant that I could control in the server box. I've often played the game while monitoring server load and other processes. In virtually every case, periods of high lag come and go with nothing tracable on the server. I've come to the conclusion that factors external to the server have the *highest* impact on game performance. Between each computer and the server there are generally between 10 and 30 hops that each packet must navigate. There is no doubt that this is where the true bottleneck for communication is. But more to the point, the integrity of that path is always in flux, and and is always changing/adapating to the various load conditions. Your path to the server is always changing in other words. I've often done a traceroute during game play, and will make a point of doing it during times of lag and comparing them to times without lag. My casual observations have left me with the impression that again, there's not too much I can do on the server to improve things.

The server is doing other things, and occasionally server load increases which will affect game performance. This isn't that often by the way, but it's fair to mention that sometimes the server is to blame, even if it's probably less than 1% of the time. (based on my observations), Also, I will be changing servers in the near future. I'll be getting bigger/faster/better in the same data centre. I'm hopeful ish will perform better after this upgrade, but I'm frankly not that optimistic anymore. The internet is getting busy, and is the real bottleneck. I actually noticed an increase in lag just after school got out. Lag is here to stay in my opinion, but I am very open to any suggestions that will help me improve performance on my end.

And one more thing, when I do upgrade the server, I'll probably put the game up before anything else. I'm curious to see the best it can do in an otherwise idle server. I'll try and give it a few days, and will announce it in this forum. (Actually, I'll probably have to ish's running at the same time for a while, one on each server.)

Just some thoughts from a game host who should really be getting other work done.

[Tank Program]

Tigers Network is on a 100mbit university connection- I've verified the bandwith on it being up to 45mbits. Most of the players on it are from Europe, plain & simple. The server itself is under minimum load from other applications, so I don't think the internal latency there is too bad. But I agree with ishAdmin- the majority of lag is caused by just too much traffic. Maybe we can all hop on Internet2 to reduce the lag . Then again... That's just a university thing. Anyway... The closer to the server you are the better, plain & simple.

[philippeqc]

Wow

That was really insightfull. To have the opinion of someone actually running a server in a data center! Really good read.

I guess the only thing left to do is to be sure that the trafic is not O(n^2) or something awefull like that. ishAdmin, did you by chance mesure the trafic (bandwidth) from the server vs the number of players involved? That would give us an anser quite fast.

Thanks again for that educated anser.

-ph

[Tank Program]

ishAdmin, if you don't know I recomend iptraf to find that stuff out.

[ishAdmin]

I'll look into that and get back to you.

[Thak]

Repeatedly 180 and lag the server? Get a better computer.

I have arthritis in my hands. I smoke the weed in order to lessen the swelling and pain. I rely on double binding to help my slow reaction time.

Thats just an excuse, I know that you used to play the game well without double binding.

[Lucifer]

To add to what ish said, and what I've said already

With 8 players on Breakfast of Demons, armagetronad-dedicated uses 5MB of memory and 38% CPU on 800mhz Duron, 256MB DDR RAM (133mhz front bus), Mandrake 10 (Linux 2.6.3), KDE 3.2, full desktop.

Edit: With only 2 players, armagetronad-dedicated is now using 6.8MB of memory, but only 22% CPU. Memory leak, maybe?

I suspect just putting an Athlon in this box will take care of 90% of the jumpy complaints the server gets right now. Increasing L# cache would make a *huge* difference. RAM is already as fast as the motherboard will take (it's an old mobo bought on clearance), and some simple math shows that armagetronad only needs slightly less than 1 MB per client.

[Lucifer]

I'll chime in here. I'm an old school programmer, am not familiar with C, nor am I familiar with programming for network communication. The last *big thing* I wrote were a couple of applications for Macs around 8 years ago. Since then I've been more of a script hacker, and mostly for linux, using Bash, php primarily. (I've dropped perl for the bloated pig it is.)

You should try Python. For scripting that's not directly web scripting (which is what php is for), Python blows all the others out of the water.

I've come to the conclusion that factors external to the server have the *highest* impact on game performance. Between each computer and the server there are generally between 10 and 30 hops that each packet must navigate. There is no doubt that this is where the true bottleneck for communication is. But more to the point, the integrity of that path is always in flux, and and is always changing/adapating to the various load conditions. Your path to the server is always changing in other words. I've often done a traceroute during game play, and will make a point of doing it during times of lag and comparing them to times without lag. My casual observations have left me with the impression that again, there's not too much I can do on the server to improve things.

There is no doubt that distance from the server is the largest cause of lag, but I think we should break it down into several subtopics, for clarity. Here we're talking about network latency. I know that every packet that leaves my machine has three hops before it hits the internet. First my router (I'm not counting the originating machine), then the cable router, then my ISPs gateway. Since I get decent pings on Breakfast, I don't worry about these hops, but those of you guys in Europe might try experimenting with pulling out any extra network crap and see if it helps your ping. Probably not, but worth screwing with.

The other control we have over network latency is in the protocol itself. The only real problem here is keeping backwards-compatibility. But there is coding here. So, I've come up with some possibilities.

First, in every round, people die. It's a fact of life. They go into spectator mode. At that point, do they remain connected to the server to receive the data? If so, then it's worth considering having them connect instead to one of the remaining clients while in spectator mode. There would be obvious lag watching as a spectator, but it would reduce server load during the round, since the server wouldn't have to serve spectators anymore.

Second, can chat be turned into P2P, instead of being routed through the server? Is it already P2P?

Third, compressing the data. I don't know how realistic this is, and it will create more latency that is internal to the server and each client, but if it's actually realistic to compress the data (if it's not already), then there's a way to move some of the lag issue to areas we do have direct control, by turning lag caused by network latency into lag caused by server and client performance.

Fourth, and this may not make a whole lot of sense at all. If the spectator suggestion is doable, then it's worth considering working out a way to make actual gameplay P2P to some extent. For example, let's say Tank has the lowest ping to a server located in California, out of all the folks in Europe. So other european clients would connect to Tank instead. They would experience greater lag from other american clients, but less lag from other european clients. OTOH, it would reduce load on the server since it's not serving 2+ european clients anymore, it's only serving 1. This same sort of delegation could be applied to other clients as well. Is this at all realistic?

All of these solutions require programming. But philippe was asking for specific suggestions.

And one more thing, when I do upgrade the server, I'll probably put the game up before anything else. I'm curious to see the best it can do in an otherwise idle server. I'll try and give it a few days, and will announce it in this forum. (Actually, I'll probably have to ish's running at the same time for a while, one on each server.)

I'll tell you what I'm specifically interested in. Can you run a benchmark suite on the server to track these sorts of things? I'm wanting to know how much impact the L# caches have, processor utilization, memory utilization, network traffic measurements (amount of data and speed for each client and aggregated for all clients). I'd like this data to somehow be indexed by number of clients connected. I'd also like to know how moving cycles affects it. I would assume the protocol is already optimized so that if you don't turn, no data is exchanged other than sync data, and I've subjectively experienced just cruising in a straight line for a second waiting for lag to clear.