Pings and Lag, suggestions?

[Swampy]

I don't mean to kick a dead horse, because I know this is one of the universes unsolved mysteries in some regards. But, I'm hoping that someone can make a suggestion that could improve things for players on Swampland "Teams".

I host two servers, one is a P4 1.3ghz (640mb ram) and Mandrake 10.1 Community. It's hosting "Swampland (Breakfast), aka, the "original" Swampland server (2.7.0 Armagetron).

The second server is an AMD 64 3000+ (512mb ram) and it's currently running Mandrake 10.2 64bit "Beta" (2.7.1 Armagetron).

Both are going through the same router, and both have static IP's and the same gateway and DNS info in both. They share a broadband connection to the Internet and both have the same software installed (binaries).

Neither is running any server software with the exception of Apache' and PHP.

Can anyone think of any reason why one server would have a higher ping and more lag then the other? The ironic piece to this is that many are claming that the older server has less lag then the original. I can't make any sense out of it.

I've tried replacing cables, the NIC, swaping ports on the router. I also changed the OS on the new box, it was running Fedora 64 for a couple of weeks.

Any suggestions?

[Lucifer]

Did you install armagetron from rpm? ie are you running the 32-bit version or did you build/install a 64-bit version of armagetron?

[Swampy]

I installed it from the RPM. I wouldn't even know where to begin building a 64 bit version

[Lucifer]

Try switching the two ports for the two machines and see what happens. You know, unplug the Either net cables and put them in each other's ports.

You see, when I run > 1 server on my one machine, all the pings on all the server processes are about the same. So I'm suspecting something in how your router is shaping the traffic, or maybe it's a network card, network card driver, or something like that.

[Swampy]

I tried changing the ports a couple of days ago and it didn't help. If I start the game on another machine in the house, I see both servers and each has about the same ping. Moments later the new server will be 100+ higher than Swampland, for example:

Swampland (Breakfast) 16
Swampland "Teams" (Breakfast) 116

Here's some info I pulled using IfConfig:
Link encap:Ethernet HWaddr 00:XX:XX:XX:XX:XX
inet addr:192.168.1.102 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: XXXX::210:XXXX:XXXX:XXXX/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:96 errors:0 dropped:0 overruns:0 frame:0
TX packets:71 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:17644 (17.2 Kb) TX bytes:7023 (6.8 Kb)
Interrupt:201 Base address:0xbc00

Does that help? Is there anything else that I could run to get some info that might?

[eggcozy]

Would it be worth trying 0.2.7.0 instead of 0.2.7.1?

[n54]

Time: 3/19/2005 4:51:53 AM
Application: armagetronad
Remote point: -censored- , port 4534
Direction: Incoming
Local point: 0.0.0.0 port 2809
Protocol: UDP

Got this while playing Swampland (not the team one) and I'm posting it here as it might (just might) answer the questions (or alternatively raise quite a few).

I'm just "guessing" it's your server name/address (since I guess the program only keeps a connection with the server it's actually playing on during gameplay) so are you rooted or are you spoofed? Or more likely: both? Either way it could explain the lag problems if somebody is doing this (lame attempt at portsniffing through armagetron I guess) and more to all connected.

If -censored- , port 4534 is not your server then someone did something very foolish (and it wasn't me).

If you're neither please explain why your server does this as I consider this server behaviour the precursor to an attempted attack. I closed the game (which froze at the arrival of the first request) and then denied all requests which continued to arrive after closing the game (about 15 all of which were 100% identical, always aiming for 0.0.0.0:2809)

Afaik:
- there should be no way for clients to directly get the ip of other clients unless they compromise the server or interrupt packets. I kinda find the first more likely than the second but one never knows, therefore the rootkit scenario on your server.
- the server has no reason to request an additional connection, and certainly not to 0.0.0.0 which is the default network address (i.e. the attacker wants to get into 2809 anywhere whatever network address I use locally)

I hope I get many insightful replies clearing up this matter

Edit 1: censored the address, it was a bit incosiderate of me, apologies offered

Edit 2: explained edit 1 and added this:
I realise that this post might seem a bit "scary" so if anyone reading it got a bit worried then don't be: just about any firewall at all should either automatically deny such a request or immediately notify you of it and ask for advice (at worst it denies it but doesn't notify you just logs it somewhere, if it doesn't do that well... then it needs reconfiguring or replacement). If you had port 2809 open or no firewall however then this might be an issue if and only if this was a malicious portsniff (something we do not know yet).

[Lucifer]

z-man? (n54, PMd)

[Z-Man]

I'm still puzzled by n54's post. I suppose he was using windows at that time, then port 2809 could well be the port AA bound its socket to locally. I don't know what the 0.0.0.0 destination IP means, because it might just as well be normal for a personal firewall to show all incoming packets with this destination address. And I suppose that the -censored- IP was different from the original server IP. To me, it looks like the kind of package clients get when the server IP changes. The repetition may just be the normal attempt of resending packages that don't get acknowledged.

Swampy, do you have a static IP? Scanning past server lists, it seems you don't. Does your ISP or router cut your connection forcibly every 24 hours or so and you have to reconnect, getting a new IP?

Anyway, if this is a portscan, it is a very poor one.

And of course, my ramblings don't help a bit about the ping issue. I'd suggest trying eggcozy´s suggestion and revert to 0.2.7.0 for testing; that could rule out coding side mistakes.

[Lucifer]

I'll answer what I know.

And I suppose that the -censored- IP was different from the original server IP. To me, it looks like the kind of package clients get when the server IP changes. The repetition may just be the normal attempt of resending packages that don't get acknowledged.

The censored IP was the server IP. I actually asked n to pull the hostname out of his post, figuring that would stop anybody who was idly curious about picking a life to destroy and it would steer them clear. Of course, if someone was determined to destroy Swampy's life (unlikely), that wouldn't stop them.

Swampy, do you have a static IP? Scanning past server lists, it seems you don't. Does your ISP or router cut your connection forcibly every 24 hours or so and you have to reconnect, getting a new IP?

I can also say that Swampy's IP doesn't change very often. The stats page used to point to his IP to get stats from his server and it worked for several weeks before it was updated with the no-ip hostname.

Anyway, if this is a portscan, it is a very poor one.

I'm inclined to think the answer is somewhere in aa's code itself and that n54 was just surprised by what he saw because he'd never seen it before. I've recently been through the syslog on swampland and didn't see anything that raised my eyebrows.

And of course, my ramblings don't help a bit about the ping issue. I'd suggest trying eggcozy´s suggestion and revert to 0.2.7.0 for testing; that could rule out coding side mistakes.

We did that earlier tonight. The lag was actually significantly better with 0.2.7.1 than with the older version and we put both servers on it to finish up the night. Pings were actually lower but only marginally so, and we shut off the main Swampland server for the test (which runs on a different machine than the team server). I expect to see Swampy start taking measurements of network latency through his router and each network card to see if it's a specific chipset on a network card, the router itself, or a network card driver. Actually I think he was thinking about putting the 32-bit Mandrake back on the 64-bit server. In this specific application, I would expect that to improve the situation all around (and might even make for a server that can take more clients).

So, swampy, after the 32/64 bit OS thing (which I think would be a huge improvement to back off to 32-bit again), I'm back to suspecting the router. Anyone have any ideas how to measure network latency across the router? Mandrake Linux and Windows SomethingOrOther are the OSs available for the test, and probably any liveCD that can be downloaded.

And yeah, we're back to suspecting stuff that's external to aa.

In the process of running the test, we also got to do a direct comparison of rubber in 0.2.7.0 and rubber in 0.2.7.1 and concluded that all servers that switch to 0.2.7.1 need to lower their rubber if they want to have the same gameplay as before the upgrade. We haven't determined yet by how much because we haven't actually lowered rubber on any of our servers (Swampy may have done this after I last spoke to him a number of hours ago).

[Yaza Yamagotchi]

I also got an incoming connection attempt from swampy's server earlier today in the same fashion as N54 described (I closed tron and had lots of follow up connection attempts), I was quite suprised and I could see no reason for the connection attempts, so I rejected them and have set a rule in my firewall to disallow any future attempts.

The IP and port were correct for swampy's server.

[n54]

I suppose he was using windows at that time.

Yup a Windows 2000 Professional

then port 2809 could well be the port AA bound its socket to locally.

Does armagetron change socketbinding during gameplay? The firewall alert came up in the middle of me playing a game (at least like ten minutes after joining the server). In addition this was a request initiated from the server (or at least using the server host name and address) and not my machine (if my armagetron client had initiated it it would not have been classified as incoming by the firewall rather it would have been a reply to an outgoing and already approved).

Another thing to note is that the request came using port 4534 (the default armagetron port). My firewall works in the way that either a packet comes in using an already approved connection (in some way initiated by me and my actions which -censored- port 4534 indeed was) or the packet is simply deleted without any kind of response to the sender. This tells me/us that the port armagetron used to communicate with the server was port 4534 (standard armagetron port) as otherwise the request would simply have been deleted without warning. (Btw Lucifer this is why I didn't bother checking the host name/IP since no other valid connections were up at the time).

I don't know what the 0.0.0.0 destination IP means, because it might just as well be normal for a personal firewall to show all incoming packets with this destination address.

Not to my knowledge, afaik the destination IP is taken from the arriving packet. 0.0.0.0 is a "nonexistent" generic network IP used as a wildcard IP to signify any network possible just like 255.255.255.255 is a "nonexistent" generic broadcast IP used as a wildcard IP to transmit a packet to absolutely any and every machine everywhere. Now my firewall knows very well which network IP and node IP my machine is in, as do I, and the firewall would essentially be giving 0 information if it gave anything other than the target IP from the interrupted packet. However an attacker has no idea which network (IP range) nor host IP I use internally and so an attacker needs to try to use 0.0.0.0 or 255.255.255.255.

And I suppose that the -censored- IP was different from the original server IP.

The censored IP was the server host name and IP - 100% identical.

Anyway, if this is a portscan, it is a very poor one.

Yes absolutely, however this is the first time I've ever experienced something like this and I guess it's pretty safe to say I've played a lot

Lucifer can you see from server logs wheter or not the server IP changed?
It should also be easy to find out if somebody adminestering Swampy's changed Swampy's armagetron port...

Actually I can do that myself...
*fiddles a bit*
yup the IP for swampy's server is still the same!
*connects to swampy using custom connect with IP and port 4534*
and yes the port is still the same too....

That kind of settles it imo, I think we need to look closer at this. I really doubt this was any kind of non-malicious packet/connection attempt.

[Z-Man]

However an attacker has no idea which network (IP range) nor host IP I use internally and so an attacker needs to try to use 0.0.0.0 or 255.255.255.255.

But how would such a packet have found the way to you? That's what puzzles me here.

The censored IP was the server host name and IP - 100% identical.

More puzzlement. But at least that may allow a natural explanation. Some internal facts:
The AA client does not normally change the source port. It never does without a reason, but sometimes, network sockets break and need to get restarted. Sockets break, for instance, when you send a packet to another machine that is online, but not listening to the destination port ( a very stupid feature for UDP, if you ask me ). This happens a lot in the server browser when you ping servers that went offline, but should not happen during normal play.
Now, from the server's perspective, early experiments with NAT routing revealed that some routers are incapable of keeping the source port of a UDP "connection" constant, so the server sees the client packages come from fluctuating ports. Therefore, code has been added to allow in packages from fluctuating ports when the rest of the connection data ( client ID and source IP ) are correct.
Now, let's move backwards. n54 and Yaza received packages from the server, but probably not with the correct destination port ( since they sent data to the server using this port and the firewall would have allowed the response ). A possible explanation for this could be that the server for one reason or another triggered the client port change code. I double checked it, the sender IP really has to be identical to the stored sender IP. So either someone managed to get your IP ( not through the server ) and forged a packet, or your client really changed its source port.
And here is where I'm lost again. If a real client port change is the reason, then at least one packet with the new port must have found its way from the client to the server. But then it would have been detected by the firewall and the incoming packet would have been accepted. I'd like a chain of socket breakage to explain this: server socket breaks because another client went offline unexpectedly, client socket breaks because the server socket was blocked for a short time, port change. But it just does not add up.

There were changes in the relevant code from 0.2.7.0 to 0.2.7.1, I'll review them for a possible explanation.

Lucifer or Swampy, it could be insightful to se the server logs ( with cleaned IPs ) for these events, if that's possible. They should be visible as timeouts where n54 or Yaza were affected. If any of the reasoning above is correct, there also should be a message like
"changing port of user 5 from 1066 to 1088."
prior to the timeout.

[n54]

However an attacker has no idea which network (IP range) nor host IP I use internally and so an attacker needs to try to use 0.0.0.0 or 255.255.255.255.

But how would such a packet have found the way to you? That's what puzzles me here.

I would say there are three general possiblities:
1. Something slightly obscure and rare got triggered in the server armagetron code
2. The game server was in some way compromised so as to give a third party knowledge of connecting IPs which allowed somebody to send malicious packets
3. The packets arriving to and from the server to clients got intercepted allowing somebody to send spoofed malicious packets

Bear in mind for 2 and 3 that getting the server sender address is very simple (and kinda has to be) so for these getting the IP of clients is the deal.

# Edit start:
And packet snooping almost always requires the snooper to be in control of a connecting device between the sender and receiver, i.e. generally not a cracked computer but a cracked router or internet router or gateway router. All in all way too much effort to spend to send a stupid almost pathetic request for an obscure port hardly in use. *Just thinking out loud and deeply insulting any possible miscreant*
# Edit end

The censored IP was the server host name and IP - 100% identical.

More puzzlement. But at least that may allow a natural explanation. Some internal facts:
The AA client does not normally change the source port. It never does without a reason, but sometimes, network sockets break and need to get restarted. Sockets break, for instance, when you send a packet to another machine that is online, but not listening to the destination port ( a very stupid feature for UDP, if you ask me ).

Fair enough but this seems very strange when the request uses the actual connection my client in this case is supposed not to listen to to ask the client to open a new connection using a port (2809) that is way different to the normal armagetron ports. Would such an automatic failsafe have any reason to switch to using 2809? Instead of for example 4535?

Now, let's move backwards. n54 and Yaza received packages from the server, but probably not with the correct destination port ( since they sent data to the server using this port and the firewall would have allowed the response ). A possible explanation for this could be that the server for one reason or another triggered the client port change code. I double checked it, the sender IP really has to be identical to the stored sender IP. So either someone managed to get your IP ( not through the server ) and forged a packet, or your client really changed its source port.
And here is where I'm lost again. If a real client port change is the reason, then at least one packet with the new port must have found its way from the client to the server. But then it would have been detected by the firewall and the incoming packet would have been accepted. I'd like a chain of socket breakage to explain this: server socket breaks because another client went offline unexpectedly, client socket breaks because the server socket was blocked for a short time, port change. But it just does not add up.

I'm sorry but this doesn't fit the facts as I see them at all Remember that the connection used to ask for the new connection used the connection you are saying failed... now does that sound logical? I don't think so and I can see why this puzzles you.

There were changes in the relevant code from 0.2.7.0 to 0.2.7.1, I'll review them for a possible explanation.

I don't want you to ruin your weekend on this, let's see the logs first.

Lucifer or Swampy, it could be insightful to se the server logs ( with cleaned IPs ) for these events, if that's possible. They should be visible as timeouts where n54 or Yaza were affected. If any of the reasoning above is correct, there also should be a message like
"changing port of user 5 from 1066 to 1088."
prior to the timeout.

Yay! Logs are great Swampy or Lucifer do not need to cleanse my IP I really don't mind and perhaps it could help as I don't have a static IP either (not that it ever seems to change unless I power down the router) so that way we can be sure nothing happened on my end either (or if you PM me the IP logged to me I can check it to whatever IP I have and see if they're the same).

I've turned on extra firewall logging of armagetron so that it might (hopefully) help shed light on things if it happens again

Whatever this is I'm sure it will help make armagetron better (or at least my understanding of armagetron lol)

[Z-Man]

My current vote would go for option 1 ( strange code triggered ). There are probably easier and more effective ways to launch attacks at either the server or its clients, just as you say.

Fair enough but this seems very strange when the request uses the actual connection my client in this case is supposed not to listen to to ask the client to open a new connection using a port (2809) that is way different to the normal armagetron ports. Would such an automatic failsafe have any reason to switch to using 2809? Instead of for example 4535?

What you call "normal" Armagetron port ( 4535, i assume ) is just the normal port on the server side. Only the server needs a socket with a known and published port number to be reachable. The client does not "bind" its socket to any specific port and gets a more or less random port number assigned by the OS. When the client initiates the connection, this port number gets automatically known tho the server so it knows how to send packets that find their way back. Fixing the clientside port does not work for two reasons: you would not be able to run two clients on one computer, and connections through a NAT router will not work because the client port gets altered by it.

I don't want you to ruin your weekend on this, let's see the logs first.

Good idea.

Now, wait a sec. n54, your network setup is: you have a router connected to the internet, your pc connects to the router. The router uses NAT. On your PC, there is an additional personal firewall that triggered the alarm. Right?
Now, how the hell got the packet the firewall complained about past the router? When a NAT router gets a packet from the internet, it tries to match it to previous outgoing connections. Only if it finds one ( or you activated port forwarding for the destination port ), the packet gets transmitted into the LAN, otherwise, it just is ignored, and that is what should have happened here. I either don't understand your firewall or I don't understand the router. What firewall do you use?