Ladle 97

A place for threads related to tournaments and the like, and things related too.

Moderator: Light

User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Re: Ladle 97

Post by Z-Man »

Sorry, still no recordings. I'll post them when I can start sorting them out before midnight.
User avatar
kyle
Reverse Outside Corner Grinder
Posts: 1876
Joined: Thu Jun 08, 2006 3:33 pm
Location: Indiana, USA, Earth, Milky Way Galaxy, Universe, Multiverse
Contact:

Re: Ladle 97

Post by kyle »

Just as a small note, I've upgraded KYLE's NYC, so test it out see how it works. (I'm currently on a horrible Internet connection, So I cannot really test)
Image
User avatar
Magi
Match Winner
Posts: 634
Joined: Fri Oct 21, 2011 9:35 pm

Re: Ladle 97

Post by Magi »

Z-Man wrote:Sorry, still no recordings. I'll post them when I can start sorting them out before midnight.
No worries, I can imagine trying to figure out what recording is what when we all had to constantly switch servers is hard.
Image Image Image Image Image Image Image Image Image
Image

bye
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Re: Ladle 97

Post by Z-Man »

And I uploaded all the throwaway in-between recordings into the directory: http://vps-zman.armagetronad.org/~manue ... s/ladle97/ in case you want to trawl them for heinous crimes against the idle chat gods. Or, more likely, in case I have forgotten to properly identify an important recording.
The x- ones are those where I mostly stalk the servers on the master list, you can ignore those.
Overrated
Match Winner
Posts: 483
Joined: Sun Feb 21, 2010 8:32 am

Re: Ladle 97

Post by Overrated »

Alright, I looked at two recordings (the semis in z-mans, about 1660s is the time of the attack), and the beginning of the finals in compger (has the argument with m&m). I did not watch the first recording of R and m&m, but what is missing (and might be in another recording pre-attack) is the amount of time dedicated to eventually changing servers. Here's some facts:
  • Magi at one point said "Keep playing until we lag out." This is important information. It seems like there could have been confusion after this. I do not recall this being said, but the recordings prove otherwise.

    After the round after the attack started, only DGM and Magi (iirc, these are mental notes, but he also contradicted himself from point 1) had said pause from R. That's only one half of R's team leaders, fair assumption is Red agreed (I can guarantee he said so in teamspeak at the least). DGM eventually asked what server we were going to after a round or so of the server being attacked.

    Initially, Koala, Eber, and Apple all said "paus(e)" at one point or another, Koala furthering up with a score, and Eber saying "unpause," but in context it looks like everyone is just saying a lot of random stuff. Mr happens to disagree and says no immediately, this obviously would lead to confusion, Apple and Mr both saying no puts both teams in a question of whether to continue. After the initial round, algid said "90-70" at one point. This is the fourth person from m&m saying that there was a pause in one way, shape, or form.

    Both sides argue in the new server and R starts the match early. Apple, in response, starts a new match immediately at the start of the following round. R proceeds to win both rounds while m&m only has 4. Arguing commences, eventually Magi kicks m&m players unaware of a 5 minute ban.

    Gazelle eventually silences certain m&m players who were not /shouting. Apple just joined the server and was silenced.

    The chat was absolutely horrendous in terms of the "jokes" and "attacks" towards Magi and his character in the finals. It went on until at least the end of the second match, and I stopped watching after that. I find this unnecessary, regardless of the circumstances. I understand being upset, but come on.
Where does this bring us? Now we have four m&m saying pause, and one saying yes, one of which, and it's up to discretion whether this should matter, being Mr who showed up one round prior to the attack (I think this information should at least be noted). Based on teamspeak, and I have no proof of this other than my word (which I hope you can trust), everyone in R was immediately saying "pause" after the initial attack starting lagging the server, even Magi, although I cannot deny what he said on point 1.

So what's the verdict on this issue? Keep in mind this is not the main issue, but should be addressed.

Magi says his mod powers should be taken away, at the very least temporarily. This I agree with currently given the circumstances. I think trying to find a time to replay the matches would be worth considering at another moment. I think this is theoretically impossible, however, given people's schedules, but if it makes things right I would be willing. Another issue is if m&m happens to win, then Rd would have to play them in order to play for ladle. I think the rest is up to debate. I don't think Magi did anything that warrants a ladle ban, but at the very least deserves taken away mod rights, and potentially team leader for some sort of length of time. I feel this is up to the community.

At this point, everyone feels Magi was a little excessive, but he also had me and others pushing him to kick and to start the matches against Rd. I apologize in that regard, and again will cite the fact that the amount of delays in this ladle eventually was the main reason for what happened, but it's not an excuse. I hope we can put this incident behind us after the proper actions have been done, and now I'm officially done debating the matter.



Now, the main issue is the DDoS attacks and how we can stop them. One of the things mentioned is getting the server off the master list, that very well might be the best option but is there anything else we can really do?
BRAWL dead. RIP.

Fort is like a box of knives, you never know when you're going to be cut.
User avatar
ConVicT
Shutout Match Winner
Posts: 1001
Joined: Fri Feb 17, 2012 2:33 am

Re: Ladle 97

Post by ConVicT »

Ban the people that we know for a fact are doing it?
User avatar
Magi
Match Winner
Posts: 634
Joined: Fri Oct 21, 2011 9:35 pm

Re: Ladle 97

Post by Magi »

ConVicT wrote:Ban the people that we know for a fact are doing it?
All they need to ddos is the server information which is pretty public which is why an underground ladle where the servers are hidden and only accessible from custom connect is being brought up. I think at the very least one or two servers should be set up this way just in case.
Image Image Image Image Image Image Image Image Image
Image

bye
User avatar
breeze
Average Program
Posts: 79
Joined: Sat May 26, 2012 4:33 pm
Location: http://armagetronad.net

Re: Ladle 97

Post by breeze »

While in theory the idea of having hidden servers makes sense, I think it would be cumbersome to have to tell 50-100 people server information. Even then you run the risk of someone leaking the information out anyway. If I'm understanding everything correctly, I don't think it's a terribly feasible option.

Can anyone even confirm with certainty that these attacks are actually occuring as stated?

If they are occuring, are they hitting the servers with legitimate traffic? If they are not couldn't we just set up iptables on the servers to deny all traffic except traffic on the legitimate ports? Alternatively, we could use iptables to deny traffic from offenders' IP addresses.
User avatar
sinewav
Graphic Artist
Posts: 6413
Joined: Wed Jan 23, 2008 3:37 am
Contact:

Re: Ladle 97

Post by sinewav »

I have a couple comments and I want them to be heard as an impartial bystander, not as someone who is picking a side, because I am not, and I want my words to be free of the weight they have been given in the past. None of the usual "sine said this so that means something, something."

First, I think it is important to find at least one backup server off the master list. Now that I have steady income again I would be willing to pay a small amount toward hosting it. If gene, Light, duke, or any other hosts want to set this up, please contact me. Also, we need to consider at least the following points:
  1. It needs to be on a unique network not currently hosting Arma servers.
  2. We need a trustworthy volunteer to keep the address safe and only give it out the the affected Team Leaders at the last possible second. Then, Team Leaders should share the address with Team Members only. (TODO: How to verify over IRC?) If you are thinking this will keep spectators out, think again. Once the match starts, the server's IP will surely be leaked and then it's a race against time to finish the Ladle before a new wave of attacks. Does this mean fewer people can see some Ladle matches? Probably, but this is a small price to pay.
  3. Once the server is discovered (it will be), the backup server needs to move to another unique network for the following month. This will be a huge pain in the arse.
Second, because the match between Rogue Tronners and m&m was not only close but apparently mishandled, it may put the community at ease if there were a rematch. However, few players have enough time to do such a thing. We barely have time to meet once a month.

My suggestion, which should be a one-time thing and in no way sets a precedent for future resolutions except in extreme cases, is to manually edit the brackets for next Ladle, putting Rogue and m&m together in the opening round and the winner meeting Rd in the Semi-Finals, then randomizing the rest.

I don't think a one time manual edit would destroy the integrity of Ladle or make next month's event any less fun as long as everyone agrees that doing so is part of the healing process. The real reason we are unhappy is not because of Magi or Mr or any other player. We are unhappy because we love good, solid competition and that has been ruined from an outside source. Don't take it out on each other.
User avatar
ConVicT
Shutout Match Winner
Posts: 1001
Joined: Fri Feb 17, 2012 2:33 am

Re: Ladle 97

Post by ConVicT »

breeze wrote:Even then you run the risk of someone leaking the information out anyway.
+1
That would definitely happen.
I imagine there'd be sign ups from people who don't even play the tourney just to get said info.
User avatar
Magi
Match Winner
Posts: 634
Joined: Fri Oct 21, 2011 9:35 pm

Re: Ladle 97

Post by Magi »

ConVicT wrote:
breeze wrote:Even then you run the risk of someone leaking the information out anyway.
+1
That would definitely happen.
I imagine there'd be sign ups from people who don't even play the tourney just to get said info.
Well I think if we made all servers underground and hidden, team leaders would only get info to the server they're playing in, and when problems arise or happen in them it's easier to limit down where the leak of information is happening, again it wouldn't be an easy process. Something like Server hosts --> Bracket/Authorities maker --> Team Leaders --> Team members.
Image Image Image Image Image Image Image Image Image
Image

bye
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Re: Ladle 97

Post by Z-Man »

ConVicT wrote:Ban the people that we know for a fact are doing it?
The nature of the attacks means they can't be traced to the origin unless you bust the rented botnet. So we probably can never prove conclusively who actually did it. Now, people who brag they did it, people who threatened to do it, yes. Share the server logs, publish their IPs, ban them.
breeze wrote:While in theory the idea of having hidden servers makes sense, I think it would be cumbersome to have to tell 50-100 people server information.
You'd just tell the team leaders (the connection data for all servers) and instruct them to share the IPs with their team only as needed.

Would anyone be willing to look into cloud hosting? With Amazon EC2, for example, you can rent virtual servers by the hour. On paper, that looks perfect for event servers.
breeze wrote:Can anyone even confirm with certainty that these attacks are actually occuring as stated?
Yep. Here's how my server saw the two attacks.
Symptom 1: Without further log messages, the server went into turtle mode. No individual IPs were blocked for excessive pinging, that sort of confirms the attack came from many sources.
Symptom 2: CPU load of the server process jumped to 100%
Symptom 3: Everyone lagged, even the recording client saw them lagging.
Non-symptom: Despite this, regular ICMP pings showed no big anomalies. Higher than normal packet loss, but not too much.
Symptom 4: After a bit, the server vanished from the net completely. Pings stopped, open ssh connections froze, new ssh connections timed out, every player lagged out at once.
1 and 2 together mean that the attack consisted of legitimate UDP packages to the game server. The high CPU load, or at least its impact on lag, would have been avoidable; "CONNECTION_LIMIT 5" would have limited the time the server would have spent processing attack packets. It can distinguish between them and the packets from established players early on. I'll change the defaults accordingly.
3 could be a direct consequence of 2 or just the effect of the entire input pipe getting clogged up. Though, since even the recorder saw horrible temporary inconsistencies and that one almost exclusively depends on the output pipe.
Annoyingly, to improve performance, I did not do a serverside recording of the attacks. So I don't know what was sent precisely and what it triggered on the server. It could be that it provoked a response and also clogged the output pile that way.
Anyway, my guess based on the almost regular ICMP-pings would be that the high CPU load, not clogged pipes, were responsible for the perceived lag. Maybe they even found a new way to cause high CPU load with relatively little effort.

All that said: I can only assume right now number 4 means that an automatic system on the hoster detected the attack and disconnected my VPS to protect the other machines running on the same host or even the entire network. That would have happened no matter what. Nothing much can be done against that except for renting a better, more expensive server.
breeze wrote:If they are not couldn't we just set up iptables on the servers to deny all traffic except traffic on the legitimate ports? Alternatively, we could use iptables to deny traffic from offenders' IP addresses.
The thing about a massive DDoS is, if they reach your machine, you have already lost. Your network interface gets saturated no matter what.
User avatar
echo.bot
On Lightcycle Grid
Posts: 25
Joined: Mon Mar 24, 2014 9:59 pm
Location: Fairfield, IA
Contact:

Re: Ladle 97

Post by echo.bot »

Overrated wrote:Alright, I looked at two recordings (the semis in z-mans, about 1660s is the time of the attack), and the beginning of the finals in compger (has the argument with m&m). I did not watch the first recording of R and m&m, but what is missing (and might be in another recording pre-attack) is the amount of time dedicated to eventually changing servers. Here's some facts:
  • Magi at one point said "Keep playing until we lag out." This is important information. It seems like there could have been confusion after this. I do not recall this being said, but the recordings prove otherwise.

    After the round after the attack started, only DGM and Magi (iirc, these are mental notes, but he also contradicted himself from point 1) had said pause from R. That's only one half of R's team leaders, fair assumption is Red agreed (I can guarantee he said so in teamspeak at the least). DGM eventually asked what server we were going to after a round or so of the server being attacked.

    Initially, Koala, Eber, and Apple all said "paus(e)" at one point or another, Koala furthering up with a score, and Eber saying "unpause," but in context it looks like everyone is just saying a lot of random stuff. Mr happens to disagree and says no immediately, this obviously would lead to confusion, Apple and Mr both saying no puts both teams in a question of whether to continue. After the initial round, algid said "90-70" at one point. This is the fourth person from m&m saying that there was a pause in one way, shape, or form.

    Both sides argue in the new server and R starts the match early. Apple, in response, starts a new match immediately at the start of the following round. R proceeds to win both rounds while m&m only has 4. Arguing commences, eventually Magi kicks m&m players unaware of a 5 minute ban.

    Gazelle eventually silences certain m&m players who were not /shouting. Apple just joined the server and was silenced.

    The chat was absolutely horrendous in terms of the "jokes" and "attacks" towards Magi and his character in the finals. It went on until at least the end of the second match, and I stopped watching after that. I find this unnecessary, regardless of the circumstances. I understand being upset, but come on.
Where does this bring us? Now we have four m&m saying pause, and one saying yes, one of which, and it's up to discretion whether this should matter, being Mr who showed up one round prior to the attack (I think this information should at least be noted). Based on teamspeak, and I have no proof of this other than my word (which I hope you can trust), everyone in R was immediately saying "pause" after the initial attack starting lagging the server, even Magi, although I cannot deny what he said on point 1.

So what's the verdict on this issue? Keep in mind this is not the main issue, but should be addressed.

Magi says his mod powers should be taken away, at the very least temporarily. This I agree with currently given the circumstances. I think trying to find a time to replay the matches would be worth considering at another moment. I think this is theoretically impossible, however, given people's schedules, but if it makes things right I would be willing. Another issue is if m&m happens to win, then Rd would have to play them in order to play for ladle. I think the rest is up to debate. I don't think Magi did anything that warrants a ladle ban, but at the very least deserves taken away mod rights, and potentially team leader for some sort of length of time. I feel this is up to the community.

At this point, everyone feels Magi was a little excessive, but he also had me and others pushing him to kick and to start the matches against Rd. I apologize in that regard, and again will cite the fact that the amount of delays in this ladle eventually was the main reason for what happened, but it's not an excuse. I hope we can put this incident behind us after the proper actions have been done, and now I'm officially done debating the matter.



Now, the main issue is the DDoS attacks and how we can stop them. One of the things mentioned is getting the server off the master list, that very well might be the best option but is there anything else we can really do?
How to address the DDoS? Better protection. Implement multiple levels of security on the servers. Blah blah blah. Anyhow. I put up a vps based on Debian that is as locked down as I can get it both with hardware and software firewalls (IPTables yadda yadda) and I am using dyDNS. But to truly stop the issue the devs would have to rework how things work between the servers, the Master servers and the client. Two things I think should be implemented:

1. A server delivers IP, port and all relevant information. This is kept on the back end/Server to Master Server side. Servers are then served up and listed using alpha numeric naming the same way they are now but with out revealing the IP information of the game servers. This was poor planning on the part of the developers and was a ticking time bomb. Anyways this system would work similar to the way DNS serves a text based address in place of an IP address.

2: To prevent people from spoofing servers and stealing logins etc, everyone who hosts a server(s) would need a private key issued to them that would be sent to the Master Server before it can be listed on the Master Server List.

I know this is a lot but it is the only real solution in my opinion other than hunting the assholes down who attack the servers and chopping their nuts off.
Post Reply