INW's logs from Ladle 49

[INW]

INW: where there real autobans (that shouldn't happen) or did everyone just time out?

I'll take a look at the logs and copy them here shortly.

[INW]

Here is what is on the logs:

Code: Select all

*** glibc detected *** /home/armagetronad/bin/armagetronad-dedicated: malloc(): memory corruption (fast): 0x0000000006fe57bf ***
======= Backtrace: =========
/lib/libc.so.6(+0x71ad6)[0x2b0e492a2ad6]
/lib/libc.so.6(+0x753ed)[0x2b0e492a63ed]
/lib/libc.so.6(__libc_malloc+0x70)[0x2b0e492a7930]
/usr/lib/libstdc++.so.6(_Znwm+0x1d)[0x2b0e48b4f6bd]
/usr/lib/libstdc++.so.6(_Znam+0x9)[0x2b0e48b4f7d9]
/home/armagetronad/bin/armagetronad-dedicated[0x55200d]
/home/armagetronad/bin/armagetronad-dedicated[0x523f6f]
/home/armagetronad/bin/armagetronad-dedicated[0x5240a2]
/home/armagetronad/bin/armagetronad-dedicated[0x4b86c5]
/home/armagetronad/bin/armagetronad-dedicated[0x41193b]
/home/armagetronad/bin/armagetronad-dedicated[0x518e1f]
/home/armagetronad/bin/armagetronad-dedicated[0x432977]
/home/armagetronad/bin/armagetronad-dedicated[0x4329a1]
/home/armagetronad/bin/armagetronad-dedicated[0x437543]
/home/armagetronad/bin/armagetronad-dedicated[0x43b2cb]
/home/armagetronad/bin/armagetronad-dedicated[0x40a6a5]
/lib/libc.so.6(__libc_start_main+0xfd)[0x2b0e4924fc4d]
/home/armagetronad/bin/armagetronad-dedicated[0x406b19]
======= Memory map: ========
00400000-005db000 r-xp 00000000 fd:00 464453638                          /home/armagetronad/bin/armagetronad-dedicated
007db000-007dd000 rw-p 001db000 fd:00 464453638                          /home/armagetronad/bin/armagetronad-dedicated
007dd000-0083b000 rw-p 007dd000 00:00 0 
06d23000-07366000 rw-p 06d23000 00:00 0                                  [heap]
2b0e47e91000-2b0e47eaf000 r-xp 00000000 fd:00 445679275                  /lib/ld-2.11.2.so
2b0e47eaf000-2b0e47ed8000 rw-p 2b0e47eaf000 00:00 0 
2b0e480ae000-2b0e480af000 r--p 0001d000 fd:00 445679275                  /lib/ld-2.11.2.so
2b0e480af000-2b0e480b0000 rw-p 0001e000 fd:00 445679275                  /lib/ld-2.11.2.so
2b0e480b0000-2b0e480b1000 rw-p 2b0e480b0000 00:00 0 
2b0e480b1000-2b0e480c8000 r-xp 00000000 fd:00 445679264                  /lib/libpthread-2.11.2.so
2b0e480c8000-2b0e482c7000 ---p 00017000 fd:00 445679264                  /lib/libpthread-2.11.2.so
2b0e482c7000-2b0e482c8000 r--p 00016000 fd:00 445679264                  /lib/libpthread-2.11.2.so
2b0e482c8000-2b0e482c9000 rw-p 00017000 fd:00 445679264                  /lib/libpthread-2.11.2.so
2b0e482c9000-2b0e482cd000 rw-p 2b0e482c9000 00:00 0 
2b0e482cd000-2b0e48413000 r-xp 00000000 fd:00 445779006                  /usr/lib/libxml2.so.2.7.8
2b0e48413000-2b0e48612000 ---p 00146000 fd:00 445779006                  /usr/lib/libxml2.so.2.7.8
2b0e48612000-2b0e4861c000 rw-p 00145000 fd:00 445779006                  /usr/lib/libxml2.so.2.7.8
2b0e4861c000-2b0e4861e000 rw-p 2b0e4861c000 00:00 0 
2b0e4861e000-2b0e48679000 r-xp 00000000 fd:00 445781230                  /usr/lib/libZThread-2.3.so.2.0.0
2b0e48679000-2b0e48879000 ---p 0005b000 fd:00 445781230                  /usr/lib/libZThread-2.3.so.2.0.0
2b0e48879000-2b0e4887c000 rw-p 0005b000 fd:00 445781230                  /usr/lib/libZThread-2.3.so.2.0.0
2b0e4887c000-2b0e48883000 r-xp 00000000 fd:00 445679265                  /lib/librt-2.11.2.so
2b0e48883000-2b0e48a82000 ---p 00007000 fd:00 445679265                  /lib/librt-2.11.2.so
2b0e48a82000-2b0e48a83000 r--p 00006000 fd:00 445679265                  /lib/librt-2.11.2.so
2b0e48a83000-2b0e48a84000 rw-p 00007000 fd:00 445679265                  /lib/librt-2.11.2.so
2b0e48a84000-2b0e48b7a000 r-xp 00000000 fd:00 445777631                  /usr/lib/libstdc++.so.6.0.13
2b0e48b7a000-2b0e48d7a000 ---p 000f6000 fd:00 445777631                  /usr/lib/libstdc++.so.6.0.13
2b0e48d7a000-2b0e48d81000 r--p 000f6000 fd:00 445777631                  /usr/lib/libstdc++.so.6.0.13
2b0e48d81000-2b0e48d83000 rw-p 000fd000 fd:00 445777631                  /usr/lib/libstdc++.so.6.0.13
2b0e48d83000-2b0e48d99000 rw-p 2b0e48d83000 00:00 0 
2b0e48d99000-2b0e48e19000 r-xp 00000000 fd:00 445679272                  /lib/libm-2.11.2.so
2b0e48e19000-2b0e49019000 ---p 00080000 fd:00 445679272                  /lib/libm-2.11.2.so
2b0e49019000-2b0e4901a000 r--p 00080000 fd:00 445679272                  /lib/libm-2.11.2.so
2b0e4901a000-2b0e4901b000 rw-p 00081000 fd:00 445679272                  /lib/libm-2.11.2.so
2b0e4901b000-2b0e49031000 r-xp 00000000 fd:00 445679298                  /lib/libgcc_s.so.1
2b0e49031000-2b0e49230000 ---p 00016000 fd:00 445679298                  /lib/libgcc_s.so.1
2b0e49230000-2b0e49231000 rw-p 00015000 fd:00 445679298                  /lib/libgcc_s.so.1
2b0e49231000-2b0e49389000 r-xp 00000000 fd:00 445679263                  /lib/libc-2.11.2.so
2b0e49389000-2b0e49588000 ---p 00158000 fd:00 445679263                  /lib/libc-2.11.2.so
2b0e49588000-2b0e4958c000 r--p 00157000 fd:00 445679263                  /lib/libc-2.11.2.so
2b0e4958c000-2b0e4958d000 rw-p 0015b000 fd:00 445679263                  /lib/libc-2.11.2.so
2b0e4958d000-2b0e49593000 rw-p 2b0e4958d000 00:00 0 
2b0e49593000-2b0e49595000 r-xp 00000000 fd:00 445679262                  /lib/libdl-2.11.2.so
2b0e49595000-2b0e49795000 ---p 00002000 fd:00 445679262                  /lib/libdl-2.11.2.so
2b0e49795000-2b0e49796000 r--p 00002000 fd:00 445679262                  /lib/libdl-2.11.2.so
2b0e49796000-2b0e49797000 rw-p 00003000 fd:00 445679262                  /lib/libdl-2.11.2.so
2b0e49797000-2b0e497ae000 r-xp 00000000 fd:00 445777730                  /usr/lib/libz.so.1.2.3.4
2b0e497ae000-2b0e499ad000 ---p 00017000 fd:00 445777730                  /usr/lib/libz.so.1.2.3.4
2b0e499ad000-2b0e499ae000 rw-p 00016000 fd:00 445777730                  /usr/lib/libz.so.1.2.3.4
2b0e499ae000-2b0e499b2000 rw-p 2b0e499ae000 00:00 0 
2b0e499b2000-2b0e499bd000 r-xp 00000000 fd:00 445679260                  /lib/libnss_files-2.11.2.so
2b0e499bd000-2b0e49bbc000 ---p 0000b000 fd:00 445679260                  /lib/libnss_files-2.11.2.so
2b0e49bbc000-2b0e49bbd000 r--p 0000a000 fd:00 445679260                  /lib/libnss_files-2.11.2.so
2b0e49bbd000-2b0e49bbe000 rw-p 0000b000 fd:00 445679260                  /lib/libnss_files-2.11.2.so
2b0e49bbe000-2b0e49bc3000 r-xp 00000000 fd:00 445679277                  /lib/libnss_dns-2.11.2.so
2b0e49bc3000-2b0e49dc2000 ---p 00005000 fd:00 445679277                  /lib/libnss_dns-2.11.2.so
2b0e49dc2000-2b0e49dc3000 r--p 00004000 fd:00 445679277                  /lib/libnss_dns-2.11.2.so
2b0e49dc3000-2b0e49dc4000 rw-p 00005000 fd:00 445679277                  /lib/libnss_dns-2.11.2.so
2b0e49dc4000-2b0e49dd7000 r-xp 00000000 fd:00 445679256                  /lib/libresolv-2.11.2.so
2b0e49dd7000-2b0e49fd6000 ---p 00013000 fd:00 445679256                  /lib/libresolv-2.11.2.so
2b0e49fd6000-2b0e49fd7000 r--p 00012000 fd:00 445679256                  /lib/libresolv-2.11.2.so
2b0e49fd7000-2b0e49fd8000 rw-p 00013000 fd:00 445679256                  /lib/libresolv-2.11.2.so
2b0e49fd8000-2b0e49fda000 rw-p 2b0e49fd8000 00:00 0 
2b0e49fda000-2b0e49fdb000 ---p 2b0e49fda000 00:00 0 
2b0e49fdb000-2b0e4a7db000 rw-p 2b0e49fdb000 00:00 0 
2b0e4a7db000-2b0e4a7dc000 ---p 2b0e4a7db000 00:00 0 
2b0e4a7dc000-2b0e4afdc000 rw-p 2b0e4a7dc000 00:00 0 
2b0e4afdc000-2b0e4afdd000 ---p 2b0e4afdc000 00:00 0 
2b0e4afdd000-2b0e4b7dd000 rw-p 2b0e4afdd000 00:00 0 
2b0e4b7dd000-2b0e4b7de000 ---p 2b0e4b7dd000 00:00 0 
2b0e4b7de000-2b0e4bfde000 rw-p 2b0e4b7de000 00:00 0 
2b0e4c000000-2b0e4c621000 rw-p 2b0e4c000000 00:00 0 
2b0e4c621000-2b0e50000000 ---p 2b0e4c621000 00:00 0 
2b0e50045000-2b0e50092000 rw-p 2b0e50045000 00:00 0 
7fffd0e11000-7fffd0e26000 rw-p 7ffffffe9000 00:00 0                      [stack]
7fffd0f03000-7fffd0f05000 r-xp 7fffd0f03000 00:00 0                      [vdso]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0                  [vsyscall]
/home/armagetronad/scripts/srv: line 16:  1594 Aborted                 $loc/bin/armagetronad-dedicated --vardir $loc/servers/$1/var
Server has crashed. It will restart immediately..., press CTRL-C to cancel

"memory corruption"
Seems like a memory problem? Not enough RAM?

[INW]

I was also getting a ton of server timeouts in between rounds like this:

Code: Select all

Timeout! Reason:
[0] User 28 does not know about netobject 7450 ( gCycle nr. 7450 owned by ePlayerNetID nr. 4093, name \\sine.wav ).
[0] User 28 does not know about netobject 7449 ( gCycle nr. 7449 owned by ePlayerNetID nr. 4262, name \\eXzz ).
[0] User 28 does not know about netobject 7448 ( gCycle nr. 7448 owned by ePlayerNetID nr. 4173, name \\xNVY.wav ).
[0] User 28 does not know about netobject 7447 ( gCycle nr. 7447 owned by ePlayerNetID nr. 4387, name \\spin ).
[0] User 28 does not know about netobject 7446 ( gCycle nr. 7446 owned by ePlayerNetID nr. 3924, name \\akira ).
[0] User 28 does not know about netobject 7445 ( gCycle nr. 7445 owned by ePlayerNetID nr. 4004, name \\liz ).
[0] User 28 does not know about netobject 7444 ( gCycle nr. 7444 owned by ePlayerNetID nr. 4133, name ct-wap ).
[0] User 28 does not know about netobject 7443 ( gCycle nr. 7443 owned by ePlayerNetID nr. 4633, name CTxZaP ).
[0] User 28 does not know about netobject 7442 ( gCycle nr. 7442 owned by ePlayerNetID nr. 3884, name ct_Cronix ).
[0] User 28 does not know about netobject 7441 ( gCycle nr. 7441 owned by ePlayerNetID nr. 4217, name ct|Puuquie ).
[0] User 28 does not know about netobject 7440 ( gCycle nr. 7440 owned by ePlayerNetID nr. 4593, name cT-GaZ ).
[0] User 28 does not know about netobject 7439 ( gCycle nr. 7439 owned by ePlayerNetID nr. 3964, name CtxWoned ).
[0] User 28 does not know about netobject 7438 ( Nameless NetObject nr. 7438 ).
[0] User 28 does not know about netobject 7437 ( Nameless NetObject nr. 7437 ).
[0] User 28 does not know about netobject 7341 ( ePlayerNetID nr. 7341, name CT~Voodoo ).
[0] User 28 does not know about netobject 6187 ( ePlayerNetID nr. 6187, name \\Dimebag ).
[0] User 28 does not know about netobject 5049 ( ePlayerNetID nr. 5049, name uNk.House ).
[0] User 28 does not know about netobject 4989 ( ePlayerNetID nr. 4989, name cTxMB53 ).
[0] User 28 does not know about netobject 4948 ( ePlayerNetID nr. 4948, name free kill ).
[0] User 28 does not know about netobject 4842 ( ePlayerNetID nr. 4842, name J-Dawg ).
[0] User 28 does not know about netobject 4796 ( ePlayerNetID nr. 4796, name .pRu|Word ).
[0] User 28 does not know about netobject 4713 ( ePlayerNetID nr. 4713, name -*kült*- ).
[0] User 28 does not know about netobject 4633 ( ePlayerNetID nr. 4633, name CTxZaP ).
[0] User 28 does not know about netobject 4593 ( ePlayerNetID nr. 4593, name cT-GaZ ).
[0] User 28 does not know about netobject 4553 ( ePlayerNetID nr. 4553, name CT_emphasis ).
[0] User 28 does not know about netobject 4511 ( ePlayerNetID nr. 4511, name ct|pike ).
[0] User 28 does not know about netobject 4471 ( ePlayerNetID nr. 4471, name CT_emmy ).
[0] User 28 does not know about netobject 4429 ( ePlayerNetID nr. 4429, name \\toy ).
[0] User 28 does not know about netobject 4387 ( ePlayerNetID nr. 4387, name \\spin ).
[0] User 28 does not know about netobject 4343 ( ePlayerNetID nr. 4343, name ct|safari ).
[0] User 28 does not know about netobject 4303 ( ePlayerNetID nr. 4303, name Defender ).
[0] User 28 does not know about netobject 4262 ( ePlayerNetID nr. 4262, name \\eXzz ).
[0] User 28 does not know about netobject 4217 ( ePlayerNetID nr. 4217, name ct|Puuquie ).
[0] User 28 does not know about netobject 4173 ( ePlayerNetID nr. 4173, name \\xNVY.wav ).
[0] User 28 does not know about netobject 4133 ( ePlayerNetID nr. 4133, name ct-wap ).
[0] User 28 does not know about netobject 4093 ( ePlayerNetID nr. 4093, name \\sine.wav ).
[0] User 28 does not know about netobject 4053 ( ePlayerNetID nr. 4053, name Orion ).
[0] User 28 does not know about netobject 4010 ( Team Team gold ).
[0] User 28 does not know about netobject 4005 ( Team Team blue ).
[0] User 28 does not know about netobject 4004 ( ePlayerNetID nr. 4004, name \\liz ).
[0] User 28 does not know about netobject 3964 ( ePlayerNetID nr. 3964, name CtxWoned ).
[0] User 28 does not know about netobject 3924 ( ePlayerNetID nr. 3924, name \\akira ).
[0] User 28 does not know about netobject 3884 ( ePlayerNetID nr. 3884, name ct_Cronix ).
[0] User 28 does not know about netobject 3844 ( Nameless NetObject nr. 3844 ).
[0] User 28 does not know about netobject 3843 ( Nameless NetObject nr. 3843 ).
[0]

[INW]

I also got this when entering turtle mode:

Code: Select all

[0] Server is under heavy load. Turtle mode activated. Communication with new clients is on a budget.
[0] Flood protection bans 108.4.183.135 after 10 pings in 0.14 seconds.
[0] Flood protection bans 109.90.142.64 after 10 pings in 0.1 seconds.
[0] Flood protection bans 174.99.27.8 after 10 pings in 0.19 seconds.
[0] Flood protection bans 98.119.178.15 after 10 pings in 0.15 seconds.
[0] Flood protection bans 82.74.66.227 after 10 pings in 0.19 seconds.
[0] Flood protection bans 24.160.139.12 after 20 pings in 0.52 seconds.
[101] Answering ping poll from 68.232.117.222:64691.
[0] Flood protection bans 193.153.105.175 after 50 pings in 3.95 seconds.
[101] Answering ping poll from 184.100.235.8:60024.
[0] Flood protection bans 76.110.145.144 after 100 pings in 9 seconds.
[101] Answering ping poll from 201.246.89.83:27704.
[101] Answering ping poll from 67.86.150.191:49764.
[101] Answering ping poll from 67.86.150.191:49764.
[101] Answering ping poll from 67.86.150.191:49764.
[101] Answering ping poll from 67.86.150.191:49764.
[101] Answering ping poll from 67.86.150.191:49764. This is the last logged poll.

Just too much load for the server to handle I guess.

[Z-Man]

MAXCLIENTS 100 may be a bit optimistic

The timeout thing is normal, that's just user 28 having trouble.

Did turtle mode and the ping flood bans happen after the crash? If so, they're simply the poor dropped clients trying to keep up the connection. And the ban doesn't ban them from the server completely, it just makes the server not respond in the server browser, and it times out after a minute or so.

And then the cause is would simply be crash. Yeah, it could be the server running out of memory, but that's unlikely unless there is a leak we don't know about (which is unlikely, too) It could also just be just a bug. Unfortunately, the backtrace is pretty much meaningless in this form Did you compile the server yourself? If so, could you compile another one in the exact same way (same source, same compiler, same settings), just with CFLAGS=-g and CXXFLAGS=-g added, then send the result over?

And even if that gives a meaningful backtrace, memory corruption bugs are notorious for triggering crashes far away from their cause, so I don't know how much good it'll do. But it's worth a shot.

[Titanoboa]

I timed out in the end of finals btw. It didn't feel good to miss the end of the finals even though we were almost beaten when it happened.

[INW]

When recompiling, should I overwrite the current version I have on the vps or create a new folder, keeping everything I have?

[Z-Man]

Make a new directory. You want to keep the old one to rebuild the server on source upgrades.

[INW]

Ya, I said folder >windows newb<

I'll do that soon.

Also, how with the backtrace show up in the logs with this? Does the server have to crash to show those logs?

[Z-Man]

I don't know I don't know how that particular backtrace is generated, so I don't know wheter having debug symbols available if it happens again will change anything in the immediate output. But what I do know is that there is some way to convert the hex addresses to code file/function/line number information manually, I'm going to try that with the backtrace we already have. May work, may fail. It's not guaranteed that the second build will result in code at the exact same addresses.

[teen]

seems like a syncing limit occurred if that makes any sense.

[epsy]

Just happened in fort6 and ignored one of the (playing) clients on latest trunk+boostthread+warmup. We also had it happen on latest trunk in 2v2v2v2.

Maybe MB53/ct|kyle and Durka can post logs.

[dariv]

Oh was that me? My client crashed at about that time yesterday when we were starting a 6v6 pickup. Chaos ensued!