Crash reporting

[Vitty]

Armagetron's debug recording feature isn't the most useful due to people not usually recording when arma crashes, and not always being able to reproduce the crash when they are recording. It could do with some sort of crash handler to generate dumps for people to upload. (Or have tron upload them automatically somewhere when a crash occurs.)

I've had a quick look at Google Breakpad which looks suitable, but are there any other suggestions of what could be used?

[Z-Man]

This certainly would help Windows and Mac clients. I'm less convinced it's useful for Linux; many people compile the client themselves there or use distribution supplied binaries where we wouldn't have the fitting debug information. And the whole thing is quite orthogonal to our debug recordings; debug recordings don't catch the crash itself very well, but if they catch something, they also record what happens before the crash, which usually would be more important. Most crashes are just final consequences of the program getting itself into a strange and unexpected state.

[Vitty]

I'm less convinced it's useful for Linux; many people compile the client themselves there

Would more frequent releases not help partially solve this problem, if you can call it that? I was reading through the release process for 0.3.1 and it seems as if it was very drawn out. From what I understand, 0.[EVEN] are stable releases and 0.[ODD] are development releases. Releasing alpha's for a development version doesn't make much sense to me :s

Would something similar to what WINE do be a better idea? where a new development release gets put out every 2 weeks? (Although obviously, 2 weeks would be too close together given the activity of development on arma).

use distribution supplied binaries where we wouldn't have the fitting debug information.

As distributions tend to only upload the stable releases that hopefully wouldn't be too much of a problem.

[Vitty]

Do you have a preference on where the dumps should be stored? I was thinking of either creating a new dump folder or just saving them inside the var folder.

[Z-Man]

More frequent releases would help, but they won't make the problem go away. It's easier to maintain an up to date build on your machine from bzr source on most distributions, and I guess even small differences in build systems cause the whole 'distribute stripped binaries, keep debug symbols for yourself for later' system to break down. On the popular Ubuntu, we have the PPA method which can bring updates to players in binary form just like regular updates, but the build there is done on remote compile farms and differs for the various supported Ubuntu versions out there. I just don't think we'll get enough usable data out of it to make it worth the effort. The crashes we can fix will happen on all platforms anyway.

And, yeah, releases are a lot of work. Much is automated, but many things can't be (uploads to Launchpad, booting up Windows for the build, waiting for the build to finish). No way we can get to a regular, permanent schedule.

The var folder is perfectly fine.

[Vitty]

Actually implementing a simple crash dump is only a few lines added to the code so there's no problem of it being worth the effort. (Locally, I've already implemented it for linux.) I could add a switch to ./configure that enables breakpad (it'd be disabled by default) and any time a release is built, it could be enabled. That way, anytime a dump is submitted you could be sure the symbols are available. (Keep a branch on LP with updated symbols for new releases?). For the ubuntu PPA, it's possible to produce breakpad symbols so we'd have them for those builds aswell.

As for actual releases, the process could be even more automated. autopackage (or Listaller), rpms and deb should require no interaction what so ever except the execution and a final sign off by the person releasing. The windows build could also be completely automated if build issues are maintained during development. I could develop some scripts to make that possible fairly soon but it'd take some time for Mac as i'm unfamiliar with the build system.

[Z-Man]

Actually implementing a simple crash dump is only a few lines added to the code so there's no problem of it being worth the effort. (Locally, I've already implemented it for linux.) I could add a switch to ./configure that enables breakpad (it'd be disabled by default) and any time a release is built, it could be enabled.

If it's so easy, sure, go ahead, I'll merge it. Can't hurt, can it? Can you add a test to ./configure that checks for the presence of the tools?

For the ubuntu PPA, it's possible to produce breakpad symbols so we'd have them for those builds aswell.

It is? I figured you'd have to integrate the generation of the symbols into the .deb build process and have them packed into a separate .deb you later have to download.

As for actual releases, the process could be even more automated. autopackage (or Listaller), rpms and deb should require no interaction what so ever except the execution and a final sign off by the person releasing.

We have that already at that stage, in the 'build' branch. It's basically 'bzr pull/svn update; make; make upload'.

The windows build could also be completely automated if build issues are maintained during development. I could develop some scripts to make that possible fairly soon but it'd take some time for Mac as i'm unfamiliar with the build system.

The main hurdle with those platforms, should I be the one doing builds, is to turn the bloody things on and waiting for the builds to finish, and for the Mac, digging it out (it's no longer in active use since it started malfunctioning). I'm not easy to convince any script can help there The process I currently use is simple enough: copy source from linux box, unpack, open project file, hit compile button, wait, copy result to linux box (in between the 'make' and 'make upload' steps, 'make' prepares the source).

What's the most painful part, apart from the waiting, are the upload procedures; they involve a lot of messing with web forms in browsers, pasting in information, signing and uploading files, sorting them into the right categories. Automation for those would be welcome. Ooh, found something that would help partially: http://www.piware.de/2009/09/automated- ... launchpad/

And for 'real' releases, anything more than a weekly snapshot, there is still more work to do that's impossible to automate: you need to check everything basically works, you need to tag the sources, release notes need to be written and pasted to the upload sites. A proper release used to take up essentially all my free time allocated to Tron for a day.

[Vitty]

If it's so easy, sure, go ahead, I'll merge it. Can't hurt, can it? Can you add a test to ./configure that checks for the presence of the tools?

I've started it here: https://code.launchpad.net/~vitty/armag ... d-breakpad
The library is just the bare minimum exception handler. The code to use that and dump the crash is very minimal. It won't be used unless --enable-breakpad is passed to ./configure so it won't make a difference to anyone who would continue to checkout from bzr. This works flawlessly for the linux build as is. Just need to set up the windows/mac library to build and it'll be ready for merge.

The technicalities of when and how to produce the debugging symbols can be tackled once it's fully working on Windows and Mac. As you said, the dumping feature being there can't hurt regardless of the symbols.

It is? I figured you'd have to integrate the generation of the symbols into the .deb build process and have them packed into a separate .deb you later have to download.

I'd assume that's the way it's done, but i've not looked into how it's done yet. I'm just aware of the chromium ppa which produces a new chromium-browser-breakpadsymbols package everytime it's built.

[Vitty]

The technicalities of when and how to produce the debugging symbols can be tackled once it's fully working on Windows and Mac. As you said, the dumping feature being there can't hurt regardless of the symbols.

I've been looking into this more. It looks like the best way to do this would be to compile arma with the debugging symbols (-g), dump the symbols, and then strip them out of the executable afterwards. That also looks to be the way they're doing it for chromium and firefox.

If the symbol dumping application was included within the thirdparty directory then you could have the symbols dumped whenever somebody compiles from trunk. They can then just upload there own symbols file along with the crash dump. That should make this feature more useful to linux aswell as mac and windows

[Z-Man]

Ok, tested it on Linux. Added a crashing config function, and after going through the whole process, this is what minidump_stackwalk produces, apart from status messages:

Code: Select all

 0  armagetronad!sg_Crash [gArmagetron.cpp : 89 + 0x3]
    eip = 0x08077a29   esp = 0xbfb277f8   ebp = 0xbfb27808   ebx = 0x00000001
    esi = 0x082881c2   edi = 0xbfb284e4   eax = 0x00000000   ecx = 0x00000000
    edx = 0x08077a19   efl = 0x00010282
    Found by: given as instruction pointer in context
 1  armagetronad!tConfItemFunc::ReadVal [tConfiguration.cpp : 1061 + 0xd]
    eip = 0x082b35a8   esp = 0xbfb27810   ebp = 0xbfb27828   ebx = 0x00000001
    esi = 0x082881c2   edi = 0xbfb284e4
    Found by: call frame info
 2  armagetronad!tConfItemBase::LoadLine [tConfiguration.cpp : 426 + 0x18]
    eip = 0x082b1511   esp = 0xbfb27830   ebp = 0xbfb278b8   ebx = 0x00000001
    esi = 0x082881c2   edi = 0xbfb284e4
    Found by: call frame info
 3  armagetronad!tConfItemBase::LoadAll [tConfiguration.cpp : 718 + 0xd]
    eip = 0x082b2549   esp = 0xbfb278c0   ebp = 0xbfb279f8   ebx = 0x00000018
    esi = 0x082881c2   edi = 0xbfb284e4
    Found by: call frame info
 4  armagetronad!gMemuItemConsole::Event [gMenus.cpp : 622 + 0x15]
    eip = 0x080cce8e   esp = 0xbfb27a00   ebp = 0xbfb27af8   ebx = 0x00000018
    esi = 0x082881c2   edi = 0xbfb284e4
    Found by: call frame info
...

That's a stack trace all right, but a really bare one. Nothing apart from file/line/function. Local variables are usually required as well to get obscure crashes fixed. Is there a breakpad-internal way to get them? Edit: hmm, probably not, it's in the 'future plans' of some document on the Wiki.

(It's possible to convert the bp crash dump into a core dump and use gdb on that, but then the symbol dump appears to be useless then and you essentially need to keep the unstripped binary or a different symbol dump, and the core dump will be incomplete, so that procedure raises the question "why not use core dumps directly? Well, apart from logistics and uniformity, of course.")

[Z-Man]

On my way to Windows integration. Compilation of the game is fine (thanks to a mingw patch I found), but we'll probably need to compile the client library in Visual Studio and put the binary into winlibs. No biggie, just a bit of work. Now, off to finding out how this gyp stuff is supposed to work.

Edit: oh, bummer. That kind of stuff doesn't work for C++, of course. Soo, let's see. Switch Arma back to VisualC++, with all the problems that's going to cause, try to compile breakpad on mingw, or be content with crash reporting from Linux and Mac users? I'll try breakpad+mingw first.

[Z-Man]

Yeah, that worked all right, with only a couple of source changes. Crash dumps are generated now.

Next problem: are they actually useful? Hard to tell. I couldn't yet get the symbol extractor to run. The supplied binary requires some DLL from an older SDK, the VisualC project file is missing some headers (probably from the same SDK). Documentation does not exist, so it's "google missing thing, hope that tells me what to install". Yeah... taking a break for a bit. It's still very likely that the symbol extractor simply won't work with mingw debug symbols.

[Z-Man]

Nope. Won't work with mingw. The executable symbols can't be read by the breakpad tools, the generated dumps can't be used in Visual Studio directly (naturally) or gdb and can't be transformed into core dumps gdb on Windows could read. So that's a dead end. It seems like on Windows, we'd need to switch to Microsoft tools (at least the compiler) to make use of breakpad, and that is all kinds of trouble. It makes setting up a development system much more complicated, Microsoft's headers often do weird shit messing even with C++ standard library headers and the chances innocent changes on Linux or Mac break the Windows build rise. I'm not sure this would be worth it. (Doesn't mean the whole project is worthless, every subset of platforms with crash reporting can give valuable data, it's just a different subset than I would have liked.)