We’re in a hard position when testing GMod updates. It’s an area I’ve wanted to improve for a long time. It’s really hard to test them properly. Let me explain.

User Content

There is so much content in GMod. So much user created content. So many different ways GMod is used. So to truly test an update we’d have to try every piece of content, every map, every model. This is obviously impossible for us to do.

Engine Updates

When we update the engine (or change the network datatables) it renders the new version incompatible with the old version. This means that we can’t join old servers, and people using the old client can’t join our development servers.

This means that all the developers on the forums that ask for features or bug fixes can’t really verify that they’ve been fixed (it is possible – but only with a lot of steam.cfg editing and file copying).

The Beta

The answer to a lot of the problems of testing is to test it via a beta version. This sounds like a good idea except for two big problems. Hardly anyone uses the beta version, and of those that do – hardly anyone reports bugs.

So you get the situation where the beta is there and it’s being updated. But it doesn’t get anywhere near the same level of testing because everyone has their crazy addons and gamemodes already installed in the live version. So that’s what they play on.

Hire Testers

A lot of people suggest this. Sure – hire 10 people to play GMod all day – to test every area. How much would that cost? £150k a year?

It does sound like it would work, but at the same time I hate this idea. It’s not just the money. It’s the fact that you have 10 people hanging around that have nothing to do 95{a1da5f31666ba9e4b613f74c475fa44930e0dd96d95a00cdfc356ce4f15804bc} of the time.

Plus it’s not even that effective. They still can’t simulate the hundreds of gamemodes, the thousands of server configurations. It still won’t catch all of the bugs.

Unit Testing

This is one area that we can do better in – that I want to improve on. It’s basically running a program that will verify the result of a bunch of function calls – to make sure they’re all operating properly. 

This would work for a lot of things – it just requires a bit of investment in time to set it all up. To be 100{a1da5f31666ba9e4b613f74c475fa44930e0dd96d95a00cdfc356ce4f15804bc} effective we’d need to set it up with EVERY function though.

I am envisioning a Lua script that can do it all automatically though. I mean, we run a script – it somehow calls all the functions on a player object, or an entity object. Then it stores all the results (or errors). Then the next update you run it again – and if they don’t all line up with the previous results you show the diff.

Maybe they’re different results because you fixed a bug, maybe they’re different because you created one. Either way we’re at least aware of it.

Generally..

Generally though this isn’t an issue because if I change anything in code I have only changed one thing. If I change a single lua file – it can only affect so many things – so it’s easy to test. I don’t need to test EVERYTHING else – because I didn’t affect them.

So usually between updates there’s not a great deal that could have broken. As long as everything compiles without errors, and the build server’s configuration hasn’t changed – everything is simple.

But sometimes..

Sometimes I merge in with Valve’s latest code, and apply all the enhancements and bug fixes they’ve added. In this case everything needs a good test. This is what happened with the update on Thursday that I had to roll back.

Although the huge majority of the merge has no affect on GMod at all, sometimes Valve has edited a file in the same area that I have edited it previously. In this case the code needs to be merged by hand. This is just a case of looking at the original code, at the changes I made, and at the changes Valve has made. Then I need to decide whether to get rid of my change, or put it after Valve’s change, or before Valve’s change.

Sometimes these merges go wrong. As is the case with the update on Thursday.

Pink Maps

One of the errors in Thursday’s update was that maps were pink that had been downloaded from the workshop. This was because the filesystem couldn’t mount the pak file that was inside the map, so couldn’t get the custom textures.

This is one of those bugs that should have never really got past us – but it did. As soon as we made the update live it was plainly obvious that it was a problem, and what the fix was.

Solution

So we’ve added ‘check workshop map gm_atomic’ to our list of things to check pre-update.

This is kind of what we’re going to have to do, build a check-list of things that can fuck up – and check it manually before each update. On each platform.