Are Game Developers 15 Years Behind?

[Van-B]

So what are you saying Christo?

IMO your just trolling. Frankly that article was so full of digs at game developers, I just wonder what you are doing here. I didn't even realise that was your article until now, because I don't know why someone would post that here, if not to deliberately offend. I could understand why you would bring these things up if you were getting back into game development.

Don't let your own experience of bedroom coding tarnish your opinion of game developers. Remember that most of us here are hobbyists - most of us probably have jobs in IT, software engineering, most of us are probably just like you except that we write games instead of blogs.

[christo]

There's two reasons. Firstly, yes I'm getting back into game development.

Secondly, I think there's a competitive advantage to be had in this competitive commercial field. This doesn't directly affect bedroom or hobbyist programmers (who I adore, not least because it reminds me of my childhood). I do go into quite a bit of detail in the blog post and I followed it up with a more sedate blog post if you bother to read it.

Problem is, as I say in my second post, some game programmers have become very emotional about this. Most have not, however, because they either agree or disagree and they can leave it at that. I know many game programmers professionally and I deal with them on the subject of software tooling and process issues. I think the dramatic growth in the online gaming and online game distribution sides of the game business are going to have an impact on software processes and make them more effective for game developers than they were in the past.

I don't think this is being a snob. The things I talk about in the article, especially automated testing, these are techniques that solve a problem. If you're a bedroom programmer you may not have that problem to solve - that or you don't recognise it as soluble.

If you work in an experienced professional team and you expect the software to feed your kids, then making sure you can keep up the pace of development over time, this shit matters. Of course, it doesn't matter if you spend years in development and then ship the thing and it makes all its money in 6 weeks and then you throw most of the code away and start from scratch. It may not help as much if your process is dominated by asset production. It may not help if you are making tiny flash or iphone games that are done by one dude in a few weeks. But if the code base is anything like non-game software products or other kinds of software, then well, maybe it will benefit from the same things that those software teams have discovered can work to solve that real probem.

[Van-B]

Good to know. I strongly advise leaving the protocols behind when you leave work for the day though. Hobbyist game development has to be fun, and somewhat organic, otherwise you'll likely spend a long time procrastinating like I do everytime I over think a project. In my experience, the quicker a game project get's to being playable, the more likely it is to be finished.

[Falco Girgis]

Problem is, as I say in my second post, some game programmers have become very emotional about this. Most have not, however, because they either agree or disagree and they can leave it at that. I know many game programmers professionally and I deal with them on the subject of software tooling and process issues. I think the dramatic growth in the online gaming and online game distribution sides of the game business are going to have an impact on software processes and make them more effective for game developers than they were in the past.

You see, I have a problem with this. You like to say we get "emotional" very often as your defense. Emotional as in what, we passionately disagree? As in hearing such generalizations piss us off? What kind of self-respecting, self-sacrificing game developer who has confidence in his own abilities would NOT be offended by hearing somebody else essentially tell you that you're doing it wrong?

You're accusing us of being defensive just as much as you are being "emotionally" offensive. Your article did not say "Should game developers use unit testing?" Instead, it declared that the entire state of game development as a software engineering profession is a decade-and-a-half behind because it does not necessarily adhere to your beliefs? Then you are shocked when people get pissed, passionately disagree, or display any sort of emotional response? What?

There's nothing wrong with having an opinion. You're goddamn right that I think I'm doing what's best for my projects, and I'd hope that everybody here feels the same way (or you should get off your ass and do something differently). What IS annoying is people who think that their experience magically means that they know how to handle or manage somebody else's project--like thinking that indie/commercial game development should be handled like any other software development endeavor. You are far too assertive of your opinions on others.

I don't think this is being a snob. The things I talk about in the article, especially automated testing, these are techniques that solve a problem. If you're a bedroom programmer you may not have that problem to solve - that or you don't recognise it as soluble.

I think you were being a snob. I think you were being a snob because of how you presented your argument, because of how you made the assumption that the game development market should follow YOUR views of software engineering. As I said before, instead of simply asking why unit testing is not more prevalent and pointing out the benefits that it could bring to the table, you decided to go on a full-blown crusade.

But if the code base is anything like non-game software products or other kinds of software, then well, maybe it will benefit from the same things that those software teams have discovered can work to solve that real probem.

You're goddamn right. Who DOESN'T stand to benefit from software that isn't buggy? But how does not performing this software testing put somebody FIFTEEN YEARS in the past? Agile wasn't around 15 years ago. Hell, UML wasn't even around then. 15 years ago, we were transitioning away from ANSI C and into object-oriented development...

[christo]

Here's what a programmer from EA Tiburon said:

I worked at EA for eight years. My running joke-that-was-not-a-joke was that we were twenty years behind the times, but I was probably exaggerating. Fifteen is a pretty good estimate.
At EA, I think the problem was that the company culture was built like this:
A bunch of bedroom hackers in their teens started studios and shipped games ten to twenty years ago.
Those games got successful, and the studios scaled up. The founders hired young people under them.
More staff churn. Fresh grads come in, burned out experienced coders go out.

I'm also not just talking about "agile". That's only one practice and I'm certainly not saying you have to be agile to be up to date, any more than you have to be 3d to be up to date.

I'm talking about a range of engineering practices that directly contribute to project outcomes. Automated testing, continuous integration, source control, backups.

Now of course there are plenty of games companies who do these things, however in my direct experience, statistically speaking, game companies seem to be left shifted on that normal distribution of tool and process adoption. Is it 15 years? 14? 10? Whatever.

Now I haven't done a statistically valid sample but I'm totally open - which you can read in my blog post - to the idea that games are not suited to some of these practices, but I haven't been particularly convinced by those arguments so far. More detail here.

Professionally, I'm interested in these issues of tool and process adoption and I'm interested in games development.

In the late 90s I was exposed to automated testing techniques and I have since seen the adoption of this practice grow to something of an industry standard. These days, if you're not doing automated unit testing in a non-game, you're backwards. I have said this in my conference talks. People didn't get offended.

But I do understand that people got offended. You can be as surprised as you like that I should have known people would be offended, but maybe that's an example of something you know better than me. Are you open to the possibility that I know about something else better than you or has your offense provided a convenient reason to ignore me?

Read the shorter, less offensive followup.

[krilik]

I think you're arguing a misconception more than anything.

In your follow up you directly address the misconception and then blow it off as negligible.

If the answers are things like "How could you test Starcraft 1?" or "You can't test for things like game balance" then I'm sorry that's not a convincing reason! How is it obviously impossible to automatically test Starcraft?

The flow of game logic can be tested so its not broken. But that doesn't make it fun. You are always going to have to go into the code and tweak variables, redo logic, remove logic, design new logic, to design gameplay elements that actually entertain people. And to say that you can write a test that can predict whether or not that segment of code is going to "work" (in the sense that it will be fun to play) is outrageous. Manual testing is a necessity of game development.

You talk about Blizzard adopting such approaches, but you fail to mention that they spend months on PTR servers for WoW modifying the game based on manual tests. And they go through several build iterations, not just tracking bugs, but redesigning gameplay elements.

Point is, gameplay is always going to have to be manually tested. Writing an automated test to see if it "works" in all senses is not going to work. In my opinion its going to be a waste of time trying to write tests for everything you do since you are going to be forced to manually test the game anyway. And in some cases being forced to rewrite, remove, design new tests for changing elements. For bigger projects I could argue the necessity of automated testing outweighs the costs of doing it (but I don't think you have a convincing argument against the commercial industry through one post written by a guy who supposedly work for a studio at EA and failed to give any specific details as to how this gap existed), but for hobby/indie developers I think its a waste of time and effort that can be spent manually testing and fixing in the same amount of time you are going to be writing automated tests for things that aren't as important as what you have to manually test.

[avansc]

I was not offended but any means. but just as a note.

I can vouch that for the in-house stuff at IBM in some labs, and most everything qualcomm does in the RTP, as well as some labs at EMC, unit testing is low on the totem pole. They do have automated tests, regression and all that.

I cant comment on what the average game does. Personally I have never really used any for of testing on my projects. Their scope just does not warrant it.

I may have the complete wrong idea about unit testing, or am doing it wrong. I think it would be a rather good blog post... Id like to see how you manage to keep your unit test from blowing up if you say split a class, or consolidate 2 or more.

[qpHalcy0n]

I agree. No offense taken here. I agree on most points.

I used to develop for Terminal Reality, Inc here in the Dallas area as a graphics programmer and shipped two titles. Now this was 10+ years ago, so I'm no longer qualified to answer questions as to what they do in the industry now. 10 years ago, though you would be correct. The testing was all blackbox and fairly primitive, but it worked. The terrain of game development has changed significantly in 10 years and we have the XBox and PS2 to thank for that. Now I still do have contacts who are currently working in industry and I can totally assure you that professional developers are certainly on top of this issue. I won't say that things can't be tested, but it's sortof this situation where you have alot of momentum riding in the direction that things have been trending towards for 15+ years. It's an industry centered about technological innovation, and as such it seems that where resources are available that is where they are shifted. It's not that they do not have resources dedicated to testing methodology, they do. But game development is a car running at 150mph down a track, and modern testing is a car that started 10 seconds behind it with only slightly better acceleration. It's going to take a bit for the two to completely catch up I think. I've said this in prior posts, but for major players in the game development arena, their main squeeze is on technology licensing. In that regard, code reusability and stability as absolutely paramount. To that extent, I think that game programming and business programming methodologies are merging to some degree. Personally, I hope so because I think that game developers in the past could be accused of being a bit myopic and could certainly benefit from advances in other areas besides pure technological innovation......and I think they are.

That said, the only point I disagree with is that it sortof seems like you might be throwing indie devs under the bus a bit. Granted, I am a professional developer turned indie developer, so I find myself in a very interesting situation where I've seen the big game and the big picture, and now I'm confronted with some of the finer grained details. What I can say is that indie developers are a subset of a pool of some of the most talented minds and programmers that I've met. Now what makes a good developer is not necessarily what makes an exceptional programmer. Some of the finest minds in the field might be poor programmers. However, with most hobbyist game developers, we have an obligation to some other occupation. I have an obligation to my educational institution to do well and learn. This leaves me with very little time to work on the projects I'm truly passionate about working on. So it's a finer usage of my time to stay on top of the technological curve than it is to ensure that my software is thoroughly tested. It is not commercially distributed, I know what platforms its designed to run on, the functionality is limited. In other words, it is not robust. It does one thing, and it does this one thing well. It's a research platform essentially. There are MANY other reasons for my software to fail than poor testing on my part...the primary reason being hardware incompatibility.

I will admit, being a product of "15 years ago", I am a bit naive to the most up-to-date unit testing techniques. As I'm sure you're most assuredly naive to the most up-to-date rendering techniques. That is my field. Testing and tooling is yours. However, when your game sees light, I'm not going to accuse you of not using the latest that modern hardware has to offer you and say "Man...that was very 1997...blech!". We can help each other...sure. I've never been involved in a project where that was NOT the case, but I think there's a way to go about it and it seems that the reason your post seems so inflammatory is because it appears to be "on the attack".......just a thought.

The beginning to this two way cooperation and dialogue, however, is not to insinuate that we (independent developers) are somehow naive...nor would we insinuate that you are. Remember, YOU came HERE.

[christo]

I think you're arguing a misconception more than anything.

In your follow up you directly address the misconception and then blow it off as negligible.

If the answers are things like "How could you test Starcraft 1?" or "You can't test for things like game balance" then I'm sorry that's not a convincing reason! How is it obviously impossible to automatically test Starcraft?

I'm directly questioning the reason behind an assertion. I'm not blowing it off.

The flow of game logic can be tested so its not broken. But that doesn't make it fun. You are always going to have to go into the code and tweak variables, redo logic, remove logic, design new logic, to design gameplay elements that actually entertain people. And to say that you can write a test that can predict whether or not that segment of code is going to "work" (in the sense that it will be fun to play) is outrageous. Manual testing is a necessity of game development.

You can't automatically test whether the game is fun. You test whether the game code does what you intended it to do. If you intended it to be fun and failed, your tests won't tell you.

You talk about Blizzard adopting such approaches, but you fail to mention that they spend months on PTR servers for WoW modifying the game based on manual tests. And they go through several build iterations, not just tracking bugs, but redesigning gameplay elements.

[/quote]

Manual testing will always exist. But it's obviously it is best automate what can be automated wherever it is cheaper than manual testing. If something cannot be automated then you don't automate it. If it can be automated but it's cheaper to manually test then you don't automated it.

However, X cannot be tested is a common mistaken conclusion. I've seen this mistake made since I learned about unit testing the late 90s.

[christo]

I agree. No offense taken here. I agree on most points.

Thanks for the detailed comment.

The beginning to this two way cooperation and dialogue, however, is not to insinuate that we (independent developers) are somehow naive...nor would we insinuate that you are. Remember, YOU came HERE.

You can insinuate it all you like, I admit it! I am naive about many things, certainly rendering, and I don't mind being told. That's part of why I came here! I've participated in a few discussions, I've asked questions and attempted to introduce topics I know about, for example:

How do Sprite Artists Work Question about art asset pipeline in game development
Test Driven Development Introduction to TDD
Programming Languages
Bitbucket in the context of free development tools

I might be different to many of the other people on this forum - I don't know for sure - but perhaps variety is beneficial as you say.