I Pity The Fool Who Doesn't Write Unit Tests

blog
41

First off I love TDD. I get more satisfaction out of my code when I write the tests first. I feel like I fly through my task list and my mistakes are few and far between.

Second, I don’t do much TDD. If it’s something I’ve done a million times before, then I just write the code. But there’s always that one little bug that takes time to root out and find, even though I’m an expert on this or that. TDD, I have very few bugs during and QA finds even fewer bugs after. No TDD, there’s always something. And even if it only takes me 2 minutes to find the bug, that’s 2 minutes too long, not to mention the tester’s time I’ve wasted. Shame on me for being so darn brilliant.

I think a key difference between the people who embrace TDD and those that don’t is your comfort level with code. I mean raw code, semicolons etc. If you use whiteboards and/or UML first, or need to get a rough design doc on paper before you can commit to code, then TDD probably doesn’t resonate the same way. If code is your first language and you have to translate that to english to talk to other human beings, then for you, tests are the only form of design you can really embrace. They’re always accurate, and let’s face it, you don’t trust design docs or comments anyway.

Now I’m also a huge refactorer and I like nothing better than ripping up huge chunks of code to accomodate a new wrinkle in the requirements. At that point TDD hurts, at least until you embrace the delete key. Those tests are going to need hours of rewrites? DELETE. First find the lowest higher level above the construction zone and make sure you have a safety net. Then just blow the rest away. TDD the reconstruction, or not, who cares? The tests are there to help you along, not hold you back. I’ve got the behavior I want to enforce at some level and if that doesn’t break I don’t really care what happens below now. Sprinkle in unit tests in whatever fashion you please until you feel good about the code.

42

I think any good developer writes “test programs” for the code they write. I mean how can the code be “finished” if you haven’t see it perform its task correctly?

The best advise I was ever given (in this area):

Developers should spend a little more time on the quality of the code in their test programs and then Don’t Throw Them Away when your done. Check them in!

43

I would be amused by the posters that believe we should eschew unit tests if I was not so appalled. Of course unit tests are not the answer to everything. Of course TDD is not the be all and end all. Of course unit tests are expensive and time-consuming to develop. Does that mean we should not make use of them? NO!! Good grief, that attitude is unconscionable. Here’s the news, folks: most developers out there are pretty clueless. Just because you are writing and maintaining your well-designed, careefully constructed, expert code does not mean you will always be doing that. One day, some clueless wage slave will be sitting at 3 a.m. in front of your perfact C++ code with a stream of drool escaping the corner of a slack mouth, thinking “WTF is bind2nd?”, with a production outage he has to solve because some other clueless wage slave broke the code with a one-liner that could not POSSIBLY be wrong and did not bother to even compile the code before it was checked in. What, someone like that should be fired? Yes, of course. Are they? Not nearly often enough, if ever. People are hopelessly short-sighted if they think their great code (and I say that sincerely) is not going to be murdered for expediency. Would unit tests help? Truthfully, maybe, and maybe not. If they are integrated into the build and run automatically, they could save the day. If not, well, good luck. The clueless wage slave probably doesn’t even know about them and wouldn’t run them.
Really, I am not some ivory tower academic who is promoting unit tests because I read a couple of books and web sites on the topic. I have been in the trenches for decades, like many of you, battling some of the most heinous code ever to escape the worst developers’ twisted minds, heaven help us all. I have also seen carefully crafted code exhibit some subtly nasty bugs because the boundary conditions were not tested, even though there were unit tests in place. Some that code was even mine, I am sorry to say. So unit tests are no panacea. You need other levels of testing - system level, end to end, and so on, absolutely. But doing without unit tests altogether in a real-life situation seems to me to be total insanity. You can reduce the expense of writing unit tests by focusing them on critical layers of the code. I have in general found that the lower, infrastructural layers of a system are the most critical, because everything else depends on them. The foundations need to be rock solid. Unit tests are the safety net that stops those foundations from crumbling, if I can mix some metaphors. Nothing has applicability to every single situation. Like Aloysha said, one size does NOT fit all. Agreed! But I cannot think of any real-world project that would not benefit substantially from some degree of unit testing.
And I have to tell you: having a good architecture without a good implementation can sink a system just as deep as the converse situation. However, a bad implementation is potentially correctable, while a bad architecture is practically impossible to fix.
Final thoughts: you can make use of unit tests without subscribing to TDD. If your management gives you the time :wink: just write them after the code is written (although you will have great fun refactoring the parts of your code that are untestable without having the benefit of, um, unit tests). For those people who talk about testing DBMS, UI, network, I have two words: mock objects. Again, not a panacea, and it can be a lot of hard work, but still… it’s nasty how those pesky unit tests can show up irritating code dependencies, isn’t it? It kind of forces better (or at least more flexible) design, I think.
Bottom line: you would indeed be foolish if you did not seriously consider incorporating a unit testing strategy into your real-world project.

44

I’d say Brian comes close to my developer test philosophy.

If something is complex, make it simpler. If it can’t be made simpler, try to move as many complicated bits into a testable location and test that before you declare it “done”.

But don’t waste your time testing the simple parts. 20% of the code has 80% of the bugs; spend your time there instead. You don’t have to TDD everything.

I don’t have a problem with unit tests, per se. What I have a problem with is this one-size-fits-all, unit-tests-solve-everything attitude I hear so often from TDD/UT advocates.

45

My problem with unit testing is that it isn’t testing the right thing. In fact, it is testing the part that is least likely to fail.

Pure code, the kind unit tests target, is easy to write correctly. It is very rare for me to have a bug code I wrote entirely myself.

Most of my bugs come from boundaries. Code that interacts with…
the database.
the file system.
the GUI.

Unit tests simply don’t address those concerns.

Can you write repeatable tests that account for the database? Sure, but it isn’t easy. It takes a tremendous amount of planning.

I my own line of work, I find that writing formal test plans that account for every boundary (GUI, filesystem, network) are far more effective.

And by the way, McConnell’s Code Complete 2 cited a study that showed Unit and Regression tests to be among the least effective testing strategies. Other techniques, like formal code reviews, were far more effective.

46

My question got lost in the debate. So I’ll ask again.

Where does one get started in using Unit Tests? Is there a book I should be reading? or a website? or what?

I am a solo developer, with no budget. I don’t even have a dedicated testing computer. And I have to test and deploy apps (all in-house) in ASP.Net, VB.Net 1.1, VBA (Excell and Access) and a bit of Word.

47

Also check out the pragmatic programmers, they have a book for junit and one for nunit. You can also take a look at JB Rainsbergers Junit Recipes, it’s a great reference, even if you don’t use junit (presuming you can translate from junit to whatever).

None of these books require that you be a TDD fire-breathing zealot, though it’s usually implied that it would keep your ears clean.

If you want to test code that’s already written and doesn’t seem to be testable: Working Effectively with Legacy Code, by Michael Feathers is awesome. This book tells you what’s the bare minimum you can do to make your life easier without having to start from an ivory tower of perfect tests, 100% coverage etc.

There’s also tons of websites and tutorial surrounding the tools. Junit for java, Nunit for .Net and hundreds of others for all the other languages.

Good luck!

48

“Dave, I think you’re using the term “unit testing” to refer to all kinds of testing. It’s not – there are tons of different testing techniques besides unit testing. Yes, unit testing tests variation of inputs, and traditional QA testing tests variation of inputs, and the difference between the two is a matter of scale.”

Actually, I’m thinking of the Unit Testing environments I use on a daily basis to get my work done: JUnit, NUnit and RUnit, from the command line and inside Eclipse. I’m thinking of a variety of “asserts” I’ve written for load testing iterations through large recordset returns, using Schema objects and SQL to test for constraints, missing foreign keys, missing relational records, any number of weird edge cases I can think of.

Can I think of all of them, all the time? No. Have I EVER thought of a database or filesystem situation that I couldn’t test for with an assertion from any one of the libraries I’ve mentioned?

No.

“It is bad testing practice to verify that an API call results in such-and-such database modifications and such-and-such XML being sent over the network. Because that’s testing implementation details, details which are the first things to break once any refactoring happens. You should test what goes IN to an API and what comes OUT of it only.”

This is another Red Herring. Yes, Unit Testing only guarantees that – Shock! – the I/O for the method in question is correct. It doesn’t guarantee integration will be perfect and it doesn’t put QA departments all over the world out of work. We know this.

But if there’s an edge case you’re worried about – missing PKID’s, missing constraints, missing records, questions of volume (maybe not TDD best practice, but I can do it, so sue me), in general you can write an assertion for it, and the overhead for doing so is minimal. It literally takes me seconds to write a decent unit test for 99% of the methods I code. And then after that it saves me many minutes, maybe hours, as I don’t have to run the whole exe to test any changes – I just have to compile the class and its test class and execute the test. No clicking through GUIs, no setting up watch varibles or breakpoints, no relying on my own “eyeballing” of the output through several iterations, hoping I remembered to check everything the same every time. Unit Testing simply automates and speeds up the things I’d be doing anyway.

Yes, if the recordsets are large and complex, the setups will take a little longer, but, once again, this is NOT the fault of the unit tests. The tests are STILL making my life easier by AUTOMATING what I’d be DOING ANYWAY.

49

“Bottom line: Unit Tests are often good. Unit Testing Evangelism is always bad.”

I agree with this. I’m not trying to oversell TDD or make it out to be the panacea some people may think it is.

I just find it amusing when the anti-TDD zealots try to tell me what I’m NOT doing on a daily basis with unit tests.

50

Aloysha, I agree 100% with your last post!
Especially the part about being screwed if a team consists of total monkeys. Not that most of the team I work with are monkeys, but certainly some of them would qualify. My late father used to say, “If you pay peanuts, you get monkeys”. That’s the sad truth, although regrettably the converse is not always true: if you pay a lot, you don’t always get a great developer (but you are more likely to than if you underpay).
The real simian tragedy is that by their actions, many corporations today demonstrate that they believe 3 coders at $33/hr to be better than one real thinker at $100/hr., and that you can just switch people in and out like relays in a fusebox without a serious loss of productivity.

51

Aaron wrote:

“If design problems account for 80% of the debugging (I’d say that’s a conservative estimate) then even if unit tests halved the time spent on code problems, it’s still only saved you 10%, and that may very well be offset by the time spent creating and maintaining the unit tests.”

I’ve never seen design cover 80% of the bugs. Maybe you work in a department of 20-year veterans with immaculate coding practices, but in my world there are a lot of recent BS and young coders still working it out. But even if I were in a deptartment with a bunch of ubercodemensches, there are still things design couldn’t do.

Brilliant design cannot write methods. You cannot design null checks. Design cannot guarantee that when other code monkeys break my code, I’ll know about it within the hour. Our dept. does a lot of design, QA and code review. Testing is just one tool in the box, but it’s an essential tool that covers things design and code reviews cannot.

And I’ll beat this horse again – it’s faster to code with unit tests than without. It’s even faster to code good design with unit tests. There shouldn’t be a dichotomy between the two, but in my day-to-day reality, I don’t always have control over what requirements I get, or who’s in charge of the design process. But I do decide whether my code is going to be tested or not.

52

“The utopia of separating UI and business code.”

Hmm, I’m about 95-99% in utopia then.

What a naive statement. Another damned hack.

53

Going back a bit here, but the latest version of Visual Studio Team System is geared explictly for unit testing. Microsoft sees the need for testing at the code/unit level.

I have worked on more than a few functional and system test projects in QA. The worst part about testing at this level is finding a bug in production that a unit test could have caught. They exist, they are not unicorns. But what saddens me is how quickly a developer will place the blame on the QA group. We cannot test everything, unit testing is essential.

Resistance to unit testing from developers is the first part of the problem. But lack of understanding from business sponsors and business analysts alike only complicates the issue. If you deliver a poorly designed system to a developer you will get a system, but it won’t be what you really had in mind.

The politics of this makes me sick. Developers need to follow requirments, and code what is asked. The BA’s need to deliver solid requirements and design specs. The QA group needs to over see the ENTIRE process, start to end. Otherwise your just doing Quality Control, at which point you are better off not bothering to test. The QA group is always relagated to the role of “tester”, this is why bugs end up in production.

I used to be a developer, I know how painful it is to have your ability as an artist dumbed down to the science of the printing press. But it is a necessary evil.

Development groups need to be more open minded, this is why you see so much offshoring of development projects. Offshore resources will do as you say to a T. Keep in mind you will find holes all over your design when someone in New Delhi codes exactly what you sent them. Again QA has to sign off on that design before submission, but this goes back to the politics of software.

If you have ever approached a development team with the suggestion of someone else, maybe in QA, writing unit tests you know what reaction you will get. First they will ask “Why do unit testing?” Then they will ask “Who is going to write the tests?” And when you tell them QA/junior developer they tell you “They have no clue how to write a unit test, they can’t understand my code, and they will ask me questions all the time, how do you expect me to get anything done?”

At the end of the day any company that is finding problems in their production apps needs to look at their SDLC. The groups that participate in the lifecycle need to have clear definion of their roles and clearly documented expectations. The groups need to have a HEALTHY amount of dependency on one another. Politics need to be kept out of the process. Without this you will ALWAYS find defects in production regardless of what methodology or “religon” you follow.

54

“One day, some clueless wage slave will be sitting at 3 a.m. in front of your perfact C++ code with a stream of drool escaping the corner of a slack mouth, thinking “WTF is bind2nd?””

Hahah! Nice mental image there. Don’t think I’ve ever used bind2nd, but point understood …

I think if your coworkers really are total monkeys, then you’re just f*cked no matter what you do. No process or technique – not even one as wonderful as TDD – can save your ass in that situation. So for the sake of this conversation I’m assuming competent coworkers – knowing full well that it’s not always the case, but if you don’t have competent coworkers then any process improvement is giving medicine to a corpse.

“But doing without unit tests altogether in a real-life situation seems to me to be total insanity.”

I totally agree. Scroll up, you’ll see I’ve never advocated such a thing.

“the assumption is made that doing test driven work somehow requires that you not think ahead and do the design up front”.

TDD has many arguments for it, so not all TDD advocates use TDD as an excuse to write crap and then refactor. But some do. Of those, the argument that “TDD makes refactoring easier” is the most attractive one of the lot, because refactoring is what they do ALL THE TIME. Refactoring should be much rarer than that. One need not adopt a process that optimizes for constant refactoring.

Anyways I think we’re having a Colbertesque “George Bush: great president or greatest president?” argument here. I think we all can cite specific unit tests or developer tests that have helped us greatly in the past. But there is still a lot of art to them, and there are still too many ways to write ineffective or unproductive unit tests. And, you know, people who point that out the limitations of unit testing keep getting branded as God-hating anti-TDD commie Luddites (even though everyone accepts that unit testing does have its limitations).

No religious debate here …

55

I like how it is said I’m attacking a straw man and then the assumption is made that doing test driven work somehow requires that you not think ahead and do the design up front… or even to try to get it right the first time.

I see your “straw man” and raise you a “failure to address the points” such as the reality that not only golden children will touch code and thus designing a system that will detect failures of imagination is a good idea.

56

Would Mr T write tests before he went and smashed stuff up? I dont think so.

57

Hi all,

Would Mr T write tests before he went and smashed stuff up? I dont think so. - yourMateNick on February 20

hahahahahha lol. If you really think about it, he acctually did the tests and they were called practice. If he didnt, he might just get smashed up himself.

On a smilar note, however, I do think that unit tests are kind of waste of time. But I will definatly try them to really feel what all this hub-bub is about. From what I have read and understood, it seems more like unit tests as a tests which one can do while they code any way. Debugging and hacking works the best for me up till now, and writing more code specificaly for tests for situations I KNOW for a fact that will change will eventually require change in test modules as well…then why bother?

But as I said I will have a go at them.

58

I do a lot of my low-level (as opposed to end-user-oriented) testing in the debugger. In essence, I don’t trust a piece of code until I have forced it through all possible code paths. This might sound daunting - even impossible - but it is not. With a good debugger - and I se Visual Studio which is truly excellent - you can put your code through hoops in short order. Simulate an error condition by changing the result returned by MySuperDuperDooWopADiddyFunction () to false. Use ‘set next statement’ to walk through a rarely used and difficult to reach code path. Ensure that a particular message box is displayed, at least once, to see that it looks acceptable on the screen and reads OK. Use edit-and-continue to fix a minor bug there and then and walk it through to check that it works.

The list is endless. If you don’t work this way, you are missing out.

Paul Sanders
www.alpinesoft.co.uk

59

I do a lot of my low-level (as opposed to end-user-oriented) testing in the debugger. In essence, I don’t trust a piece of code until I have forced it through all possible code paths. This might sound daunting - even impossible - but it is not. With a good debugger - and I se Visual Studio which is truly excellent - you can put your code through hoops in short order. Simulate an error condition by changing the result returned by MySuperDuperDooWopADiddyFunction () to false. Use ‘set next statement’ to walk through a rarely used and difficult to reach code path. Ensure that a particular message box is displayed, at least once, to see that it looks acceptable on the screen and reads OK. Use edit-and-continue to fix a minor bug there and then and walk it through to check that it works.

The list is endless. If you don’t work this way, you are missing out.

Paul Sanders
http://www.alpinesoft.co.uk

60

I fail at unit testing.
I can never figure out what inputs to test and how to know what the corresponding outputs should be.