Coding tests are a fact of life when you interview for a developer job. They’ve been written about plenty, and the conventional wisdom is they’re very useful. For example, the venerable “Joel on Software” wrote about them in his “Guerrilla Guide to Interviewing,” and Jeff Atwood wrote about them in “Getting the Interview Phone Screen Right.”
Well, coding tests make sense for very junior job openings, where you’re looking for just a “coder”. Or, if the candidate has zero prior work experience. Otherwise, they’re worthless.
Here’s the typical coding test
The candidate…
- Visits the company for a day of interviewing, with each interview being 45–60 minutes. One of the interviews includes a coding test, but I have seen interview loops set up with more than one test.
- Is asked to code a simple problem. (E.g., “reverse a string”)
- Is asked to do the coding on a whiteboard, without any assistance.
- Does the test while the interviewer remains in the room, watching.
- Has their answer reviewed/judged/graded by the interviewer. Sometimes others are brought in to review the answer.
All firefighters must be able to roast a marshmallow
Does any of this remotely resemble how you develop code in your company? I’ll bet you don’t code standing up at a whiteboard with someone watching the clock, without any tools. So why do you think it’s a good predictor of anything?
Virtually everything about a coding test, with the exception of the air in the room, is wrong.
The social environment is all wrong
A normal job candidate is already stressed out. Coding tests ratchet up the stress by at least an order of magnitude.
The interview slot is, say, 45 minutes long, and he knows the clock is ticking. Most interviewers say something reassuring, like, “It’s not important to get the right answer. I’m more interested in how you approach the problem.” No candidate believes this. Nor should they, because a candidate with the “right” answer will be perceived as better than one without, all other things being equal.
The interviewer usually remains in the room, watching him as he draws chicken-scratch on the board, chases dead ends, etc. I know I do my best work when someone’s watching me, and waiting for me to finish. How about you?
It’s true that a standard interview’s social environment isn’t identical to a normal work environment. But the intersection between this and a normal work environment is ø.
The development environment is all wrong
When I design or code something, I’m at my computer.
With Python, it’s easy to dive into the interpreter and experiment. But no matter what your coding language is, I’m sure you also sometimes use small code experiments as part of your development process.
I can search technical sites, news groups, or forums if I need to. I can even e-mail or IM someone with a question. When I write code, I use a language-sensitive editor, which highlights syntax errors, displays library documentation, etc.
I have a copy of Python Essential Reference at my side. Maybe I also have CSS Pocket Reference or Python Cookbook. No matter what your language is, I’m sure you also rely on some texts when you code.
None of this exists in a coding test. The intersection between a coding test and a normal programming environment is ø.
You’re testing for the wrong thing
The tests must be simple because time is limited. Hence, the tests are “reverse a string,” “find the largest value in an array,” etc. This is simplicity to the point of triviality, and it’s not testing what you think it is.
The candidate is always obsessed with demonstrating that he knows the language’s syntax. (Most interviewers reassure candidates on this point, and no candidate believes them.) This means a large amount of candidate energy is going into syntax checking, instead of solution crafting.
Then there all the aptitudes and skills that coding tests don’t even touch. You won’t see how the candidate documents their code. You won’t see how the candidate approaches unit tests. You won’t see how well the candidate follows check-in rules. Etc.
It can be insulting
There’s nothing like having a great work history, impeccable references, and a resume with tons of relevant experience — and then being asked to write a function that capitalizes the first letter of every word in a string.
Remember, I’m writing here about someone other than a totally green-behind-the-ears candidate.
It can be sadistic
See the “social environment” section.
A good coding test would take too long
In a good coding test, you’d give the candidate a table, a computer with Internet access, some reference texts, a pad of paper, and three or four hours to work on a problem. And then you’d spend at least 15 minutes to review their solution, given the likely complexity of a programming problem that needed three or four hours to solve.
This would allow for a far more realistic appraisal of the candidate’s skills. It wouldn’t be perfect, but it would have a few things in common with the real world, of which the standard coding tests do not.
The problem is that nobody has the time for this sort of test.
Back to the future
Most (nearly all?) other disciplines don’t do their equivalent of a coding test. For example, if you were interviewing…
- A CFO, you wouldn’t ask her to do a math problem on the whiteboard
- A firefighter, you wouldn’t set the wastebasket on fire and ask her to pour water on it
- A hardware engineer, you wouldn’t ask her to solder a chip to a board
- An artist, you wouldn’t ask her to draw a picture on the whiteboard
They instead rely upon the applicant’s references, their work history, and their answers to questions.
You’re probably trying to discover this with a coding test:
- How the candidate thinks approaches problem-solving
- The accuracy of the candidate’s work history
- The accuracy of the candidate’s representations of language familiarity
- The candidate’s use of good development practices
In which case…
You can uncover all this, with far greater accuracy, by asking good interview questions, and doing good reference checks. This may seem unsophisticated, but given how most companies do both of these, it’s really (unfortunately) a radical suggestion.
Good interview questions should be a given, but many interviewers ask terrible questions. I once interviewed for a VP Engineering position, and was asked if I knew Moore’s Law. I nearly fell off my chair.
And most companies don’t do good reference checks, and many don’t do them at all. A reference gives you a window into the applicant’s performance in a variety of real-world situations, which is orders of magnitude better than a coding test. And you can do a reference check using your own contacts, as long as they’re not at the candidate’s current employer. I’ll take these over a coding test any day of the week.
14 responses so far ↓
Joe // February 15, 2008 at 6:53 PM |
Now I’m curious. What inspired this post?
Coding test during Interviews // February 15, 2008 at 8:27 PM |
[...] from the Seek Nuance blog comments on coding test during interviews: I’ll bet you don’t code standing up at a [...]
jonathan // February 15, 2008 at 10:13 PM |
i completely agree with this post….i recently had to conduct two interviews while filling a coder position. i know the guys were software engineers, so asking them to code something would simply be stupid. i was mostly interested in finding out how easily and likely is it for these guys to figure stuff creatively. also to know if they are in the know of what’s going on in the interweb in terms of design, best practices, trends etc etc when it comes to programming. my system might be slightly flawed, but i need it’s turned out good so far.
any other things i could ask would be highly appreciated
John // February 16, 2008 at 8:31 AM |
Joe: The impetus was the “Coding Horror” post about interview phone screens. In it, Jeff Atwood lauds another blogger’s phone screen advice, which included giving a coding test over the phone. You read that right – a coding test over the phone.
I had an immediate visceral disagreement with this. I considered how my previous companies handled interview loops. The result being what you read here.
John // February 16, 2008 at 8:40 AM |
Jonathan: Thanks for writing. Just so I’m clear, I think coding tests make sense if you’re hiring someone with no prior work experience (e.g., someone fresh out of college, or who’s switching careers), or if you’re hiring into a very junior coding position. I.e., one which is literally only coding.
If they already have work experience, I’m with you: Do a proper background check and ask good interview questions. If you can’t figure them out in 45 minutes to an hour of probing questions plus good reference checking, don’t hire them.
This is perhaps a topic for another post, but I like to ask a mixture of practical and open-ended questions. I’m not a fan of the Microsoft-popularized “intelligence” questions, like, “If your car was full of gas and weighed 3000 lbs., and you drove onto a bridge with a weight limit of 3000 lbs. in its center, would it collapse?”
Bryan McGinty // February 17, 2008 at 8:02 PM |
> Most interviewers say something reassuring, like, “It’s not important to get the right answer. I’m more interested in how you approach the problem.” No candidate believes this.
Well, that is what I am looking for when I ask coding questions in an interview.
> Nor should they, because a candidate with the “right” answer will be perceived as better than one without, all other things being equal.
If a candidate gets the right answer I would assume that s/he has encountered that question before, so I would ask another.
I do agree that asking a question on reversing a string is useless, but asking someone to write something that shows you know how to manipulate bits for a device driver writer position is reasonable. Also asking an interviewee to evaluate a recursive Fibonacci function is valid.
jonathan // February 17, 2008 at 10:04 PM |
@bryan
wouldn’t it just be easier for him/her to show you a project they’ve worked on?
I’ve heard that some companies do not allow interviewers to request code from interviees(for legal reasons), but for exacmple’s sake, lets say you can.
some might argue that they worked in a team an didn’t do all the coding….but then again you want them to work well in a team so it’s not that bad.
i just think that there should be a better way to figure out if the person is the right one for the job without having to code on a white board.
John // February 18, 2008 at 10:15 AM |
Bryan: We’ll have to agree to disagree.
As I argued in this post, I agree with a coding test if you’re interviewing a green, no-prior-work-experience individual for a device driver position. But it’s a waste of time if she has 10 years’ development experience writing device drivers for 3Com and Qualcomm. Asking her a couple of probing questions, and doing good reference checks, will tell you 1e+06 more than making her write bit-twiddling code in an artificial environment.
Rob // February 19, 2008 at 6:43 PM |
For senior folks, I think whiteboard tests are a good springboard to a broader discussion about design ideas. I.e. “Why did you do?” But to expect anyone fairly senior to write significant code on the board in 15 minutes is crazy.
Plus, Eclipse does 3/4 of my work anymore, and Firefox+AltaVista does the other 1/4.
I once had an interview where the guy had a printed program he had people analyze. Instead, he asked me to write it from scratch on the whiteboard. When I did just that, he said my answer was “prepared”. Huh whaaaaa? I smiled and asked him, “Come on, seriously, how many people get that one right?”
Rob // February 19, 2008 at 6:50 PM |
One more thought — about your “sadistic” comment.
One critical thing I look for as an interviewee is “Can I TALK to these people?”
If I put code on the whiteboard and they point out all the semi-colons I missed, or start laughing and tearing it down, I usually just coast through the rest of the interview, thankful that I am about to dodge yet another bullet.
John // February 20, 2008 at 9:42 AM |
Rob: I agree that you would have indeed dodged a bullet. You seem to have a good view of the work-life balance.
But would you agree that most individuals wouldn’t take it as well as you?
Perhaps such companies deserve the staff they hire. But I see the problem as so many companies using them that they’re impossible to avoid. And the companies shouldn’t use them in the first place.
Rob // February 20, 2008 at 12:04 PM |
Ya; I agree white board coding is a waste of interiew time. My only exception is that if it’s done very, very quickly, and used as a lead-in to a broader conversation.
But in general, I always assume that the people interviewing me are very good at programming, and very, very, very bad at interviewing.
The ubitquity of whiteboard coding is good evidence of that.
So I always try to move the interview along, by a) talking a lot about my strengths, b) just admitting “I don’t know” or “I haven’t done much of that” about my weaknesses and getting back to my strengths, c) throwing something on the table, like some diagrams or sample code, to distract them.
If it’s an argumentative crew, I challenge them on some subjective technical point & run the clock down. I know if I’ve won if they say, “Darn, we’re out of time, and I had a whiteboard problem I wanted you to do.”
Jason // June 11, 2008 at 12:30 PM |
I noticed nobody mentioned that another big downside to a whiteboard test is that most of us with years and years of experience haven’t written anything by hand since we were in grade school. Having typed just about every word I’ve written in the past 20 years, my handwriting is barely legible and writing out .net code with the nice long framework call names on a whiteboard makes me feel like someone put me in slow motion. They may give you an hour to solve the problem but in reality one only gets about 10 minutes after you subtract the time it takes to physically chicken scratch it out and periodically erase scribbles only you can read.
John // June 11, 2008 at 6:27 PM |
@Jason: Yep, I agree. My handwriting is sometimes so bad that I have trouble reading it. The entire whiteboard coding test environment is a Fail in every respect.