jump to navigation

Who (Doesn’t) Recruit the Best Computer Science graduates? March 26, 2008

Posted by Imran Ghory in Computer Science, recruitment.

While doing some research on how software companies recruit new graduates I came up with a strategy to break firms into separate categories depending on which universities they targeted for recruitment. The results of this research threw up an interesting anomaly – while in general it followed the pattern one might expect (“high-prestige” technical firms going after the best unis, defence and consultancy firms after the next tier, misc business apps the next tier, and so on).

However there was one notable exception to this pattern. Microsoft.

Despite the firms bad reputation among “geeks” Microsoft is still a prestigious technical firm to work for, one which has a number of notable researchers and developers in a number of fields. Yet when it comes to recruiting students Microsoft seems to have given up believing in itself and is now targeting “average” computer science students rather than the best and the brightest.

But before I go further I’d like to explain my methodology. I built up a database recording which universities are targeted by software companies and used the The Times League Table for Computer Science as a benchmark to rank computer science departments.

The actual information about where firms hire most of their graduates is unfortunately not available to the public, however we do have a good proxy measures. We can see the universities that firms target (i.e universities where they run recruitment talks, events, etc.), which due to these activities very nature is public information. Using this information we can cluster firms together based upon the ranks of the universities they target.

To show by example:

Here we show the universities target by three “prestigious” technical firms (Microsoft, Google and Data Connection).

Google is obviously a “hot” destination for Computer Science graduates and is well known for wanting to hire the smartest people. Data Connection is a telecoms software firm with a very strong reputation in the UK for software development excellence, they’re frequently ranked as one of the most desirable firms to work for in the UK.

The average ranking for a Computer Science department targeted by Google is 5.5, for Data Connection a slightly higher 9.6. For Microsoft it’s 36.2. The difference is staggering.

Google and Data Connection are targeting the top-tier of universities, Microsoft are targeting third/fourth/fifth tier unis, it’s as if they’ve given up on getting the best and have settled for the “average” in-order to avoid having to compete at the high-end.

That’s not to say Microsoft don’t recruit students from the top-tier, but I’d be willing to bet they recruit a lot more from the middling unis. And it’s not to say that there aren’t good computer science students at the middling universities, there are. But there are a lot more (possibly a majority of) “top” computer science students in the top-tier than the middle-tiers.

There was a time not that long ago when a number of very smart people came through the graduate recruitment of Microsoft, many of them are prominent in the field today, but it seems now that Microsoft has given up trying to hire the best and develop them into superstars. And I think that’s a shame for both Microsoft and the industry as a whole.

[Incidently if any of the Google recruitment team read this – please sort out your recruitment events calendar – it’s incredibly poorly designed and nearly impossible to use]


Recruitment and the Mythical Year of Experience December 6, 2007

Posted by Imran Ghory in recruitment, Software development.

When it comes to recruitment the basic unit of measurement for ability is “years of experience” – this is something that many developers very vocally consider inappropriate, often due to the amount of experience being requested seeming excessive. However if you consider it from the viewpoint of the company hiring, it does make a certain amount of sense.

There is a strong correlation between the years of experience an average developer has and their ability as a software developer.

You are not the average software developer.

As has been discussed elsewhere in the blogosphere recently, if you read programming blogs for fun then you are not the average. Most software developers do not read programming blogs or even programming books for fun. Not that there’s anything wrong with that, but many software developers seem to have an innate assumption that other software developers are like them (i.e. I read books, so they must read books too), which in part is responsible for the frustration with the “years of experience” measurement.

The average developer’s skills are closely correlated to the amount of experience they have because for the average developer on-the-job experience is where they learn and become better developers. Hence it makes sense to use years of experience as a measure, it’s not exactly what we want to measure but it’s a better proxy measurement then anything else in common use.

But the next obvious question is: Why do firms ask for many more years than they seem to need ?

Lets consider a company that goes out and interviews a lot of people who’ve applied for a job. Say the average applicant applying has five years of experience, and from all their interviewing they find the candidates are too weak for the role. So they bump up the requirement to seven years. Not because they don’t think there’s anyone with five years experience who can do the job, but that the average developer with five years experience can’t and that the number of people with five years experience who can do the job is significantly less than those with seven years experience.

It’s purely a number game. If only one-percent of those with five years experience are good enough for your needs but five-percent of those with seven years are, then requiring seven years of experience will significantly decrease the number of interviews you have to conduct. Say you expect to interview a hundred candidates – would you rather be left with five candidates who are strong enough (and can then be judged on other criteria such as fit, etc.) or just one candidate who may or may not be a match for your company ?

However it’s actually worse than this situation, as the average software developer is actually significantly better than the average software developer who’s on the job market. People who are above average for their level of experience get hired very quickly because firms realize a good thing when they see it. People who are below average stay on the market for longer and therefore apply for more jobs. So not only are the people looking for jobs worse than average, the worst are also applying for a lot more jobs.

This means that if you want someone with the skills of an average developer with five years of experience, you might have to actually hire someone with ten years of experience – who’s significantly below average for their level of experience but is roughly equivalent to the average developer with five years of experience.

So if you’re an above average developer, you need to realize that the “years of experience” requirement isn’t written with you in mind. Rather it’s written for the below average developer on the market for whom it actually makes sense.

Why logic puzzles make good interview questions January 10, 2007

Posted by Imran Ghory in job interviews, recruitment.

Logic puzzles in interviews seem to be one of those things that everyone either loves or hates, but speaking as an interviewer I find logic questions critically important in deciding between candidates – far more than “behavioural” or “situational” type questions that HR peeps seem to favour.

My aim in this post is to convince those of you that hate them, that they are actually useful questions to ask (and be asked) and give everyone else an insight why interviewers especially at big tech companies seem to love this type of question.

To begin with I feel I should draw a separating line between logic puzzles and brain teasers. Logic puzzles are of the type where there’s no “trick” but rather puzzles which have answers that can be logically deduced. Often these are puzzles where there are a number of different valid solutions but you’re asked to find the optimal solution, examples of this type of question include the Rope Bridge and The Orb.

Brain teasers however often have a “trick” – something unusual you have to guess often about the assumptions of the problem. While these can provide some insight into the candidate I think their value is limited, and that they’re far weaker than pure logic puzzles.

I’d like to show what an interviewer can find out about a candidate from a logic puzzle via an example, walking through an interviewer asking the question and showing how various canidates respond:

Interviewer: Imagine you have eight coins, seven of which weigh the same and one that doesn’t (it’s heavier). You need to use a pair of scales to find out what’s the odd one out.

If this seems familiar it probably is, this is apparently one of the most asked logic puzzles in the known universe. It’s used in interviews by Microsoft, Amazon, Google, and probably hundreds of other firms. It’s also in pretty much every book on the topic and on dozens of websites. Now for some example responses:

Candidate Alice: What’s a pair of scales ?

Candidate Bob: How much heavier is the other coin ?

Candidate Carol: Well you could weigh them one-by-one until you find the pair which is uneven, and then….

All of the above responses are good, both Alice and Bob recognized that they had been given a problem they didn’t fully understand so they took the step of asking relevant questions to clarify the problem in their own mind before trying to solve it.

This is a situation that occurs frequently in intellectually demanding jobs, people are frequently asked to solve problems for others. Everyone has different experience and background and will understand the problem differently. This a major advantage, in a team diversity will help you come up with new solutions to a problem. However it also means that everyone will interpret the problem differently, and asking question to clarify what the problem is can be a key part of ensuring everyone’s on the same page.

Carol’s right too, she’s come up with a valid solution right-off-the-bat, it’s not the optimal solution but it is a correct one. We can discuss it and see if she can improve it.

Now a bad response:

Candidate Dan: I don’t know.

If the only purpose of asking logic questions was to catch people who gave this answer, then they’d still be worth asking. This is a candidate you never want in your team. I have three hypothesis about Dan-esqe candidates,

  • They really can’t solve the problem at all
    • This indicates the candidate struggles with problems which intelligent 5 year-olds can solve, not a good sign.
  • They can’t be bothered to try
    • Not a good sign of motivation
  • They don’t want to answer it
    • If they can’t explain why the don’t want to answer the question then their communication skills are probably too weak. In most jobs refusing to do something without giving any reason is likely to be unconducive to a good working environment.

Luckily all three are grounds for rejecting a candidate so interviewers don’t have to think too hard about which the candidate falls into.

So lets look at the next stage of this response, where the candidate has understood the problem and has come up with the trivial solution that Carol has above.

Interviewer: Yes that would work, how many times would you have to use the scale to find the solution using that method in the average and worst case ?

Candidate Carol: Four and eight respectively

Interviewer: Are you sure about that ?

Candidate Carol: hmm, four and ten ?

Carol’s initial response is both a good and bad answer, Carol managed to correctly estimate the average and worst times, showing an understanding of how the speed of the solution depends quite a lot on luck. However her answers aren’t the exact correct values, and worse still when she’s challenged she come’s back with a worse answer indicating that maybe she just guessed.

At this point the interviewer might start to push her on how she got those numbers, if it turns out she starts wildly guessing answers while under pressure, she might not be suitable for a work environment which is pressure heavy and detail-critical.

But now lets move on:

Interviewer: So can you think of a better solution to this problem which lets you use the balance fewer times ?

Candidate Errol: You can split the coins in half. weigh them against each other, throw away the lighter ones, and repeat until you only have one left.

This is a good answer, I’d expect anyone coming from a Comp Sci or Maths background to get this far without any help. People from other backgrounds I might nudge in the right way and see if they can find the solution.

However if you’re from the first group and don’t get this solution I’d be a little worried, it wouldn’t be an instant reject but it’d definitely be a question-mark. Maths/Comp Sci people should have seen similar methods of tackling problems as part of their studies, and should be able to reason about this problem from their previous experiences.

If the candidate came straight to this solution skipping the obvious solution then I’d ask them about the average/worst times for this solution instead. But otherwise back to improving:

Interviewer: Do you think this is the best solution ?

Candidate Fred: I think it is

Interviewer: (smiling) I’ll give you a clue, it isn’t.

Interviewer: You’ve come up with solutions balancing one-against-one and four-against-four, what else can you do ?

Candidate Fred: Three-and-Three ?

Interviewer: What do you learn if you do that ?

Candidate Fred: Which three is heaviest

Interviewer: What if the scale is balanced ?

Candidate Fred: The heavy coin must be one of the other two — you can split the coins three ways every time.

Interviewer: Yep, that’s the optimal solution, do you know for N coins how long it’ll take ?

Candidate Fred: About Log N

Interviewer: What base ?

Candidate Fred: Base 3

That’s typically how the best solution is arrived at by a good candidate. How much help I give the candidate depends how much they’re struggling, some candidates just click about going through the other combinations and come up with the answer. Some like Fred manage to come through with a little help, some candidates just get stuck. I think you can guess which order I want to hire those candidates.

And for bonus points:

Interviewer; Can you prove it’s the optimal solution ?

Candidate Goyle: No I’m not sure how, but I’m fairly sure it is.

Candidate Hermione: Each weighing gives you three pieces of information (if the left side is heavier, if it’s balanced, if the right side is heavier), that is one trit (trinary bit) of information. As the heavy coin can be anyone of the N you need at least enough trits to give N outcomes. Which means you need at least log3 of N goes, which happens to be how many our solution takes. So there can’t be a solution better than the one we discussed.

I don’t really expect a candidate to be able to answer this, but it’s a good sign if a candidate can. People who are solid on the practical side but can excel on the theoretical side as well are rare indeed and it’s well worth identifying them.

So hopefully that’s shown you that there’s a lot of information about the candidate that an interviewer can get by asking a logic question. The interviewer doesn’t just want the candidate to show off their intellect by coming up with the correct answer. They want to see the process by which the candidate got to that answer, as that process is likely to be one that the successful candidate will have to go through frequently during their job.

If you’re interested in learning more about this sort of logic puzzle I’d recommend Moving Mount Fuji (the book; not the task) which gives a good history of how logic puzzles became a part of corporate interviewing. If you’re an (ex)student looking for your first job in the tech sector I’d recommend Programming Interviews Exposed which gives lots of logic and programming questions which are typical in technical interviews.