How to Evaluate the Performance of Software Engineers

If you manage software engineers, evaluating how well they perform is essential.

You’ll want to know if they are productive, write good code, focus on the right things, leave good PR comments, etc.

So how do you do this with confidence, without wasting a lot of time and effort?

One way to evaluate someone’s performance is by asking peers and relying on your intuition. It’s probably the most common way engineering managers gauge performance, but intuition might not match reality.

When I first started managing, I didn’t quite know how to dig in and spot issues in my team early enough, only to have my manager point them out to me.

Does that sound familiar? Maybe that’s why you are here, reading this post.

This was an actual conversation with my manager. He’s incredibly technical and could identify issues from a mile away, things I wasn’t able to see even though they were staring at me.

After this happened a couple of times I decided I had to learn this skill. So what did I do?

Well, I booked a meeting with him, took a notepad, and asked exactly what he does and how he’s able to spot these things. This way, I could identify issues quicker so I could coach my team more effectively.

This was a few years ago. Since then, I have expanded on those fundamentals and have developed my own “performance zoom-in” evaluation system which I’d love to share with you.

A word of caution

If you are looking for a tool that generates a performance score, this post is not for you. Check out Waydev or Pluralsight Flow. I don’t personally use them, but that’s for you to decide.

The reason I’m not a fan is that every organization and engineer is different and trying to reduce them to a metric will give you many false positives/negatives. You’ll inadvertently end up punishing great software engineers that don’t fit the mold.

Instead, use this zoom-in to ask questions, understand the situation, and adjust your management style to the individuals you are coaching.

What is this “performance zoom-in” thing?

It’s a set of questions that can help you get a broad idea of how a software engineer is performing.

Using it you can “zoom in” on your direct reports or skip-levels to confirm firsthand that everything is going well and get ideas on things they can improve.

This can be done more frequently for new hires or less frequently for engineers that have ramped up and you want to check on them every now and then. (see Gatekeepers and Gardeners)

In the end, you’ll arrive at one of these outcomes:

1. 💯 Doing great, rock on!
2. 🧐 Something seems off, let me ask what’s up and offer help
3. ⚠️ A pattern of underperformance is starting to emerge, I need to take action

I’ve got too much going on, why bother?

This can help surface issues sooner and give you more ideas for actionable feedback.

It can be particularly helpful for software engineers that have just started in a new role and may need help calibrating expectations.

For example, it’s possible for a new TL to think a team member is “doing well” when in fact they should set the bar higher and expect more – a great occasion for the TL to grow as well as the IC.

However, if you just take their word you may find yourself in a situation where people are stagnant and may think they are meeting or even exceeding expectations when that might not be the case.

It is more important to take a glance on a regular basis than doing a thorough and time-consuming investigation during a performance review. Or even worse, not doing anything at all.

Even a glance can be revealing. Then, you can further dig in if you need to.

Running a Performance Zoom-in

Look at Github

Open up the engineer’s Github user page and look at their contribution graph. Then ask yourself:

• Are there obvious gaps?
  • Are they doing too much research? Too little?
  • Is it because of vacation?
  • Is it because of designing a new system?
  • Is it because of bugs requiring too much investigation and too little coding?
• Is the average GH contribution color very light green?
  • Is it because of low contributions?
  • Did they make any giant refactors, variable renames, or project splits that resulted in dense contributions in a few days and brought the average contribution color down?
  • Are they squash-merging?
• Drill into a “good week”
  • Do they make tons of small commits or one large?
    • Maybe they can strike a better balance?
  • Do they work on a project all week and commit on Friday?
    • Does it make sense to spread out commits?
• Do they get stuck for long periods of time?
  • Maybe they can talk with their TL/mentor more frequently?
• Sample some PRs
  • Are they substantial feature additions?
  • Are they just renaming things and moving code around?
    • It’s ok to have some of that but if it’s the majority of their contributions, you should drill in and ask them
  • Are they infrequent, small, 1-line PRs?
    • Ask if they were working on bugs
    • If this is common, maybe they should work with their TL to balance the variety of tasks they work on
  • Do they get a lot of comments on their PRs?
  • Do they need many code revisions to ship a PR?
  • Does the code look legible? Can you make sense of what’s going on even if you aren’t familiar with the project?
  • Do they explain what the changes are in the PR description?
  • Do they link to relevant issues/jiras/slack convos?
• Sample some commits
  • Do the commit messages make any sense?
  • Are they documenting their thought process?
  • Do they leave links to other issues or conversations, documenting the decisions that were made?
  • Are commits too small? Too big?
• Do they do code reviews?
  • How many? Too many? Too few?
  • What types of comments do they leave in PRs?
    • Do they just respond with “ship it”?
    • Do they just find nits?
    • Anything substantial?
    • Are their comments constructive? Are they too nitpicky? Too particular?
    • Can you find anything you’d change in that one sampled PR?
• Open up another well-performing engineer’s contributions of a similar level
  • How do they compare?
    • Volume of work
    • Volume of PRs
    • Quality of code

Check out their weekly status updates

• Are they consistent? Do they submit them every week?
• Do they work on impactful projects? Is it too much fluff?
• Do they keep good notes? Do they link to relevant issues/jiras/slack?
• Do they balance mentoring vs coding?
• What types of projects do they work on?
  • Is it mostly bugs? Jiras?
  • Do they work on bigger projects?
    • Too big / too small for their level?

For managers:

• Should they delegate more?
• Do they delegate too much?
• What’s the balance of manager work vs coding?
• Do they review their team member’s status updates?
  • Are they consistent? Timely?
    • Do they take too long to review?
  • What kinds of comments do they leave?
    • Do they ask questions?
    • Do they celebrate wins?
    • Do they share/document feedback?

Slack

• What types of conversations happen in the team channel?
• Is there comradery? Is everyone supportive and helpful?
• Does it take them too long to reply?
• Do they reply too quickly?
  • Do they miss things? Or don’t give other teammates a chance to respond?
  • Do they promote an unhealthy “always available” culture?
• If the team channel is too quiet, do they chat in DMs?
  • Maybe encourage them to use public channels so more people can benefit?
  • If there are too many details in the public channel maybe create a backroom?
• Do they spend most of their time in casual channels?
• Do they ask too many questions, too quickly?
  • Can they try on their own for a little longer before asking certain questions?

Pagers / Alerts

• Do they triage pagers or just dismiss them?

• Do they acknowledge on time?

• Do they drill in to find the root cause?

• Do they spend too much time drilling in?

• Do they do weekly audits?

• How heavy is the alert volume?

• Do they have an alerts/robots slack room?

• Are they paying appropriate attention?

Calendar

• Do they have too many meetings?
• Are the meetings too long?
• Do they meet with their ICs for 1:1s?
• How often do they meet with their triad?
• Do they do weekly alert reviews?
• Do they ever reach out to other peers to grow their network?
• Do they do planning meetings? Retros?

What to do when you are done researching?

By now you should have a general idea if the engineer is doing great, if you need to ask questions to clarify, or if your feedback hasn’t been acted on and a bad pattern is emerging.

If you identify something for the first time, let them know but also give them the benefit of the doubt – it might just be an off week.

If the same behavior persists, more detailed notes with specific examples can help with coaching and performance management. Oftentimes, little things are not significant on their own but on aggregate may reveal a more worrisome pattern.

Here are some sample conversations to help you manage different outcomes:

Outcome 1: Doing great, rock on! 💯

Sample conversation:

Hey, I was looking at your recent contributions and noticed that you are doing really well.

Specifically:

• Your code contributions are consistent and you seem to be working on a healthy mix of new features and bugs
• When you spot something that can be improved you don’t hesitate to open quick refactor/cleanup PRs but at the same time you don’t just focus on cleanup and ignore the team priorities
• Your PRs don’t need too many comments before shipping
• You seem to be leaving good comments on your team’s PRs

All in all, great work! Keep it up!

Outcome 2: Something seems off; I need to check-in 🧐

Sample conversation:

– Manager: Hey, I noticed this month might not have been as productive. Specifically, there was a period of 2 weeks where your contributions were low. Is everything ok?

– Engineer: Oh yeah, I was tracking down a handful of gnarly bugs, profiling the JVM etc. All that’s done and you can see this past week I’m back cranking.

– Mgr: Oh cool, that makes sense.

Outcome 3: A worrisome pattern is starting to emerge ⚠️

Sample conversation 1: you are missing something

– Mgr: Hey, I noticed that over the past month your contributions are not very strong. Last time we talked about it you mentioned that it was because you were focusing on bugs. Is that still the case?

– Eng: Yeah, I only get bugs assigned and this codebase is a mess and it’s really hard to triage. And you know that these sorts of bugs take like a week to find and a single line of code to fix. How am I supposed to have strong contributions?

– Mgr: Oh I see, I hadn’t realized you were only working on bugs. My bad! I’ll try to rotate bugs and features better across the team.

Sample conversation 2: they need to pick up the pace

– Mgr: Hey, I noticed that over the past month your contributions are not very strong. Last time we talked about it you mentioned that it was because you were focusing on bugs. Is that still the case?

– Eng: Yeah, I only get bugs assigned and this codebase is a mess and it’s really hard to triage. And you know that these sorts of bugs take a week to find and a line of code to fix. How am I supposed to have strong contributions?

– Mgr: As you know, we try to rotate bug fixes to everyone on the team. Do you feel like you are getting more bugs to fix or harder ones?

– Eng: Well, I don’t know. All I know is that I have too much on my plate.

– Mgr: Ok, I hear you. At the same time, we’ll need to find a solution because you should be able to tackle these faster and get to feature work as well, just like the rest of the engineers. Would it help if we paired on a couple of bugs? Maybe I can share some tips to help you speed up triaging.

– Eng: I would actually appreciate that, thank you.

To sum it up

Being able to confidently evaluate the performance of software engineers on your team is an important skill for engineering leaders.

Metrics and tools might be something you can look into, but at the end of the day, things are not black and white. All engineers are different and the way teams and organizations are managed can also differ.

So, instead of using rigid metrics or just relying on gut feeling and word of mouth, use a set of questions that can help give you a better idea of what’s going on.

Then use what you find to check in early, confirm that your findings are accurate or if you are missing something, and then provide feedback.

This way your engineers have an opportunity to be heard but also receive actionable feedback so they can continue growing.

This post was originally published on September 1, 2021

---