Scaling pains

Realistically, there are probably a bunch of people at the company who shouldn’t be here. And part of my hope by raising expectations and having more aggressive goals, and just kind of turning up the heat a little bit, is that I think some of you might just say that this place isn’t for you. And that self-selection is okay with me.

The above quote was said by Mark Zuckerberg a couple of weeks ago. This comes in preparation for a looming recession. Meta alongside other big tech companies, is also freezing hiring across many areas. Mark wants a bigger bang for the buck. I am not surprised. It’s estimated that Meta currently employs between 25 to 30 thousand software engineers. That’s a hell lot of expensive salaries to pay. I don’t work at Meta, but if I did and heard this I wouldn’t feel great about myself. It’s quite simple: if an employee is underperforming and are allowed to continue that trajectory they set the new (lower) bar. It feels that Mark is not happy with the existing performance and wants to push that bar higher. The ends are legitimate, the means, who knows. I feel that this might be outside of his control. This is not specific to Meta. Rather a more general problem across big companies.

The Mythical Man-Month discusses at great length about team sizes and the complexity of having a lot of humans working in the same product. There are some fundamental differences, compared to typical teams I have seen across the years:

• Keep teams small - around 10 individuals. A single person calls the shots (the surgeon). They are aware of the whole design and all the code. The hypothesis is that it “ensures the conceptual integrity of the work”. Moreover, because there’s a single person driving the team “there are no differences of interests”.
• Scaling to 200 engineers entails someone coordinating these 20 surgeons. This person, or small group of individuals, would keep the conceptual integrity of the whole system.

The book then goes into great detail about this approach. But it also warns about the downsides. The most striking is potentially oppressing the creativity of 180 individuals - 20 odd people have unilateral power over their vertical. Is this even realistic these days? Or even desirable?

During my time at a now public company, we went from 120 odd people to more than a 1000. All this happened in 3 years. It was a period of massive growth - both in terms of headcount and internal processes. When I joined, the company was operating on what I call an agency model. We had around 30 engineers underneath a single Product Manager (PM). He was aided by a couple of Scrum Masters. Work was treated like a project - with a beginning and end. Once done, the next project would come. Rinse and repeat. Tasks were discussed during sprint planning. I am being kind when I use the term “discussed”. The more accurate expression is “handed over”. Engineers had no ownership over the work they were doing. I suspect that this is true in most companies, even when they count themselves as proud agile practitioners. There was little collaboration between iOS, Android, Web or backend. There was a strong bound among individuals inside each function. Despite a big detachment between engineering and the end-user, engineers were quite efficient - but not effective. Everyone knew most of the codebase. People were not silo-ed and they would jump between different parts without too much of a hassle. I could see that the team of 8 individuals I was managing was busy.

Two and half years later. We are now knees deep into the famous Spotify Model. Every 1:1 with my Manager is about why I am being so slow to hire people. I find it impossible to explain to myself why we are hiring so many. I cannot conceive of what these people would work on. I am now managing more than 25 individuals - my team tripled. Each squad has at least two iOS engineers. The ones that don’t, are either considered understaffed, or of low importance to the company. The reasoning for two engineers is simple: if an engineer goes on vacation the squad’s work cannot stop. Engineers are now swamped in meetings. Either because they are attending to their own squad affairs, or aligning with someone else. More than 60% of the engineers are way below their capacity. They spend time on things outside product work. This isn’t necessarily bad. It keeps people motivated and things interesting. In my head it becomes difficult to justify having 25 people, when 10 - 12 can do the same job. I see the same signs everywhere in the industry. At one point a director wanted to triple is organisation in two months (from 20 individuals to 60). These are the true colours of a bloated, inefficient organisation.

These are the things I have noticed with this model. They are not exclusively derived from my own experience, but from things I have read and heard:

• Most teams are not self-sufficient. There was always something needed from someone else. Engineering Managers, Product Directors and whatnot were dragged into meetings to make sense of these dependencies. If a team cannot be split into two, instead of growing, they should do less. Many people believe that by throwing more people into a problem, it will go away. This is called wishful thinking and the Mythical Man-Month discusses this topic in great length.
• Few squads were capable of justifying their head count. A squad lead must be held accountable when they ask for a bigger budget. Why is that? What is the purpose? And what are the success metrics to justify hiring? This is true for both squad leads, but also for Tribe Directors. Especially, when they want to spawn another squad within their organisation.
• Often squads had little to do, while a few others were overworked. In an agency model, we would ask engineers in the former group to work with the latter. If these squads are in different tribes, this would be impossible to do. Directors can be quite possessive about their engineers - even if the net result would be positive.
• If a squad, or tribe, expands, it should also shrink. But the latter is never encouraged, or appreciated. It’s uncommon to find managers that are frugal with their budgets and honest about their team’s workload. It was also frowned upon when an engineer wanted to move from one team to another. I had difficult conversations with other managers about this topic. The question was always the same: why do you want to unload that engineer to my team? Rather than appreciating that an engineer chooses their team, among many.

I see a couple of inflection points when scaling product teams:

• 2 - 3 engineers: Engineers are self-sufficient. Medium to senior engineers can organise work among themselves. Information flow has little resistance. Everyone knows what others are doing. Depending on how specialised a task is, anyone can pick it. A PM, or someone that represents Product exists. Trust is high, so common problems are easy to overcome. There’s no politics, since there’s no critical mass. Hiring the right people is one of the most important things the CEO can do.
• 4 - 7 engineers: It’s doable for engineers to organise their work, but success is not guaranteed. Specially when tasks have ramifications on someone else’s work. At this point an Engineering Manager is helpful, to make sure everyone is rowing in the same direction. Competent coders do not necessarily make competent facilitators. Assuming a PM exists, their focus is not organising work. This is a recurrent issue I see in teams: PM acts as a Scrum Master, instead of focusing on product. The team is well gelled, since there aren’t many people. Culture wise everyone is well aligned. If politics starts to manifest, at this scale, managers need to intervene immediately. This is likely toxic behaviour from one or two people. Hiring remains important and bad elements are removed immediately. Processes are light. If meetings take more than 20% of the ICs time, there’s something wrong.
• 10 - 20 engineers: At least two PMs exist. A PM shouldn’t have more than 5 - 6 engineers as part of their team. They should report to a Head of Product, or the CEO. A corresponding number of EM’s should exist. An EM serves as both a partner and counterbalance to the product. A company now needs to orchestrate two, or more, product teams. How well the teams work together, will dictate the success of the company. This is when you see if a Head of Product and Head of Engineering are worth their salt. Information flow is now a conscious effort from everyone - it’s no longer an afterthought. Culture is likely ramified, with some individuals not adhering to the original behaviours. This is not bad, rather a consequence of having more people. Politics might exist, but they can be managed to an acceptable level. Processes continue to be light. Managers should have some touchpoint (either sync, or async) about what’s happening within their team. The CEO, or someone from the C level, should share regular updates. This brings alignment and motivates everyone.
• +20 engineers: More teams to coordinate. Like the previous step, but with a higher chance of people stepping on each other. Politics become apparent, because individuals have not built trust with each other. This is a natural consequence of scale. The company starts to lose the “family” feeling that once had. It’s difficult not only to get to know people outside your team, but to maintain previous relationships - there’s not enough time. Managers are attentive to opportunities between unrelated teams. Managers of managers make sure their direct reports are aligned. With the wrong incentives, managers can, and will, work against each other. The CEO and CTO should pay attention to what’s happening with the ICs. This can be done via skip-level meetings. Every two months, at least, everyone should have the opportunity to express their thoughts (anonymously). This can be achieved with platforms like Peakon. Managers should continue to strive to make sure their teams are independent. This is a simple proposition, but can be difficult to accomplish.

All the above is underlined with one fundamental truth: every new hire creates a new set of challenges. Most managers unfortunately don’t value the lessons from books like Mythical Man-Month. Either this is conscious, or not, I am not too sure. It’s incredibly difficult to scale engineering teams well. There’s this romantic view that problems are solved by throwing engineers at them. They are not, and if one isn’t careful it will backfire.