Balancing technical uplift with product development

Closely behind the holy wars of programming languages, is the battle of technical uplift versus product feature development. It exists in every software organisation around the world, however, some are able to better balance the two more evenly. Software developers are best placed to understand how to keep their platforms stable, online and scaling with customer growth demands. This results in the software team typically being motivated in focusing on technical uplift initiatives to improve scalability and reducing technical debt. Whereas the product team knows and understands the customer, and are motivated to continuously find new ways to engage and inspire them. This results in the product team focusing on initiatives that drive new capabilities and features that deliver a better customer experience than their competitors. Finding a technique to balance both of these potentially conflicting and competing priorities can be challenging and fuelled with justified passion from both sides. In mature and collaborative organisations though, it promotes healthy and constructive debates on what initiatives add greater value.


It is important to note that when referring to technical uplift, it is not specifically referring to only technical debt. Although technical uplift sometimes needs to occur due to long standing technical debt, technical debt is the result of specific historical decisions knowingly being made that results in a platform becoming more expensive to maintain or potentially scale. Some technical debt is bad debt, in that the cost of repaying those decisions becomes exponentially higher as time passes. Whereas other debt is more easily repaid at a later point in time. A crucial call out to understand is that new technical debt is accepted and taken on as a team, it isn’t forced onto the team by stakeholders - that is not a software engineering team culture you want to let evolve.

Why the decision is not binary

Balancing technical uplift and product feature build rarely needs to be a binary decision. From a software developer’s perspective, they work day in and day out within the platform’s codebase. They understand many of the pain points that result in scalability issues, areas of the code that every software developer avoids working on because the slightest change can cause unknown production issues, or the frustration of repetitive and manual processes that have evolved over time making the team ineffective. By failing to ensure the software team has a voice in prioritising technical uplift can result in three key impacts. Firstly, the organisation is not trusting the most experienced people working on the platform who understand the key technical issues being faced and how to improve them. Secondly, the organisation is not solving fundamental problems within the platform that will enable faster and more effective development and deployment of new product features into production. Lastly, without trusting the team and empowering them to improve the codebase and platform, it will be detrimental to the culture within the team and ultimately the higher performing software developers will disengage and move on.

From the product perspective, product owners and product managers attempt to deliver the best possible customer experience. They listen to customers, gather feedback and aspire to release a range of new features to continually attract new customers or retain existing ones. They understand that doing this well will lead to an increase in profitability for the organisation. By not prioritising the development of new capabilities, a platform will be irrelevant as it eventually loses customers to competing organisations and platforms. The thought of losing customers is demotivating for anyone working within a product role and can lead to an increased turnover of product owners and managers within the organisation.

As mentioned above this is not a binary decision, in that an organisation does not have to focus 100% on technical uplift or 100% on product feature build. A healthy balance needs to be found between the two to ensure platform scalability, new product development and retention of team members within the organisation. Even an early stage startup focused entirely on growth needs to dedicate a percentage of time on technical uplift to ensure platforms don’t become too expensive to scale and maintain which is known to have crippled many startups throughout their scale-up phase. Twitter shared an insightful post on their blog many years ago that spoke to serious performance issues in relation to their search capability. It was a result of years of technical debt in their Ruby codebase which made it challenging to add features and improve the reliability of their search engine. However, the organisation trusted and empowered the software engineering team to implement a solution which resulted in the rewrite of their search capability which improved searching speeds on twitter three fold.

Don’t be binary

In a previous software leadership role, I remember walking into a team facing considerable technical challenges specific to the scalability of the platforms running in production. This was due to the organisation scaling up incredibly quickly over their recent years which had resulted in unfortunate architectural decisions being made, as well as a continual focus on product growth with little to no priority on technical uplift. The software team was frustrated and they were beginning to see a higher than normal number of resignations. This needed to be resolved. Over the next 24 months the team began focusing their efforts on fundamental technical uplift. Platforms were decommissioned, others were re-architected, all alongside a much needed modernisation of software engineering tools and practices. The team was investing effort into rapidly evolving their platforms to support the organisation’s expansion aspirations. The first 18 months brought a breath of fresh air within the software team as there was now motivation and support to improve. At this point in time, the software teams were focusing on average 80% of their efforts on technical uplift which was the nearly exact opposite 18 months prior. Throughout the next six months though, the team that had once only wanted to improve their technology and platforms began to question their purpose within the organisation. Sure, their platforms were considerably more stable and scalable, they were even able to deploy production changes within hours instead of months, but something was missing. The organisation's product vision was missing and teams started asking questions:
  • Why are we building this product?
  • Who are our actual customers?
  • What is our vision for this product?
The mindset across the group began showing signs of drifting back to where it was 18 months prior, which leads me to the point. It is unhealthy to have a software team focus 100% of their effort on technical uplift and ignore product feature development. Likewise you cannot expect a software developer to remain engaged if focusing solely on product feature development and disregarding any technical uplift or hygiene. A balance is required, and this balance is different in every team. This balance may also evolve every 6-12 months due to an organisation's ever changing environment. As a software leader, you need to be constantly pulse checking where teams are spending their effort. Your role is to motivate teams with a balance of technical uplift and product feature development and ensure a purpose for both exists, and is known.

How to find a healthy balance

To solve the challenge between technical uplift versus product feature development it is important to accept that there needs to be a balance of the two at all times. Rarely if ever, will a team need to focus 100% of their effort on one or the other, nor should you want a team to due to the reasons discussed above. As with most aspects of software leadership there isn’t a single approach that can be taken to achieve this outcome. However, the below approaches act as guidelines that can help ensure a balance between technical uplift and product feature development.

Clarify and make clear responsibilities

It is important the software team is responsible and given ownership for two distinct areas. The first being how a feature is solutioned. Software developers are the closest to the code, they understand the impacts different solutioning decisions have on the maintainability and scalability of the codebase. Ensuring software teams own solutioning, it avoids non-technical individuals outside of the team who are less qualified making fundamental technical decisions that could be detrimental to the health of the platform. As part of solutioning, teams should always gather input from relevant stakeholders but this is purely in a consultative capacity. This will also reduce the potential of corners being cut in solutions that normally always results in incurring technical debt.

The second is owning work effort. The software team undertakes the development, testing and deployment of work items, thus are the only ones qualified to estimate the work effort required. Is it impossible and naive to believe a stakeholder can define the work effort required by the team when they do not understand the low level intricacies of the platform. Stakeholders are in their right and should share constraints that may be related to compliance, marketing deadlines or similar with the team. If a stakeholder attempts to define work effort and commit a team to it, as a software leader you need to clearly set expectations that they do not have the relevant expertise to make this decision and that it is up to the team to own work effort estimation. It can be necessary to communicate this message with senior leaders across the organisation, while keeping individuals honest when noticing it occurring.

Where a stakeholder may get involved in conversations is around scaling back feature scope to reduce work effort required if there are legitimate time constraints. By allowing the software team to own work effort estimation you are ensuring that non-functionals are taken into account, that technical unknowns are accounted for, and that the team will solution in line with the software team's target state.

Making technical uplift visible

If technical uplift is not documented, not prioritised and not made visible, team's will always struggle to make progress in it. It would be like walking up an escalator in the wrong direction, the amount of additional effort required is considerably more. Teams wouldn’t and shouldn’t start work on a new product feature without first understanding the outcomes, the priority, defining relevant user stories and documenting a solution. Why should technical uplift work be any different? Documenting technical uplift involves the team defining their most important initiatives, determining the value, prioritising them relative to each other and then work effort estimating them as a team. Try to focus on the immediate top 10 technical uplift initiatives compared to hundreds to avoid overwhelming not just the team but the wider organisation. With technical uplift documented it can now have an opportunity to be discussed and prioritised alongside other priorities coming into the team. It is no longer a hidden list of vague work items in the minds of people within the team.

Justifying technical uplift initiatives smarter

Outside of making technical uplift visible, the other single reason why it can struggle to be prioritised comes down to an inability to justify it in a way that is understood by the organisation. As a software leader you are responsible for being the voice of reason and support when it comes to prioritising technical uplift. Your role needs to understand how the organisation justifies and prioritises initiatives so you can apply the same approach to technical uplift. Below are three common ways organisations prioritise:
  • Return on Investment (ROI):
    • ROI = total estimated value / cost to implement
    • Cost to implement includes all costs associated to develop and maintain including people costs, infrastructure costs, license costs etc..
  • Competitor feature parity.
  • Cost of Delay (CoD) is the cost to the organisation of not implementing the feature by a certain point in time. This may be lost revenue, compliance fines or lost customers due to a competitor moving faster.
Organisations will quite often use a combination of prioritisation methods as they mature to ensure a more complete justification. By understanding and applying the same justification to technical uplift initiatives it creates a level playing field allowing apples to be compared with apples.

Let’s explore a simple example using ROI as the only method to prioritise both a technical uplift or product initiative across an organisation. The product initiative will bring $240,000 of profit in the first year, at a total work effort cost of $75,000 (which equates to three software engineers for three months earning $100,000 a year). There is also a yearly cloud infrastructure cost of $12,000. ROI for the first year is calculated by:
  • ROI = total profit / total cost
  • ROI = $240,000 / ($75,000 + $12,000)
  • ROI = 2.75

This provides an ROI of 2.75.

The technical uplift initiative will deliver $420,000 of savings by deprecating a legacy platform reducing the need for a small team currently supporting it. The total work effort cost is $100,000 (which equates to 2 software engineers for 6 months earning $100,000). ROI is calculated by:
  • ROI = total profit / total cost
  • ROI = $420,000 / $100,000
  • ROI = 4.2
This provides an ROI of 4.2.

Without taking into account any other prioritisation approach it is clear that the technical uplift initiative provides a higher ROI to the organisation and should potentially be prioritised first.

It is also within your best interest to think carefully about the real costs of managing platforms that technical uplift will simplify and improve. If an initiative is wanting to automate deployment pipelines for a specific platform, it is valuable to understand the costs in maintaining the existing manual deployment processes. You may have four teams spending three hours each week manually releasing code. Over the course of just one year the team would spend $31,200 worth of effort manually releasing. This cost will continue to be incurred every year until the deployment process is automated. The cost of manually deploying also takes precious capacity away from developing new product features. When describing impacts and value in this way, it becomes even easier to rally support from product management as they are selfishly (and rightly so) focused on delivering more product features, faster.

Finally, it is in your best interest to coach software leaders within your team on these prioritisation approaches. This will result in many of the justification and prioritisation conversations being able to happen directly within the team and avoid the need for your role to become involved in every single one of them.

Coupling technical uplift with product features

Sounds simple right? If your organisation is product led, there will be a never ending list of new product initiatives in the backlog. One of your responsibilities of a software leader is to continually look for opportunities and synergies between technical uplift initiatives and new product initiatives. By coupling a new product feature with a relevant technical uplift improvement that resides in the same area of the codebase, it can often reduce effort around testing which creates efficiencies when compared to delivering them both in isolation.

For example, there may be a product feature to implement an additional payment provider to the customer. Within the technical uplift backlog there could be an initiative to improve the level of unit testing that exists within the platforms payment service. There is a clear relationship between these two items of work. In terms of efficiency, there is a benefit to increase the unit testing coverage within the payment service while adding an additional payment provider. Not only do these two initiatives reside in the same area of the codebase, but the work effort in completing them together is also less than the sum of both their work efforts in isolation. This is a result of the effort required to test the platform's payment implementation can be performed just once, not twice. Secondly, the software developers will already be familiar with the payment section of the codebase thus being more efficient, rather than context switching back at a future point in time.

Another example may revolve around the need to support demand in customer growth which requires introducing additional servers into the rotation. The software team also has a technical uplift initiative in the backlog to move all cloud infrastructure into code. Taking a short term tactical approach would see the team simply add an additional server manually into the rotation pool. However, this wouldn’t move the software team any closer to their target state of infrastructure as code. An approach could be to implement the additional server as infrastructure as code, but manually add the server into the rotation. This provides the benefit of meeting the product requirement but also aligning a portion of the work to the software teams target state without re-implementing the entire load balancing capability as code. Of course, if it is possible to also move the load balancer to infrastructure as code, it should be considered.

Identifying synergies between technical uplift and product initiatives has been one of the most successful approaches I have been able to follow to ensure a healthy balance.

Larger technical uplift sometimes requires a project

In some instances, a technical uplift initiative is simply too large in work effort to be coupled into a product initiative. A fundamental goal of software teams is to ensure they release small and release often. This implies avoiding weeks or even months of code not being released into production. When a technical uplift initiative requires considerable work effort to see it through to completion, it needs to be treated as a project. This involves defining clear outcomes, measures of success, documenting high level scope usually as user stories, estimating work effort and a justification aligned to the organisation's process.

Using the technical uplift example of implementing an automated build pipeline, the justification may look like:
  • The current cost to the organisation of not implementing
    • Four teams each undertake one release every week.
    • Each release costs four hours of work effort.
    • As each release is after hours it also incurs three hours Time in Lieu (TIL).
    • This sums up to 1,456 hours a year releasing features.
    • This equates to $72,800.
  • Total cost in work effort to implement is 988 hours
    • This equates to two software engineers for three months.
    • The total work effort costs $49,400.
  • By entirely automating the platforms build pipelines, after the first year there will be a cost saving of $23,400 and then $72,800 for subsequent years.
That is a very persuasive justification as this initiative pays for itself after just 9 months. Taking it one step further, as the teams are now able to release new product features with zero manual effort it enables smaller feature releases more frequently. This reduces production deployment risks while delivering value add features to customers faster than ever before. I would challenge you to find any product role who wouldn’t support this initiative within the organisation.

Ensuring a combined organisational roadmap

In many organisations it can make sense to have a separate technical uplift roadmap and product initiative roadmap. This can support a more efficient planning process within specific teams as these roadmaps are often owned by different leaders. However, at an organisational level when organisational priorities are published, it is imperative to have a single and aligned roadmap that includes both technical uplift and product initiatives prioritised side by side.

Breaking down larger technical uplift initiatives

The larger the work effort is for an initiative, the harder it is going to be to justify and prioritise. Organisational leaders usually struggle to support large initiatives as their value won’t be realised for a considerable time into the future. Prioritising a six month work effort product initiative is challenging enough, let alone a six month technical uplift initiative. As a software leader, you need to be continuously looking for ways to break larger initiatives into smaller work items that can be delivered into production sooner.

For example, take the technical initiative to implement a new organisational wide logging and monitoring platform that has a total work effort of seven months. By breaking it down into smaller work items as seen below, the effort required to justify is greatly decreased, risks are reduced and work items that can run in parallel may become more clear.
  1. One sprint to research and document a solution.
  2. Three sprints to build out the base logging and monitoring platform.
  3. Two sprints to upskill and train all software teams.
  4. Three sprints to integrate logging into the first platform.
  5. One sprint to implement monitoring into the first platform.
  6. ... Repeat steps 4 and 5 for each additional platform.
Always focus on finding the smallest item of work that can be released into production that still adds value to the organisation.

Definition of Done

A Definition of Done (DoD) is essential for every software team to define and proactively adhere to. The purpose of a Definition of Done is to improve the overall quality of the capabilities that the team deploys into production. Quite often a Definition of Done includes specific non-functional requirements (NFRs) that a team values, which may include:
  • Automated testing exists for each feature.
  • Solution and code has been peer reviewed.
  • Documentation has been created or updated.
  • Monitoring has been implemented to ensure observability of all critical paths.
  • Product owner has reviewed and signed off on the new capability.
A Definition of Done does not reduce existing technical debt, however, it plays a part in reducing new technical debt being created which would otherwise need to be paid back in the future.

TL;DR

  • The decision between technical uplift and product initiatives shouldn’t be a binary decision. An organisation needs to find a healthy balance between the two to ensure platform scalability, new product development and retention of team members.
  • Technical uplift initiatives need to be documented, prioritised and made visible to the wider organisation otherwise the team will always struggle to make progress.
  • Continually look for opportunities to couple technical uplift with new product initiatives. This will encourage efficiencies by reducing testing time and reducing context shifting the software team may face by tackling both initiatives separately.


This blog post is taken from chapter 7 of my book "Leading software teams with context, not control" - I hope you enjoyed it. If you did, check out my book that has 22 other chapters on leading software teams on Leanpub as well as other online stores like Amazon in eBook or print format.


0 comments:

Post a Comment