Manage Technical Debt by Codifying policies in the SDLC

What is Technical Debt? 

The term technical debt is an abstract construct used to describe the potential long-term impact on development programs by decisions made at the program’s start.

For example, a fast-paced development that focuses on a minimum time to market by taking shortcuts through the development lifecycle will probably require significant long-term maintenance effort to resolve issues created by the rapid development.

This long-term effort is the technical debt that developers must repay if the end product is to become a polished, fully functional item. Thus, a fast and dirty development will incur a high technical debt.

By comparison, a slow-paced development that identifies and resolves all issues during the development life cycle without deferring the implementation of features will deliver a polished, fully functional item with low technical debt. That is, maintenance efforts to resolve issues will be minor.

Therefore, there is a tradeoff between the pace and thoroughness of the development lifecycle with the incurred technical debt. Furthermore, each program will have its unique aims, objectives, challenges, and dependencies that influence the tradeoff. Therefore, there is no optimum technical debt; it depends on the nature of the program.

The critical point is that technical debt is always an inherent part of doing business. The key is to identify and manage the debt.

The typical factors that influence the technical debt are:

• Business strategies and objectives
• Program complexity and dependencies
• Program timescales and deadlines
• Financial constraints
• Design decisions and constraints
• Alignment between strategic and tactical decision-making
• Technology decisions and constraints
• Resource ability and availability

The short-term program factors determine the magnitude of the technical debt. The long-term success of the program depends on the management of this debt.

How Does Technical Debt Accumulate?

Decisions made in a development program can lead to the creation of technical debt.

• This can be a deliberate and prudent act where time or budgetary pressure results in decisions to defer implementing functions. This postponement will need addressing at a future time. Hence a debt has been created, but the means to repay the debt is known. Resolving the debt can be planned and budgeted as the debt is quantified.
• This can conversely be a deliberate and reckless act where shortcuts are taken, such as deploying untested applications or selecting an architecture that will work with the current deployment but not support future innovation. Here a debt has been created, but the size of the debt and the means to repay it are unknown. It may not even be possible to repay the debt.
• Debt can also be inadvertently created when developers make poor decisions due to a lack of knowledge or experience. For example, attempting to make a function work by bodging a solution that may appear to work will recklessly create an unquantifiable debt.
• However, if the deficiencies are recognized, developers can prudently produce an interim solution with known limitations. This approach generates a quantifiable technical debt that developers can repay once the required knowledge or resources become available.

Deliberate and prudent accumulation of technical debt is a necessary tradeoff in rapid development programs to meet timescales by deferring part of the development program to a later phase. This action is planned and controlled, with a definable solution to repay the debt. Sometimes decisions need to be made that create technical debt that the business will never repay. Where the cost of repaying a debt outweighs the benefits, a risk-based business decision may conclude that the debt is acceptable. Not all technical debt is bad.

Figure 1: Sources of Technical Debt

The Impact of Technical Debt

The increasing popularity of migrating infrastructure and services to a cloud environment brought technical debt to prominence. As a result of the benefits of financial savings and technical flexibility, there was a natural attraction to cloud migration for businesses looking to reduce costs and improve processes.

The issue was that the migration process was not always straightforward. Some organizations found that they could not migrate existing services and procedures, that they needed to throw them all away and start from scratch. And the reason was technical debt. Businesses found that short-term decisions taken in the past prevented them from having a quick and simple migration process.

Once the concept of technical debt was out in the open, organizations then found that it existed everywhere, from software applications to consumer products, network infrastructure to end-user devices. Thus, identifying and quantifying technical debt provided the stimulus for businesses to manage it.

Running outdated servers that create processing constrictions or operating systems beyond their end-of-life are classic examples of the accumulation of technical debt.

Cloud migration not only helped many organizations resolve their technical debt issues, but it also identified that they had unknowingly accumulated this debt in the first place.

The Cost of Technical Debt

Deliberate prudent technical debt can be costed based on the time and resources required to implement any deferred functionality. Inadvertent and reckless technical debt is a much more complex problem to solve. Resolving issues caused by substandard infrastructure, poor implementation decisions, or design and development errors can be challenging to quantify. The unquantified issues may be simple to fix, or more often, they can place a substantial financial burden on the business to resolve. Replacing the entire infrastructure or refactoring the whole codebase may prove economically unviable, bringing the program to a premature end.

Typically, the cost of repaying any technical debt comes down to the following factors needed to remove the debt:

• The workforce resources required to implement changes;
• The cost of any changes to infrastructure or the wider environment to resolve issues;
• The impact on program timelines due to additional work;
• The effect on revenue streams due to delays and end-user takeup;
• The interrelated influence on reputation, customer confidence, and sales;
• The implications for financial investment requirements to fund the change program;
• The incurrence of financial penalties from non-compliances or incidents;
• The costs of increased maintenance and support activities;
• The increased cost of implementing future innovations.
• Manual processes are significantly slower in comparison with automated solutions, often presenting a bottleneck to development processes.
• Manual processes are subject to human error, and error rates for manual actions are significant in magnitude and influenced by factors such as time pressure.

Reducing Technical Debt

Deliberate and prudent technical debt is by definition managed and controlled. The main problem is the intentional and reckless creation of inadvertent debt that inhibits innovation and the adoption of new technologies. From a business point of view, these issues impact growth and competitiveness. A business that cannot adapt to changing market trends will quickly lose market share.

Many factors influence technical debt; the key to keeping this debt to manageable levels is primarily down to following good development practices, such as:

• Robust architecture that is flexible, adaptable, and scalable to support the product over its lifecycle. An architecture that does not meet these criteria will prevent product evolution, stifling its working life and constraining future development.
• High-quality development artifacts that accurately and efficiently implement the required functionality while minimizing unwanted and unexpected behavior. Suitable quality components will be easier to maintain and contain fewer errors that will need fixing at a future date. High-quality artifacts will also take less effort to verify and validate due to deduced rework from test failures.
• Thorough quality assurance practices capture issues in practices, processes, and products as early as possible in the development lifecycle. Identifying and correcting problems inherent in the development process will prevent systemic errors that impact the entire development artifacts and create a maintainability challenge.
• Periodic refactoring processes applied to developed products can prolong product life and minimize the accumulation of technical debt for deployed products that inhibit evolution and enhancement.

All these good practices come down to the creation of policies and standards that impose good development practices. However, implementing and enforcing policies and standards is a challenge, mainly where resource restrictions and pressure to achieve program milestones are brought to bear.

Manage Technical Debt by Codifying Policies and Standards

The traditional enforcement of policies and standards used manual processes built around review and audit procedures. While nominally effective, this approach has two significant drawbacks.

In a rapid development cycle, manual processes add delays. As a result, management applies pressure to speed up the processes. Unfortunately, this pressure introduces more errors. The extra errors contribute to increasing the inadvertent technical debt.

The solution to maintaining high quality in a rapid development cycle with the minimum of technical debt is to look at ways to reduce the overhead of managing policies and standards and automate their enforcement. The answer is codifying these policies and standards.

The principle of policies and standards as code is simply writing the rules that the policies and standards defined in a high-level language that automated processes can interpret. Enforcing these rules can be automated alongside development processes to eliminate bottlenecks and human error.

An additional benefit is that these rules can now be subject to the same best practices as the development artifacts that they apply to, improving their quality. Configuration control, automated testing, and automated deployment can all manage the policies and standards.

By codifying policies and standards, developers can undertake automated checks for inconsistencies and contradictions. This capability is critical when changes to regulatory requirements or market-led legislation require integrating additional standards into the overall ruleset for the development activities. You can learn more about using continuous compliance checks to enforce cloud security and best practices at the build and deployment time here.

A final benefit is that by representing policy and standard-based rules as code, the information and logic about the rules have a direct representation in code that developers can augment with comments. In addition, each rule’s purpose will have an unambiguous definition in a form that developers can comprehend. Studies show that developers are more likely to conform to a rule if they understand why it is applied.

If you’re a Kubernetes user, take a look at our Kubernetes Governance webinar replay - Managing your Policies & Standards.

Conclusion

The development and operation lifecycle has changed significantly with highly automated DevOps processes. Unfortunately, this advancement has led to a culture that promotes creating minimum viable products quickly, followed by product updates to add functionality and resolve issues. As a result, this approach encourages the creation of vast levels of technical debt that will need to be repaid further down the line.

The attraction of this approach is that it achieves the initial development quickly, which visibly meets a business target. However, the long-term debt repayment is less visible, buried in the maintenance activities. Therefore, where meeting the time to market goal is more important than creating a high-quality, fully functional, and error-free product, large technical debt will always be the outcome.

The challenge for developers is to reduce this debt as much as possible by building high quality into the development processes without compromising timelines. Codifying policies and standards are the solution to this problem, automating quality control and reducing errors in deliverables to minimize inadvertent technical debt. This will leave the known, managed, and controlled deliberate technical debt to be resolved as part of future developments.

Magalix empowers organizations to integrate policy-as-code and standards-as-code into DevOps processes and culture. This service allows businesses to apply governance to manage technical debt, enforce quality standards and reduce risks across infrastructure and workflows. Find out how to integrate your governance and compliance scenarios with the Magalix Platform Overview.

To learn more about how we can support your development processes, please get in touch.