Finding a role for Enterprise Architecture in a world of agile and digital transformation

I gave a presentation at the recent British Computer Society (BCS) Enterprise Architecture Conference. A replay of the talk is available and this post is an outline of the main points. Over the last few years I have worked with many architects and quite a few are unhappy. Executives might appreciate the efforts that go into models and designs but things move on so quickly that, often, the architect’s work does not keep pace with events or arrives too late to be useful. The architect might sadly slope away, loaded down with copies of densely packed designs which are now only good for the recycling bin, or they might become angry at being ignored. Either way, it is not a pleasant situation.

I shared with the conference some ideas about why this is happening and offered some suggestions of how to respond.

Relentless change and uncertainty

The source of this unhappiness is change and uncertainty. The surprising results in recent elections (particularly in the UK and US) provide lots of engaging and entertaining material for talks about uncertainty but they are probably too removed from most architects’ work to provide any lessons. More practical examples are the forward plans or target architectures that consume a great deal of architecture effort. Let’s imagine we are back in 2012 and considering the target architecture for the future - looking ahead 5 years is typical. In 2012 this seemed like a perfectly sensible thing to do but, with the benefit of hindsight, we can see how easy it is to get things wrong.

First of all consider some of the things which did not exist in 2012. This includes new technical components (such as Docker’s first release in 2013) but also new ways of working (four year old slack and three year old trello are the default tools for many of my teams), new business models (Monzo was founded in 2015), new competion (Airbnb expanding into Europe) and other threats.

But it gets worse. What about all of those things which existed in 2012 but we ignored, either because they had not broken into the mainstream or were considered too imature. It may have been ok to put off serverless technology, like node.js, in your plans but could you really afford to exclude Android, which is now dominating that part of industy? There are also the things we assumed would be important and stable which have become obsolete. There were several people in the BCS audience who, like me, had been caught by surprise at how quickly people abandoned their, once cherrished, Blackberry devices. Companies and whole segments can shrink extraordinarily quickly if they run into problems.

Blistering pace

To add to the uncertainty, things do not always develop or progress at a steady pace. Enterprise architects are familiar with “Moores Law” but many other products and services exhibit this kind of growth. The value for money of personal DNA sequencing has improved by a factor of 10 over the last 10 years. The improvement for solar power and mobile data is 20 and 25 respectively. How confident can we be that the trade-offs we are making will still be sensible in 2 or 3 years, let alone 5 years and beyond? It shoud be no surprise that exectives are finding this sort of long range planning less and less useful.

If some other disciplines had found ways to cope with all this change and uncertainty perhaps we could learn from them? Product development teams (especially for software products) can provide useful lessons. When I started my career, project timescales were typically measured in years. Today, teams are using short, fast cycles to work at a much faster pace. Even in the public sector, which has a poor reputation for speed and agility, you can find teams who can go from discussing an idea to launching a new public service in a matter of weeks. The giants of the internet era move even faster and there are opportunities to use machine learning and artificial intelligence to go beyond the pace and scale of the fastest human teams.

What these teams have learnt is that:

There have obviously been some mistakes and setbacks along the way but many organisations find they have a better chance of getting better results using the kind of agile techniques employed by the best digital teams. An agile team does not have to be flawless, it just has to do better than the alternatives. Traditional contributions from architects are not so much better that they are worth the wait. Even when dealing with sensitive topics like security, safety and compliance agile techniques often get to better answers more quickly.

However, that is not the end of the story. There are some things which agile and digital teams struggle with where architects and their tools might be able to help. Here are four to consider.

Breaks in the flow

Sometimes teams come across breaks in the flow of iterations. Initially things start well and each iteration gets better until something breaks down. A common example arises in efforts to move existing processes online. Initially a paper form might be copied, more-or-less, into an electronic form and then iterated to make it more useful and accessible. Occasionally, several iterations in, the team discover that there is a better way to arrange the service which does not require a form at all and it is dropped. Perhaps architects can be looking out for these breaks and helping agile teams handle them.

Troublesome users

Fast iterations with users wins but getting to your users is not always easy. Competitors, campaigners and criminals may not want your service to be a success, at least not in the way you intend, but they still need to be understood. Users with different roles, for example, customers, service agents and investors may have apparently conflicting needs which must be unravelled or traded. This might be an area where architecture tools can help.

Sharing and re-use

We are still learning how to develop common standards and share components in a user-centred, agile way. Good teams will investigate what they can share or re-use but deliberate coordination is hard and teams can end up creating duplicate components which are highly tuned to their users. Even when teams identify components which are good candidates for sharing and re-use it can re-ignite the challenges of finding the right users and stakeholders for them. The layered approach that architects often use may provide some new ways to deal with this challenge.

Are you sure you want to delete?

The final challenge I have picked out for this post is that products and services are not just built from software. There are lots of important aspects which cannot be deleted and recreated at will. Some are robust things such as facilities, fixed assets and skills. Others are extremely fragile, such as trust, relationships and collaboration. Architects are used to applying the kind of whole-systems thinking which may help agile teams handle these aspects.

Rising to the challenge

It makes sense to consider where agile and digital techniques are falling short but do not be fooled.

There is no point in using these challenges to argue against techniques which are proving so effective in practice. Teams do not adopt these techniques because they are flawless. They adopt them because they often produce better results than the traditional alternatives. It may take a few more iterations but agile teams that are focused on their users will absorb setbacks, work around them and learn to do better in future. As architects, we need to be thinking about how we can support these teams and enrich their agle approaches so that they can work even better.

Over the last few years I have had the chance to develop architecture within fast moving digital teams. Here are some of the tactics that I have found are helpful, so far.

Don’t be too ambitious

No achitect or even a team of architects can cover everything, at least not at the pace that agile teams need. We need to learn to focus and work with our stakeholders to deliberately chose where to invest our efforts. There will be little need for long, jargon-filled documents or comprehensive, detailed models. In most cases these sorts of products will not be required as they won’t be timely, robust or accessible enough to be of value. It will be necessary and useful to reach up into abstract concepts and down into details on occasions but this will need to be justified and, most importantly, supported by the teams and executives who will be expected to use the results.

Work on the problem

Achitects have traditionally spent a lot of their time looking at the components of the product and service and how they work together. This is the solution side of the architecture equation. This is the area where agile teams iterate at a fast rate and it will be very difficult for any architect to stay far enough ahead of the team to be of much use. Instead, architects could put more energy into understanding the problem side of the equation - who are the main actors, how do they interact and what results from this? Although this may be unfamiliar teritory, architecture tools and approaches are just as valid here. I find that the architecture of problems changes at a much slower pace and the insights that the architect uncovers are usually more relevant and useful to agile teams.

Whole services and business capabilities

Business capabilities provide a way of thinking about products and services that blends technology with the people involved and how they work. I am finding that a focus on capabilities encourages architects to explore things that have enduring use for agile teams such as spotting and exploiting re-use and blending slower moving changes to facilities, organisations and culture with the fast iteration of IT.

Iterate fast with your users

If fast iteration with users wins in so many areas would it also win in architecture? Architecture only comes to life when stakeholders have absorbed it and use it in their decision making. Why not treat decision makers as users and develop architecture in fast iterations with them? This is the approach I have been taking with my customers over the last few years.

It can feel uncomfortable working out loud and engaging with executives before anyone has robust results. But it is even more uncomfortable to consume a lot of time and effort on things which are not understood, are not relevent or get in the way of teams delivering value to their users and employers.

In case you missed the message

No one has yet worked out repeatable and portable method for doing architecture this way but that is ok. Let’s start, iterate and get better because:

BCS Talk Replay

About 20 minutes