A journey into Discovery
Throughout 2023 I was seconded to work in our Digital Strategy team as Discovery Lead. This post is a summary of some of the key lessons and insights the team gathered about Discovery.
An obvious reference point for our work is the UK Government Service Manual and the closely related Local Government Digital Declaration. These share some common ancestry with the approaches familiar in the commercial world but with some adaptations we need for public services. A big adaptation is public services can’t relying on profit and competition to drive performance. Money and satisfying customers still play a massive role in developing public services but, at the very least, the language needs to be different.
Some local authorities have closely aligned their work to the Service Manual and talk about Discovery as it is defined there. The start point for us was more complicated. Our IS team and Strategic Projects team both had defined project lifecycles that included a discovery phase. These phases differed from each other and the Service Manual. We also had an innovation lifecycle with another definition which had fallen out of common use. On top of that we have partners and suppliers helping us in various services who used their preferred approaches which could also include a stage called discovery.
We started with some working assumptions that we could validate and test. Our key assumptions were:
- There were some common drivers and needs behind the various flavours of discovery
- We could better meet those needs by changing the way we carried out discovery work
- We could establish a new team to build our new discovery approach and provide it as a service to the council.
We looked at the Service Manual, good practice in other organisations and some of our existing work that carried some kind of “discovery” label to make sense of how this all fitted together. In the spirit of working out loud we shared what we learned and sought feedback as we went along.
Based upon the Service Manual, we started to share a framework for discovery work that could accommodate the different types. For example, we could group work around some common needs and start to be clearer about who the discovery work was for and what the discovery team were seeking to learn from it.
 |  |  |  |  |  | |
| Discovery Scope | Investment | Technology | Management | Staff Needs | Resident Needs | Community Needs |
|---|---|---|---|---|---|---|
| Learning | How do we achieve good value? | How should we use our technology products and services? | How should we change our processes and organisation? | What do our people need to do their best work? | what do residents need to achieve their individual goals? | What to places need to reach our collective goals? |
| Examples | Business justification | IS Assessment, IS Discovery | Service Review, continuous improvement | Business Analysis | User research, service design | Strategy, consultation |
| Elements | Savings, funding, budgets, contracts | Platforms, components, applications, cyber security | Roles, procedures, data flows, team structures, operating models | Workflow, internal systems, performance measures | Journey maps, personas, user experience | Partnerships, service architecture |
We also looked at when discovery work was happening - and that wasn’t confined to a named discovery phase or stage. Discovery happened throughout the service development lifecycle, including for existing services that had been in place for a long time. The diagram below was our attempt to illustrate this.
The diagram is based upon the Service Manual stages but using terms that are more familiar in our organisation. Layered over these stages are an illustration of the typical balance of work.
- Lots of discovery work is about learning. This happens throughout the lifecycle but is front loaded. In the first phase it is the dominant sort of work.
- In later phases the focus shifts to making choices (such as which problems to prioritise, how much effort and money to commit, which options to pursue), making things (such as systems and the changes needed to take advantage of them) and finally using them and getting the benefits.
Our existing approaches and the ones we borrowed when working with partners and suppliers could be viewed in this context. They might only cover part of the life-cycle and might use different terminology but they were fundamentally variations on this common pattern.
This analysis helped us understand what a new approach to discovery needed to achieve:
- Discovery should help us make better decision e.g. learn about the root-cause problems before deciding how to fix them.
- Discovery should also help us accelerate the release of value e.g. take quick wins and make incremental improvements where that was appropriate.
We got to this point quite quickly and then our journey got more difficult. Some of the team felt we should use what we had learnt to design our new approach and launch it. Others felt we should run trials with some projects and services, learn some more and let the best approach emerge. We ended up with a mix of both. Perhaps that was the best approach but we didn’t manage how the decision was made very well.
The trials turned out to be frustrating but also provided valuable lessons. The frustrating part was that almost all of the trials got stuck, postponed or cancelled. The valuable lessons came from analysing what had gone wrong and why. We’d fallen into the trap of thinking we could make a sort of machine that would reliably churn out solutions in response to service problems. The reality was much messier. The services faced challenges that had a life of their own. Our challenges wouldn’t wait patiently to be addressed and they wouldn’t sit passively in the categories we had created for them. Here is a cartoon I drew in an attempt to lighten the mood when we shared these lessons as, to our team, they could feel like a failure. A machine was never going to be a useful way to think about this work. An aquarium was probably more helpful!
I took three big lessons away from this:
- Responsiveness and flexibility are more important than standardisation and consistency. Our challenges wouldn’t fit into neat catergories. We might come across a crisis situation where a supplier has let us down and our residents could be impacted very quickly. We might be monitoring some new social trend or government policy proposals so we can be ready when it hits our service demand in the future. We might be working in the infinite range of alternatives in the middle. The teams in these situations need to learn very different things over vastly different timescales and are exposed to very different risks. These are not favourable conditions for a centralised, standardised discovery service.
- Good discovery that meets our needs is an inherent part of running a service well. Services can use specialist help but they can’t hand over the job and step aside. It can be tempting to try a short cut and commission another team (internally or a supplier) to carry out some discrete discovery work. This can result in very professional outputs but those are rarely what we need. The driving reasons to do discovery work are to make smarter decisions and to shape subsequent delivery to optimise the release of value. The discovery insight needs to end up in the hearts and heads of the decision makers and delivery teams. A report at the end of a discovery stage, even a highly polished report, is a pretty poor way to achieve this.
- The final lesson is that, confusingly, the discovery stage isn’t the first step in discovery. The first step, or probably several steps, is building the team, providing the right conditions and facing up to the constraints. These are steps of negotiation and making trade-offs. In almost every case we won’t be starting with all the ideal ingredients for the sort of discovery work we might see in text books or promotional brochures. Some of those ingredients might be in reach and it might be worth fighting to get them. Often, we need to accept the limitations and get on as best we can, acknowledging that the results will be compromised but still valuable. Occasionally, we will find discovery work will have very limited value right now and should accept this and move on.
I am still working out how to apply these lessons to our Discovery work. We are experimenting with doing just enough discovery when needed rather than forcing this into a one-size-fits-all process. As illustrated in the cartoon below, I like to think about this as a test tube which we fill up just enough to move on to the next stage. If we find we don’t have enough information, certainty or new ideas we can top up the test tube with the right sort of discovery work. That could be by using one of our established processes or doing something we are not used to such as building a prototype, performing an experiment or running a limited trial.
We are also experimenting with using principles and guidelines to manage the work rather than set processes. We are not sure what form this will take yet but are trying out some alternatives. An example is illustrated below which aims to preserves the idea of doing just enough discovery but wrapping this in a minimal amount of control.
 |  |  |  |
| Check the forecast | Pack rucksack | Get into nature | Head for the hills |
|---|---|---|---|
| 2 hrs effort | 2 days effort | 2 weeks effort | 2 months effort |
|
|
|
|
| Do we have broad agreement that we have something valuable to work on together? | Have we identified the right people and give them authority to do this sort of work? | Are we confident we have uncovered the top problems to solve in this area? | Have we done enough tests or collected enough evidence to be confident in the solution and delivery approach? |
Another way to look at this is extending the service life cycle further upstream. As illustrated in the diagram below, the stages that we currently treat as the start of the lifecycle are preceded by one or two more where the emphasis is on building alignment amongst the various stakeholders and building the team and the conditions for whatever sort of discovery is needed.
We are also considering where discovery work should sit in our operating model. We probably need to take a blended approach but we can take inspiration from other services within the Council. For instance, financial management in the Council is distributed throughout the organisation. There are some central finance teams setting policy and operating basic financial systems but there is a scheme of delegation and a framework of training, guidance and support that allows managers to make local spending decisions safely. Something similar works well for managing people and procurement. The diagram below shows one way this might be configured for discovery.
We are not at the end of our journey into discovery but some of the next turns in the road are visible. We’ve got some clues about what the shape and size of discovery could be in the future and some more working assumptions we can put to the test.