Test and learn

19/3/2022

‘The willingness to experiment, it turns out, is the chief indicator of how innovative a person or company will be.’ (Hal Gregorson)

Test and Learn is an experimental, adaptive technique, used to address complexity, uncertainty and innovation. It’s useful in situations where, say, past experience isn't a reliable guide for future action because e.g. critical conditions have changed. It’s also useful when moving into new, unchartered territory where the evidence needed for sound decision-making can only be generated by, ‘let's suck it and see’. It shares a lot in common with action research: create a tentative hypothesis, step forward, observe the results, try to make sense of them, refine the hypothesis, take the next step.

Test and Learn is used in fast-paced, fluid environments, such as by rapid-onset disaster response teams where conventional strategizing and planning isn't realistic or possible. By the time a detailed plan is formulated, things have moved on - and the paper it's written on is sent for recycling before the ink has dried. Test and Learn is also used by marketing teams when testing new products or services or seeking to penetrate new or not-yet-known markets. It provides tangible evidence based on customer responses which, in turn, enables change or refinement before investing further.

What psychological, relational and cultural conditions enable Test and Learn to work?

Leaders who believe and overtly support the idea that learning is a key to success, both from experiments that go well and those that fail.
Shared opportunities for ideation, hypothesis formulation, testing of assumptions and critical sense-making: e.g. via action learning.
Experiments that are sufficiently specific and observable to draw out key evidence and learning.
Meaningful recognition, celebration and reward for experimentation and learning, and for the progress that results from it.

When have you used Test and Learn? How did you do it? What difference did it make?

(See also: Unpredictable; Adaptive)

8 Comments

Kathrin

20/3/2022 03:53:37 pm

Students and teachers do test and learn on a small scale every day. Nobody can know which answers will be given and which direction the conversation in the class will take. Nobody can know which ideas students spontaneously have and which ideas are spontaneously tried out. I always want to encourage my students to keep thinking or to keep going, even if the result is different than expected.
Test and learn can completely transform a prepared lesson. But the students should have the opportunity to try out their spontaneous ideas, the teacher takes a back seat with his planning and goals and gives the students the space to test and learn.

Nick Wright

20/3/2022 04:43:52 pm

Hi Kathrin and well said. Your approach certainly sounds consistent with the Montessori teaching philosophy. I liked your comment that 'test and learn can completely transform a prepared lesson.' I guess that's the beauty of experimentation - we can hypothesise about outcomes but it's not until we actually try it and see what happens that true learning emerges. That reminds me of the difference between belief and faith. The adventure of faith is acting on what we believe as if it were true...and then discovering all kinds of things that we couldn't possibly have known in advance.

Richard Simpson

21/3/2022 08:30:55 am

Again, referring to my NHS experience and hopefully instructively - the NHS is very complex and very risk averse. The tendency is always to go for the 'tried and tested' option - usually tried and tested somewhere else and lauded as 'the solution' because it worked then and there. I'm sure a lot of informal 'test and learn' goes on, but at the organisational level the default is to come up with the grand plan, often based on a model, to gain Board approval. When things don't fit this Procustean frame, project managers double down to force reality to fit the plan. As someone once said to me regarding the NHS mindset: "It is easier to ask forgiveness than to seek approval." In other words, just do it, Adopting a 'test and learn' approach would be revolutionary (not a bad thing!) but it would also involve people at all levels, embodying learning, and removing some of the dead weight of hierarchy. In the public sector in general, the map assumes greater importance than the territory.

21/3/2022 01:31:11 pm

Hi Richard and thank you for your reflections. I found your insights from experience interesting and useful - as always! Yes, 'tried and tested' is a great way of framing a contrast to 'test and learn'. I recognised your observation that, sometimes, when things don't go to plan, 'project managers double down to force reality to fit the plan'.

Your comment that, 'the map assumes greater importance than the territory' is a very sobering one - and signals, for me, an even greater importance for critical reflection and, in particular, critical reflexivity. I think this is where techniques such as action learning can be so useful - where hypotheses, assumptions, filters sense-making etc. can be challenged constructively from a base of support.

On that theme, this short related piece may resonate? https://www.nick-wright.com/blog/spots

Haven

22/3/2022 03:27:57 pm

I was fascinated reading this blog and interested of diverse discussions. What a creative approach to test and learn. In my country we been to variety of research and adapting learning techniques and styles globally. We pattern our learning strategies approaches and styles to our greatest influence which is America. We are into extreme bureaucratic ways in learning which at the moment the most basic ways of teaching were often put into trash. Our collection of Basic Research, Action Research and many more that students made where just filed in the library where the consumers are the dust mites and termites because education now focuses on online learning. Our most common way of test is paper and pencil test which means objective type of test and theoretical quizzes.

We have also essays where students are able to explain why. In short too many unnecessary ways in education where in fact the most essential part is moulding one’s character which is the essence of life, community progress and reforms. No one among the doctors in education encourages actual ground of learning which the very basic and practical application of a fruitful learning. Most high ranking officials in our government belongs to percentage party. Which means in everything they do, it involves percentage of commissions before a single goal being achieved. So if education will go on this way it means our country will continue to produce rubbish and self centered graduates. Which is a greatest irony and failure behind those mission and vision of every school institutions.

I pray the world will innovate on learning techniques into behavioral approach than innovating complexity that provokes danger and endless corruption.

22/3/2022 03:31:13 pm

Hi Haven and thank you for posting such a thoughtful response. I think you pose some very serious and important questions vis a vis the fundamental goals and nature of education in society and the world. We often conceive of education in its narrowest terms as academic input and achievement.

Interestingly, Reg Regans (the founding 'father' of Action Learning) regarded the most valuable method of education as enabling people to grasp, struggle, wrestle-with and find or create solutions to the real-life issues they were facing and dealing with. I like his philosophy: 'Swap your difficulties, not your cleverness.'

It would be interesting to see Action Learning as a methodology applied to incorporate and address wider global, ethical issues in order to achieve spiritual, personal and social transformation. Lots to think about..!

Jonathan Bradshaw

22/3/2022 08:12:36 pm

Hi Nick and thank you for posting this article. I was interested to see the link that you made between Test and Learn and action learning.

In my experience, Test and Learn shares similarities with evidence-based practice (EBT) in nursing. It draws on the scientific deductive method which can be valuable in closed-system work where actions and consequences are linear and predictable. For example, imagine we want to create condition Z, and we believe that X + Y will achieve it. Now imagine that we know what Z is, and what X is...but we don't know what Y is. We can use Test and Learn to experiment with options. Once Z result appears, we know what Y is.

One of the criticisms or limitations of this approach is that when applied to dynamically-complex systems: e.g. human families, teams or organisations, there are too many confounding variables to make accurate predictions for the future. Any number of variables could account for what we perceive as Y and Z. It's not mechanistic. In these types of situations, qualities such as intuition and shared sense-making often prove more fruitful than empirical evidence drawn from other experiments.

I think this is where, as you say, critically-reflective methods such as action learning could help.

22/3/2022 08:27:36 pm

Thank you, Jonathan. I think that's an interesting observation concerning some of the similarities between Test and Learn and EBP. You reminded me of some supervision work I did with a PhD student involved in family systems work who, similarly critiqued EBP for reasons along the same sort of lines that you outlined so well here.

I agree that this is where action learning can add real value as a critically-reflective and critically-reflexive process. At its best, it enables participants to test hypotheses, assumptions, sense-making, conclusions drawn etc. (I guess a bit like peer review in the context of a scientific community) - drawing on questions from a diverse peer group.

I also find critiques of EBT from a social constructionist perspective particularly stimulating and useful. I love Kenneth Gergen's challenge in this short video (starting at 3 mins 05 secs), where he starts with an object that we would ordinarily recognise as a bottle of water: https://www.youtube.com/watch?v=-AsKFFX9Ib0&t=226s

Test and learn

Leave a Reply.

Nick Wright

Archives

Categories

Test and learn

Leave a Reply.

​Nick Wright

​Archives

Categories

Nick Wright

Archives