Brief at the Start
Go straight to the full length version
If we are designing technologies to be more like people, we need to approach our understanding and management of risk on the same basis
We like to think our tools are inanimate objects. But this has not been the case with digital technologies for some time, and divergence will accelerate rapidly. Management of risk is founded on trust, and if we don’t understand how an entity will behave then we need to rethink our basis of trust.
Human’s have a tendency to trust the output of technology more than they trust something that another person tells them. We need to recognise and combat this if we are to avoid the risks and hazards of the future.
By considering our technologies more like humans, with all of the inherent diversity of behaviour and unpredictability, we can understand and prepare for risk more effectively. We can then apply some of the practices and controls that society has developed over the centuries.
The challenges we face today with digital risk are significant, but they will grow exponentially with the scale and complexity of the technologies we employ. In a future where no aspect of business will not be founded on or heavily influence by these tools, we cannot thrive with an attitude of “I don’t do technology”.
Get the Full Version
“Why can’t a computer be more like a chair?”
AI is a constant feature in the news these days and a couple of things this week have struck me as worthy of more thought. Meta and OpenAI both announced that soon they will be releasing models that ‘think’ more like people and are able to consider consequences of their decisions1. Also, the FT published an article that the speed of AI development is outstripping the development of methods to assess risk2. It strikes me that the root of many of the difficulties we face may relate to a bizarre feature of Human Nature.
Information-, digital- or cyber-risk is a challenging concept, and it is likely to get increasingly challenging with AI. What principles could we put in place to make it easier. Do we need to change the way we perceive technology and reset our relationship with it? Like with George Bernard Shaw’s rhetorical question “Why can’t a woman be more like a man”, do we need to stop trying to treat technology like something that it is not.
I believe that the heart of the issue is one of Trust. If we have a reliable basis of trust in a person or in a piece of technology then we can understand the level of risk we are taking more clearly. If we don’t have a sound basis of trust then this risk becomes steeped in uncertainty. But trust is a perception, and our perception of technological reliability seems misplaced.
Why do we trust technology so much?
Why is it that a human is more likely to implicitly trust what a computer tells them than what another human tells them?
Before you hit the CAPSLOCK button and start loudly disagreeing, hear me out. How would you react if a gentleman claiming to be a prince from an African state turned up at your door offering untold riches without any conditions? And yet many people over the years have been taken in by exactly that offer received by email. Phishing, fake news over social media, and numerous of other socially engineered deceptions rely on this condition. It has also been the subject of plenty of research3 4.
While I was responsible for information systems, information management and information exploitation in an Army Headquarters I was struck by how people assumed the accuracy of a unit’s location on a screen was totally reliable. They would take a similar marker on a physical map with a healthy pinch of salt; recognising that there was uncertainty in the accuracy of the reported location, and that the unit in question may have moved significantly since they made the report. But they would be happy to zoom in to the greatest detail and ask why A Squadron or B Company was on the east side of the track rather than the west, or why they still have not moved in line with orders 30 minutes previously.
Is tech more like a hammer or a human?
Now I’m not a psychologist, but I have a theory. We know that humans are fallible. For some 300,000 years or so we have evolved our understanding of human trust. We know that people make mistakes. We know that sometimes they deliberately do things for their own benefit, even if it harms others. People can be unpredictable; sometimes by their nature, and frequently because of external factors such as sensory overload, tiredness, or anxiety. But we consider machines to be predictable and deterministic. No matter how many times we enter two large numbers into a calculator, we expect them to be added up correctly and consistently.
So, when we look at a computer, at least subconsciously we consider it to be more like a hammer than a human. A predictable tool, that will produce the result it was designed for.
But even in the case of conventional, non-AI technology, this is a fallacy. Computers are designed and programmed by fallible humans. We all make mistakes. Those mistakes are transferred to the code we write, and in turn into the results that this code produces. The more complex the code the less certainty there will be of accurate and consistent results. We are also dependent on our perspective being similar to that of the person who designed a system. If ambiguous problems are interpreted differently by the designer, then the chances that we will misinterpret the results will increase.
We like to consider our tools to be as predictable as a hammer, but too frequently they operate more like the humans who created them.
Its probably a statement of the blindingly obvious, but this situation is only likely to get more extreme with the evolution of AI. We are actively designing technologies to operate more like humans. We want them to learn, and draw insight from that learning in new situations. The question we ask a system today may well produce a different answer if asked again in the future, because the information and ‘experiences’ that the answer is based on will change. Like if I ask someone a question when they are 10, 18, 30 and 60 years old I should expect to get very different answers, particularly if the question is based on philosophy or beliefs.
How does this affect the risks of employing increasingly advanced technologies?
If our technologies are becoming more like humans than hammers, then how does this affect the risks? Well, the diversity and unpredictability of people is something that we are familiar with and have been managing for some time. Let’s look at the similarities, because, after all, in many situations we are looking to use technology to do tasks currently done by people.
We know that people misunderstand a task, because language is ambiguous, and interpretation can be based on an individual’s perspective. We all have different value systems, which influences where we focus our efforts, where we might cut corners and what we might avoid altogether. At an extreme these different values may lead to behaviour that is negligent or even malicious. People can be subverted or coerced into do things against their will. All these behaviours have parallels with complex tech, and AI in particular.
Ambiguity will always create uncertainty and risk. AI models are based on value systems that are intended to steer them towards the most desired outcome; but those value systems may be imperfect, particularly when defined in the past for unforeseen situations in the future. And we all know that technology can be compromised to produce undesirable outcomes.
But I think there are some fundamental differences as well, particularly with regards to intrinsic controls, feedback, and extremes. People-based organisations tend to have inherent dampers that reduce extremes (though populist politics that leverages the amygdala might provide evidence against this at times!).
If we recruit one person to do a task they might be a ‘good egg’ or a bad one. But if we recruit a team of ten the chances are that the different perspectives will challenge extreme behaviours. The normal distribution will reduce the impact of the extremes. Greater diversity will increase this effect. Clearly this does not eliminate risk, and a very strong character might be able to influence the team, but it does introduce some resistance.
Diversity will be minimised though in a ‘team’ based on same AI model, feeding from the same knowledge base, using the same value systems, and learning directly from each other. Diversity can dilute risk, but commonality can reinforce it like an echo chamber. We have seen this already with runaway trading algorithms that are tipped out of control by the positive feedback of their value systems.
Are Digital Risk and Business Risk becoming the same?
Let’s for a moment follow the trajectory that we are taking with technology into the AI age. We will use digital intelligent tools wherever we can to do tasks currently done by humans. That’s not new, and in my opinion unavoidable. So, progressively every aspect of business will be decided or influenced by digital systems that use digital entities to operate on digital objects, and produce outcomes that will be digital in nature before they transition into the physical world.
From this description it strikes me that there will not be many risks that don’t have a very significant digital component. I’d argue that we will be increasingly unable to separate our management of cyber-, information- or digital-risk (whichever term you want to use) from most business risks. In the future, if not already, the current construct of a CISO function managing information risk separate from financial, competitive, health and safety and other business risks would appear to be rather quaint. Instead, I’m not sure any area of business management will be able to claim that they ‘don’t do technology’ and it will be more important than ever for technology risk to be managed with an intimate and universal understanding of the business.
Can we learn from how we manage human risk?
In this version of the future, it strikes me that we should look at the way we have historically managed the parallel human risks. We know that we cannot implicitly trust people. We look for evidence of a baseline of trust, and then over time allow the individual to earn more trust. Some people are more untrusting at the outset and rely more on the trust that is earned. Others are more generous at the outset but adjust that position if disappointed.
When hiring someone we will look for credentials of trust. Exam results, membership of respected bodies, evidence in criminal records, or references and recommendations from people we trust. But we don’t take that at face value. We also extend our due diligence through interviews. We allow for the uncertainty in this process by having a probationary period at the start of employment during which powers may be restricted, supervision will be greater, and tolerance of mistakes will be lower. The level of due diligence and protections that we apply tends to be proportionate to impact that a bad decision might have. We will spend far longer choosing someone who will have decision making powers over a business’s finances or administrative powers over its IT than we will in other less risky roles.
We also apply additional controls to reduce risk further in operations. We apply segregation of duties so that even a trusted person is limited in their unilateral powers. We apply supervision and oversight where the risk is severe, such as on trading floors to meet regulatory requirements.
In fact, most businesses will assess the risk and reward before they even consider standing up a new department or expanding and existing one. The business case will need to justify why it should be done, what the upside is, and what the potential downside might be. Even if it isn’t formalised we will have a feel for the level of confidence or trust we are looking for in a particular role or appointment.
Applying human risk management to technology risk management
I believe we can improve our understanding of risk by considering technology components more like people, at least at a conceptual level. I think we should be doing this to a degree already, but as AI solutions come onto the market over the years and decades to come this convergence is only going to accelerate. An AI model’s decisions are based on an unpredictable array of inputs which will change with time. They are based on a set of values that need to be maintained in line with the business values. But most importantly they will learn. Learn from their own experiences and learn from each other. This sounds to me far more like a human actor than a hammer.
- Initiation. When embarking on an initiative we need to take time to consider the inherent risks we face. Not just the discrete risks within the initiative but also the more systemic risks we need to avoid. If we are replacing a human function, then how much do we rely on the trustworthiness of the people currently doing the job.
- Selection (Recruitment). We need to decide what we mean by trust when selecting the types of technologies we might employ, and where we will apply technology versus where we want to retain a human in the loop. What frame of reference will we use to define and measure trust. What external evidence can we take and how much do we need to reinforce this will our own due diligence. For instance, government regulation and certification of AI models may provide us with a baseline of trust, but in the more sensitive and risky areas of business we will probably need to apply our own ‘interviews’ and tests.
- Design. The more risk we can design out early, the easier (and cheaper) it will be to manage the residual risk in operations. The concept of Secure by Design is important now but will become essential as we progress5. In particular, we need to apply the equivalent of segregation of duties until we understand more about how these systems will operate, learn, develop and interact over time. Applying segmentation is too often ignored today leaving us with broad, flat, and vulnerable networks; but it will be vital to contain risk in the future.
- Operations. In operations, just like with people, we need to hope for the best but prepare for the worst. This is not just about monitoring an environment. It is also about maintaining our understanding of risk and war gaming new scenarios that come to mind. The military planning process has a Question 4 which asks “Has the situation changed”. We need to industrialise this in the way we manage, maintain, and evolve our systems. The most obvious ‘big issue’ that comes to my mind is the point when operationalised Quantum Computing comes to the fore; but there will be many smaller ones as well and we need to adapt in time, if we are to overcome the consequential disruption.
Like most abstract models, this one is not perfect. I’m certainly not advocating a choice of one approach or the other. Technology and people are similar and different. We need to strive to make our technologies more deterministic in our engineering and design.
Summary
I’m optimistic by nature. So, I’m not an AI denier, nor am I an AI abolitionist. But to use the hackneyed phrase, “With great power comes great responsibility”; AI will bring great power but just like our role in bringing up our children, we can’t abrogate the responsibility to the AI as well until we have reason to trust it. If hope is the sword that we can use to prosecute opportunities, the scepticism is the shield we need to use to protect ourselves from overreach.
Let’s spend some time from today treating the technologies in our strategies and architectures more as though they will operate with the diversity and unpredictability of humans. Some will be quite immature and may require a doctrine of supervision and development like Gina Ford’s theories on childcare. Others will be more trustworthy, but nobody is fallible so we can never completely lower our guard.
- Computing, 11 April 2024, OpenAI and Meta set to unveil AI models capable of reasoning and planning ↩︎
- Financial Times, 10 April 2024, Speed of AI development stretches risk assessments to breaking point ↩︎
- Pennsylvania State University, 7 May 2019, People more likely to trust machines than humans with their private information ↩︎
- ScienceDaily, 13 April 2021, People may trust computers more than humans ↩︎
- See Digility’s views on Secure by Design at https://digility.net/secure_by_design/ ↩︎
