(Commonwealth_Europe) Over the past two years, the proportion of UK companies using at least one AI tool has doubled, rising from just 9% in 2023 to 18% by early 2025, according to the Office for National Statistics. Among larger employers, nearly one in three have adopted some form of AI technology. However, this rapid uptake has not necessarily been accompanied by a corresponding increase in understanding or expertise. Often, organizations integrate AI systems into their operations without fully grasping how these technologies function internally or their implications.
This growing reliance on artificial intelligence comes at a time when the UK is already grappling with a significant digital skills shortage, which government figures estimate is costing the economy around £63 billion annually. The lack of skilled professionals capable of developing, managing, and critically evaluating AI tools raises serious concerns about the potential consequences of this technological acceleration. Without the right expertise, companies risk making poor decisions based on systems they do not fully understand.
Spencer Pickett, Chief Technology Officer at Software Development UK, has been vocal about the risks associated with this unrestrained adoption of AI. He describes the current environment as a “gold-rush mentality,” where the focus is on rapid deployment rather than thoughtful implementation. According to Pickett, many businesses are drawn to the promise of AI’s capabilities without paying sufficient attention to the safeguards needed to ensure its responsible use. He likens the situation to hiring a PhD-level expert who refuses to explain their reasoning—highlighting the opaque nature of many modern AI systems. These systems, which are often built using machine learning algorithms, do not rely on explicitly programmed rules. Instead, they teach themselves by processing vast amounts of data, which allows them to detect patterns and make predictions. However, this also makes their internal decision-making processes difficult—if not impossible—for humans to interpret.
This lack of transparency poses a significant problem, particularly in industries where decisions must be traceable and justifiable. In sectors such as banking, insurance, and healthcare, organizations are subject to strict regulatory requirements that demand clear explanations for how decisions are made. If an AI system recommends denying a loan, making a medical diagnosis, or issuing an insurance policy, regulators and affected individuals alike must be able to understand the rationale behind the outcome. Yet with many current AI models, such explanations are either lacking or incomprehensible to non-specialists.
Pickett warns that this opacity can give rise to a host of problems. One of the most concerning is the potential for invisible errors. If an AI system makes a mistake—especially one that affects people’s lives or livelihoods—there may be no obvious way to detect or correct it, given the lack of transparency in the model’s logic. Furthermore, as regulatory oversight tightens, companies may face legal and reputational consequences if they cannot provide clear, auditable justifications for AI-driven decisions. Another risk is the erosion of trust. If employees or customers perceive AI systems as arbitrary or unfair, they may lose confidence in the technology altogether, which could hinder its adoption and effectiveness in the long run.
According to Pickett, the key to addressing these risks lies in developing tools and practices that can make AI systems more accountable and understandable. He and his team are working on methods to increase transparency, such as techniques that allow models to explain how they arrived at particular conclusions, identify potentially high-risk decisions, and ensure human oversight in sensitive situations. The goal is not to slow down innovation but to ensure it is deployed safely and ethically. By putting boundaries in place and requiring human sign-off where necessary, organizations can better manage the risks associated with AI use.
Pickett believes that the board level requires a significant cultural shift. Executives must first come to terms with the limitations of today’s AI systems before trusting them with critical decisions. Until they acknowledge that many AI tools currently function as “sealed boxes,” they may fail to ask the essential questions that protect both customers and the business itself. True innovation, he argues, begins with understanding—by metaphorically “opening the lid” and examining what’s inside. Only then can companies responsibly harness the full potential of AI while maintaining safety, accountability, and trust.
