“Picture a system that makes decisions with huge impacts on a person’s prospects – even decisions of life and death. Imagine that system is complex and opaque: it sorts people into winners and losers, but the criteria by which it does so are never made clear. Those being assessed do not know what data the system has gathered about them, or with what data theirs is being compared. And no one is willing to take responsibility for the system’s decisions – everyone claims to be fulfilling their own cog-like function.
This is the vision offered to us by Franz Kafka in his 1915 novel, The Trial. In that book, Kafka tells a parodic tale of an encounter with the apparatus of an indifferent bureaucracy. The protagonist, Josef K, does not know why he has been arrested, or what the evidence against him is; no one is willing to take responsibility for the decision, or to give him a proper account of how the system works. And it ends gloomily, with Josef K utterly defeated, resigning himself to his fate.
Fast forward 100 years and artificial intelligence and data-driven computer systems are frequently portrayed in a similar way by their critics: increasingly consequential, yet opaque and unaccountable. This is not a coincidence. There is a direct link between the trials of Josef K and the ethical and political questions raised by artificial intelligence. Contrary to the hype, this technology has not appeared fully formed in the past couple of years. As the historian Jonnie Penn has recently pointed out, it has a long history, one that is deeply entwined with state and corporate power. AI systems were developed largely to further the interests of their funders: governments, military and big business.
Most importantly, the models of decision-making that these systems sought to automate were taken directly from these bureaucracies. The two great pioneers of machine intelligence, Alan Turing and John von Neumann, both developed their prototypes in the crucible of the second world war. Under Von Neumann’s oversight, the very first task in 1946 of the very first general-purpose computer, the Eniac, was running computations for the hydrogen bomb.
In other words, the “intelligence” in “artificial intelligence” is not the intelligence of the human individual – not that of the composer, the care worker or the doctor – it is the systemic intelligence of the bureaucracy, of the machine that processes vast amounts of data about people’s lives, then categorises them, pigeonholes them, makes decisions about them, and puts them in their place. The problems of AI resemble those of the Kafkaesque state because they are a product of it. Josef K would immediately recognise the “computer says no” culture of our time”.
To save us from a Kafkaesque future, we must democratise AI | Stephen Cave https://www.theguardian.com/commentisfree/2019/jan/04/future-democratise-ai-artificial-intelligence-power
