I was attracted by the title of this essay. Seeing, naming, knowing imply a linear progression: from one to the next, Or: seeing + naming = knowing. My interest is epistemological: about what kinds of knowing and knowledge are produced by/with the kind of machine logic Khan describes, and how.
She provides some answers. For example, ‘What we have is a banal, distributed corporate information collection service running under the banner of intellectual inquiry. Its tendrils gather up our strong and weak desires to freeze us as consumers forever’. As ever, we need to look at the economic factors underlying such sociotechnical systems. Data is always collected by someone, for some purpose and in someone’s interest.
We’re all complicit in such a system, to some degree. ‘We hold the responsibility of understanding an underlying ideology of a system that interprets images,’ Khan writes, ‘and to fully grasp why it needs to pretend to be objective in order to function as a system.’
I pointed out that not everyone who works in commercial AI is evil. I recently had the pleasure of meeting Ovetta Sampson at Google, for example.
The problem is quick technological fixes that address symptoms and not causes, besides generating profit or increase productivity for some and not others. ‘In place of civic and human investment are machine vision cameras,’ Khan writes (p.41)
. . .
Where is the line that determines the contexts in which things are or are not acceptable? I mentioned my work in trying to create an ‘ethical’ AI system that doesn’t bias one culture over another. The question emerged: Are there any human behaviours that go across all cultures? Walking? (Yes, but not everyone within a culture) Smiling and laughing? (Yes, unless you’ve had too much Botox).
More broadly, if it’s not practical or efficient for a police force to investigate every single potential crime or criminal, why shouldn’t it use statistics and pattern recognition – what Khan calls ‘the outsourcing of human interpretation’?
Khan again provides some answers: ‘The software does not take into account the most deeply unethical issues involved in policing: what the police’s predispositions to the red zone are,’ for example (p.39). The AI only predicts how an area will be policed, not how crime might occur. In addition, the software enables police to shield themselves behind the supposed lack of bias of machine learning.
. . .
We can dig deeper, below the level of economics and culture, to the ontological or metaphysical level: the reality system of values and beliefs that underlies the society she describes. What kinds of knowledge, ideas, things can legitimately exist, and which are excluded or unseen?
Here again she provides some clues:
‘Take the purpose of a simulation – say, of a person moving. The goal is to capture the essence of that person moving, not to capture it perfectly’ (p.21)
‘The original determination of essence creates an over-essentialized self, which proliferates and becomes immovable, incredibly difficult to revise. ‘ (p.6)
‘There’s no break between the constructed model that’s underneath the world and the reality that is produced.’ (p.41)
This suggests a reality system that values the essence of things (language, classification, measurement) over their existence (that they exist at all, that each one is different, that they possess qualities not describable through language etc). These two align with the reality systems described by philosopher Federico Campagna. I try to describe this here.
For example, Khan mentions the mis-naming of indigenous American people by colonisers as ‘Indians’ and ‘savages’, and how this served to justify violent attitudes and actions towards them. She writes that ‘the claim to objectivity makes a similar lossy, erasing, violent, stupid, shallow misnaming of people harder to even see. What’s taking place may be comparable to what the Puritans did’.
To some extent this is present in science and it’s search for ‘universal truths’. I mentioned Alan Blackwell, who writes that if we treat AI as a science, we expect it to be the same everywhere. But if we instead regard it as literature, we would expect it to be different everywhere.
We should also be aware that liberals, too, speak their own language, as Janan Ganesh points out.
. . .
We discussed imaginaries around AI. Khan has written previously:
Artificial intelligence perches close to us, above us, like a gargoyle, or a dark angel, up on the ledge of our consciousness. Artificial intelligences are everywhere now, albeit in a narrow form – cool and thin in our hands, overheated metalwork in our laps. We are like plants bending toward their weird light, our minds reorienting in small, incremental steps towards them. (From Atlas of Anomalous AI)
We discussed this view from above: the top-down perspective, as in maps, that implies an objective, gods-eye view, but misses so much of what is visible on the ground. The difference between modelling and simulation, prediction and observed outcome. Someone mentioned that participants in studies often use the word ‘oracle’ to describe AI.
Is true objectivity, free of bias, possible? Khan proposes, ‘For a neural network to read the image “objectively,” it would have to not be made by human hands or run on historical data of any kind.’ (p.37). This is an interesting question. On one hand, arguably we already have machines talking and producing content only for other machines. On the other, can any AI really run completely autonomously? I remain skeptical.
However, as Khan points out, machines can see things in more precise ways than humans. And one promise of AI is that it might actually help to counter, or at least illuminate, human biases. Our own ways of seeing, she observes, are always occluded, hazy, partial and lazy. Shouldn’t we demand better from AI?
On the other hand, she writes:
Maybe it isn’t perfect seeing, but critical seeing that we need. Critical seeing requires constant negotiation. We negotiate incorrect or imprecise naming through revision of our own beliefs. When we see, we take in the ‘data-points’ of an image: color, form, subject, position. We organize the information into a frame that we can understand.
. . .
Khan identifies another imaginary in the dominant mindset of Silicon Valley: ‘America and technologists leaving behind the known, the body, race, politics, to enter a realm of pure engineering achievement, to be rewarded by wealth’. This is part of the ‘Californian Ideology’:
Unable to surrender wealth and power, the white people of California can instead find spiritual solace in their worship of technology. If human slaves are ultimately unreliable, then mechanical ones will have to be invented. The search for the holy grail of Artificial Intelligence reveals this desire for the Golem - a strong and loyal slave whose skin is the colour of the earth and whose innards are made of sand. Techno-utopians imagine that it is possible to obtain slave-like labour from inanimate machines. Yet, although technology can store or amplify labour, it can never remove the necessity for humans to invent, build and maintain the machines in the first place. Slave labour cannot be obtained without somebody being enslaved.
. . .
Most of the participants in this reading group session were practitioners, designing or using AI systems. We therefore discussed the contrast between abstract, poetic imaginaries of AI as a kind of alien intelligence, versus its day-to-day use as a mere tool. Someone pointed out that people’s fear of AI often disappears when they start playing with it. You shift focus to practical concerns, adjusting the metaphorical hand grip, the goggles.
One artist in the discussion was inspired by Khan’s sentence, ‘If you were to fill out a god’s eye view of society, what bodies do you imagine in it?’ What is the body of AI? What does it feels like, looks like, smells like? Here I would mention the Google engineer who told me recently that AI won’t get ver far without robotics: it needs to sense and navigate the real world.
Plants turned out to be an unexpected, recurring theme. Not only as an alternative ‘alien’ intelligence that we don’t understand completely, but also for operating on other-than-human timescales. One participant works with plants and agricultural practices, and these provided a useful comparison with AI: large-scale industrial agriculture versus small-scale cultivation and gardening (nice example here). I observed that the metaphor of human students as plants to be cultivated in the classroom has fallen out of favour in education, but maybe this works for AI.
. . .
Let’s drive towards problems and potential solutions.
‘[W]hen machinic tools moved from physical engineering to social engineering, from production of material to production of images and ideas, from workhorse machines to vision-machines, they became powerful ideological containers’ (p.23).
Now we can also add language – though Khan doesn’t discuss it much in the essay, I mentioned Yuval Noah Harari has discussed Yuval Noah Harari, who writes that now that AI has mastered language, which he regards as ‘the operating system of human culture’, it can hack the system. ‘What would it mean for humans to live in a world where a large percentage of stories, melodies, images, laws, policies and tools are shaped by nonhuman intelligence,’ Harari writes, ‘which knows how to exploit with superhuman efficiency the weaknesses, biases and addictions of the human mind — while knowing how to form intimate relationships with human beings?’
. . .
What to do about all this? What responses are possible, and useful? ‘[T]here must be space for human intervention in machine visual culture,’ writes Khan.
For one thing, she proposes, ‘Machine-learning engineers and designers deploying their vision systems must account for their blind spots instead of gesturing at the machine, offloading responsibility.’ (p.34)
We, the users and those affected by these systems, also have some responsibility. ‘We must make a practice of actively naming the flaws embedded in bad seeing. There need to be collaborative paths to a machinic naming.’ We ‘must stay alert to automation bias, in which we begin to value information produced by machines over ambiguous human observation.’ (p.35)
Artists often are well placed to critique techno-social systems like AI, and Khan mentions Mimi Ọnụọha as one. I mentioned that there is a danger that overly superficial uses of AI could have the opposite effect. As with some work critiquing surveillance, simply employing technological means as way of raising awareness (like putting surveillance cameras in the gallery) can instead normalise surveillance.
More generally, Khan asks, ‘Can we build machine vision to be critical of itself? Even as we learn to see alongside the machine’ (p.39). This is a hard problem that I try to work on: Can a machine critique itself? Can it admit when it’s wrong or biased or uses incomplete information? Can it embrace not knowing? After all, It’s called machine learning – the machine can be re-trained.