24 February 2018

[Paper] The Intuitive Appeal of Explainable Machines

Be the first to share

The Intuitive Appeal of Explainable Machines
by Andrew D. Selbst (Data & Society Research Institute; Yale Information Society Project) and Solon Barocas (Cornell University)
February 19, 2018, 59 pages

Abstract

As algorithmic decision-making has become synonymous with inexplicable decision-making, we have become obsessed with opening the black box. This Article responds to a growing chorus of legal scholars and policymakers demanding explainable machines. Their instinct makes sense; what is unexplainable is usually unaccountable. But the calls for explanation are a reaction to two distinct but often conflated properties of machine-learning models: inscrutability and non intuitiveness. Inscrutability makes one unable to fully grasp the model, while non intuitiveness means one cannot understand why the model’s rules are what they are. Solving inscrutability alone will not resolve law and policy concerns; accountability relates not merely to how models work, but whether they are justified.

In this Article, we first explain what makes models inscrutable as a technical matter. We then explore two important examples of existing regulation-by-explanation and techniques within machine learning for explaining inscrutable decisions. We show that while these techniques might allow machine learning to comply with existing laws, compliance will rarely be enough to assess whether decision-making rests on a justifiable basis.

We argue that calls for explainable machines have failed to recognize the connection between intuition and evaluation and the limitations of such an approach. A belief in the value of explanation for justification assumes that if only a model is explained, problems will reveal themselves intuitively. Machine learning, however, can uncover relationships that are both non-intuitive and legitimate, frustrating this mode of normative assessment. If justification requires understanding why the model’s rules are what they are, we should seek explanations of the process behind a model’s development and use, not just explanations of the model itself. This Article illuminates the explanation-intuition dynamic and offers documentation as an alternative approach to evaluating machine learning models.

Conclusion

Daniel Kahneman has referred to the human mind as a “machine for jumping to conclusions.” Intuition is a basic component of human reasoning, and reasoning about the law is no different. It should therefore not be surprising that we are suspicious of strange relationships in models that admit of no intuitive explanation at all. The natural inclination at this point is to regulate machine learning such that its outputs comport with intuition.

This has led to calls for regulation by explanation. Inscrutability is the property of machine learning models that is seen as the problem, and the target of the majority of proposed remedies. The legal and technical work addressing the problem of inscrutability has been motivated by different beliefs about the utility of explanations: inherent value, enabling action, and providing a way to evaluate the basis of decision-making. While the first two rationales may have their own merits, the law has more substantial and concrete concerns that must be addressed. But those that believe solving inscrutability provides a path to normative evaluation also fall short of the goal because they fail to recognize the role of intuition.

Solving inscrutability is a necessary step, but the limitations of intuition will prevent such assessment in many cases. Where intuition fails us, the task should be to find new ways to regulate machine learning so that it remains accountable. Otherwise, if we maintain an affirmative requirement for intuitive relationships, we will potentially lose out on many discoveries and opportunities that machine learning can offer, including those that would reduce bias and discrimination.

Just as restricting our evaluation to intuition will be costly, so would abandoning it entirely. Intuition serves as an important check that cannot be provided by quantitative modes of validation. But while there will always be a role for intuition, we will not always be able to use intuition to bypass the question of why the rules are the rules. Sometimes we need the developers to show their work.

Documentation can relate the subjective choices involved in applying machine learning to the normative goals of substantive law. Much of the discussion surrounding models implicates important policy discussions, but does so indirectly. Often, when models are employed to change our way of making decisions, we tend to focus too much on the technology itself, when we should be focused on the policy changes that either led to the adoption of the technology or were wrought by the adoption. Quite aside from correcting one failure mode of intuition, then, the documentation has a separate worth in laying bare the kinds of value judgments that go into designing these systems, and allowing society to engage in a clearer normative debate in the future.

We cannot and should not abandon intuition. But only by recognizing the role intuition plays in our normative reasoning can we recognize that there are other ways. To complement intuition, we need to ask whether people have made reasonable judgements about competing values under their real-world constraints. Only humans know the answer.

Be the first to share
25 April 2018
Two reports on people’s attitudes and understanding of digital technologies
Doteveryone, a UK think tank that champions responsible technology for the good of everyone in society [similar to Milan's newly founded Digital Culture Center], published two reports this year: the first one, The Digital Attitudes …
15 December 2017
The body is the missing link for truly intelligent machines
It’s tempting to think of the mind as a layer that sits on top of more primitive cognitive structures. But Ben Medlock, co-founder of SwiftKey, argues that the layered model of cognition is wrong. The point …
15 December 2017
[Book] Bit by Bit: Social Research in the Digital Age
Bit by Bit: Social Research in the Digital Age by Matthew J. Salganik Princeton University Press 2017, 448 pages [Read online] An innovative and accessible guide to doing social research in the digital age In just the past several years, we …
2 December 2017
Five ways to fix statistics
As debate rumbles on about how and how much poor statistics is to blame for poor reproducibility, Nature asked influential statisticians - Jeff Leek, Blakeley B. McShane, Andrew Gelman, David Colquhoun, Michèle B. Nuijten & …
22 November 2017
Human-centred design in global health: A scoping review of applications and contexts
Human-centred design in global health: A scoping review of applications and contexts By Alessandra N. Bazzano (1), Jane Martin (2), Elaine Hicks (3), Maille Faughnan (1), and Laura Murphy (1) PLoS ONE12(11): e0186744. Published: November 1, 2017 Health …
27 October 2017
Self-driving cars and humans must learn a common language
For all the intricate technology required for autonomous cars — the sensors to replicate eyes and ears, the computers and algorithms to serve as the car’s brains, the high-definition 3-D maps to guide them — …
4 August 2017
The human side of autonomous cars — with Nissan Research’s Melissa Cefkin
Reposted from Medium Beyond the engineering challenge of creating cars that drive themselves lies the social challenge. Before autonomous cars are ready to navigate our roads, they must be able to navigate the vastly more complicated …
2 July 2017
To improve people’s health, listen to social scientists
With an ageing population, a rise in long-term conditions, growing health inequalities and a lack of political will to ensure that funding is increased in line with demand, the UK's National Health Service has been …

We are an international experience design consultancy helping companies and organisations to innovate their products, services and processes by putting people and their experiences first.

13 June 2018
Invito: DesAlps design thinking workshop per la tua startup

Hai una startup? Hai mai pensato ai benefici che potrebbe trarre dal Design Thinking? Questa è l’opportunità per scoprirlo! DesAlps Workshop #2: Il Design Thinking per la tua startup! Giovedì 28 giugno 2018 – dalle 9:30 alle 17:00 @ I3P | Corso Castelfidardo 30/a, Torino —– Nell’ambito del progetto europeo DesAlps, un team di esperti […]

3 May 2018
Invito: DesAlps innovation workshop per le PMI del territorio piemontese

Experientia è lieta di invitare le piccole e medie imprese del territorio piemontese al: DesAlps Workshop #1: Il Design Thinking per le PMI Venerdì 18 maggio 2018 – dalle 9.00 alle 17.30 @ Rinascimenti Sociali | via Maria Vittoria 38, Torino Scopri i vantaggi che il Design Thinking può portare al tuo business per prepararvi […]

27 April 2018
Impact through Design – Event in Turin, Italy

(Scroll down for English) Come può il design generare impatto? In continuità con il summit International Days of Deans and Experts: Impact Through Design (Barcellona, aprile 2018) Torino ospita il primo evento dedicato alla prospettiva italiana sull’impatto generato attraverso il design in riferimento ai Sustainable Development Goals definiti dalle Nazioni Unite. Attori dell’ecosistema territoriale ed […]

26 February 2018
Designing Patient Solutions with Meaning: Review of technology and health apps for a better patient experience

Technology is radically changing the world of healthcare and acts at different levels in support of different stakeholders. There are several digital tools on the market (e.g. wearables, virtual reality, robotics, etc.), many of which work in integration with each other, for example, wearables connected to IOT applications. Experientia’s report provide an overview of some […]

22 December 2017
Happy new year 2018!
30 November 2017
New European study on e-Government initiatives

The latest eGovernment benchmark report of the European Commission shows significant improvement on cross-border availability of digital public services and accessibility of public websites from mobile devices in EU Member states. The study also indicates a need for improvement in transparency of public services delivery and use of supporting technology like eIDs or eDocuments. Performance […]

See all articles