Paper

This website is a collaboration on how to use uncertainty analysis and risk science to improve the fairness, friendliness, security, transparency, efficacy, and correctness of algorithms that automate or inform practical decision making.

The collaboration began in drafting some testimony to be presented to Parliament which has since developed into a manuscript for a 'perspective' paper for publication in a scholarly journal and for broader collective thinking on the subject in workshops and thinkovers. The table of contents below accesses parts of the draft paper. The Navigation panel on the right of the screen gives links to subpages and related websites.

Kindling (general thoughts and arguments)

Science-fiction writer Isaac Asimov's first "law of robotics" is that a robot may not injure a human being or, through inaction, allow a human being to come to harm. But no one has yet explained exactly what is included under 'harm', nor how a robot or computer would be able to recognise harm in the first place. This project on humane algorithms is a step toward answering at least part of that question.

Recent experience in human−machine interactions has made clear what some of the problems are, and that harm can range widely from annoyance to injustice, and from aggravation to catastrophe. Humane algorithms refer to computer software that, insofar as is possible,

• 1. Accepts control from humans, or relinquishes it back to them, in ways that humans find workable,

  2. Checks human inputs for errors and misconceptions,

  3. Self-diagnoses problems or aberrant behaviour of the algorithms,

  4. Protects privacy by securing or progressively anonymising personal information,

  5. Creates outcomes and situations that humans would judge as equitable or fair,

  6. Recognises, accepts and accommodates diversity among users,

  7. Flexibly accepts inputs from humans in disparate formats,

  8. Does not require humans to respond to queries precisely, immediately, or at all,

  9. Handles errors and unusual conditions in ways that do not result in catastrophic system failure,

  10. Makes conservative/fail-safe assumptions,

  11. Does not unnecessarily burden humans,

  12. Is transparent, or at least interrogatable, about its internal functioning,

  13. Helps humans to understand outputs and outcomes effected by the algorithms, and

  14. Complements human skills.

In essence, humane algorithms are computer algorithms that work and play well with humans, anticipating and serving their needs and frailties.

<<Is this list complete, well-structured? What are the natural groups? As a lumper, I think there are fundamentally only 4 issues:

1. Fairness (including appealability, transparency, ethics, social justice, privacy, accountability, and equity across diversity so self-driving cars don’t kill people of colour or pregnant women),

2. Workability (robustness, Bill Kahan, no aborts, recovery from errors, smooth control transfer, accommodation of diversity, user-friendliness, accepting user-specified units, no tyranny, standards for less obnoxious interfaces),

3. Risk-awareness (security, handling input imprecision, missing values, appreciation of tail risks, assumption checking, unit checking, recognition by the software when it is being used incorrectly, resistance to vandalism, gaming and misuse), and

4. Trackability (tracing, repeatability, justification trail, error correction, data updating, provenance).>>

<<Nick: In addition to this there is the problem that humans often see algorithms as more accurate or more reliable than humans are generally capable of being, so they prefer a computer's answer over advice from humans. This phenomenon has been called algorithm appreciation (Logg et all 2018). This may be related to what Ferson et al. (2015) called the "authority problem" in which people attributed much less or even zero uncertainty to numerical claims. Should a humane algorithm account for this by frequent and clear disclaimers? Perhaps this would go under the checking aspect of risk-awareness.>>

<<Nick: Trackability should be renamed accountability>>

Possible titles for the paper

How Should Algorithms Address Uncertainty?

Uncertainty and Fairness in Automated Decisions

Improving Automated Decisions by Considering Uncertainty

Abstract

<<we need an abstract. It might include some of the following theses:>>

o The faceless bureaucrat is being replaced by the heartless algorithm.

o Serious questions about fairness have been raised.

o Transparency is often desirable but is not always applicable and will not always work.

o Standards for algorithm review and monitoring are needed.

o Clarity is needed on the right of appeal in decisions made by algorithms.

o Algorithm development needs statistical science, which is underfunded.

o Algorithms should use uncertainty-aware algorithms.

o Research is needed for algorithm verification methods.

o Interfaces that collect input for algorithms should be more humane.

<<Dan: The machine learning/artificial intelligence boom that is in its early stages. It seems that many algorithms in the near future will be of the "black box" variety where a computer was given lots of data and then allowed to create its own internal rules for decision-making in order to get the desired outcome. We should more explicitly address this issue.>>

Maybe a concluding sentence is

An algorithm is humane if it is user-friendly and accommodating to people, and it handles diversity and uncertainty when appropriate, and leads to fair and just results in a way that is transparent whenever possible.

<Edo: we need to define algorithm first. We should not confuse Algorithm with interfaces or computer programme. An algorithm is a set of rules, a programme can include interfaces >

<Dominic : I think transparency is a misnomer in this context. There are lots of things we use and commonly accept despite having little to no understanding of how they work. If we adjust the discussion here and ask would we use cancer treatment 'A' that we understand perfectly but has a 80% effectiveness in treating a particular condition, or would we take treatment 'B' which we have little understanding of but has a 95% effectiveness in treating the same condition. >

1) For a scientist or engineer, the idea of Parliament holding a hearing about algorithms may at first seem quite strange, rather like having a hearing about numbers. The word ‘algorithm’ is just another word[1] for formula or decision process to arrive at a decision or computational answer to some question. Algorithms are useful when the process to find the result or decision must be a conscientious one using explicit definitions and rules that could be justified to others if necessary. Using an algorithm helps decision makers to be more objective and, hopefully, smarter and fairer in reaching their conclusions.

2) However, delegating decision making to an algorithm can also make decision making heartless and tyrannical because it takes the decision out of the hands of human beings and invests it in a computer that cannot see any mitigating conditions or other relevant issues not already considered in the algorithm. Applying an algorithm without the possibility of review or appeal, or without initial and regular reassessment of algorithm’s correctness and propriety, can easily lead to unjust and otherwise undesirable outcomes.

<<Dan: While the primary weakness of simple algorithms is that they often ignore consideration of context that might influence a decision process, they also have one ethical advantage over human judgment. Because they are not instantaneous individual decisions, algorithms allow for careful deliberation and public debate. The problem, as Scott points out, is that we don't publicly examine these algorithms - but we should. This gets at an issue rarely discussed in ethics. Philosophers often discuss what the correct ethical action might be, but gloss over the idea that even individuals who want to do the "right" thing are often forced to make fast decisions without the benefit of deliberation. Life is full of potential trolley problems. One could even imagine a future where people might select their preferred self-driving car by the driving algorithm that matches their personal risk profile. (I want a super-cautious car that is boring. My neighbor wants a sportier car that has an ever so slightly higher probability of crashing.)>>

<<Edo: Algorithms are also good but sometimes misused. They are use as shields, taking out responsibility from the decision makers (in case of a wrong or controversial decision). Algorithms should remain supporting tools, they should not take out individual responsibilities.>>

3) Although this topic has been the subject of some recent concern nationally and internationally because of apparent inequities in several cases (e.g., Devlin 2017; Kleinman 2017; Rae 2015; Sweeney 2013; Sandvig et al. 2016; Caliskan et al. 2017), it is not fundamentally a new problem at all. It is the same problem faced by institutional decision makers, including legislators and jurists. The novel element in current considerations is that algorithms are usually embedded in unseen computer programs that are becoming more and more pervasive and intrusive in the daily lives of everyone. These programs and the algorithms they use may not get the review they should be getting. The issue today is that the faceless bureaucrat is being replaced by a literally heartless algorithm.

Algorithmic unfairness

4) Although the purpose of using an algorithm is to ensure objectiveness and thus intelligence and fairness in the decision making process, there is no guarantee that results from an algorithm will actually be smarter or fairer. An algorithm can be a stupid one, and it can be designed to reflect prejudice. Even when this is not intentional, the programmers or analysts who develop algorithms are people (usually), and their biases and ill-founded preconceptions can creep into the formal algorithms. These are stupid algorithms because they are poorly designed.

For example, the Facebook News Feed is an algorithm that curates material for users who would otherwise be awash in information. While this is a useful and maybe necessary service, the algorithm has raised questions for several years regarding how its selection process works (Manjoo 2017). Facebook was relatively unconcerned about the effects of the News Feed algorithm until the 2016 U.S. presidential election when it became a news topic in its own right for propagating fake news and creating "filter bubbles" where users primarily see material that reinforces their existing beliefs.

The algorithm itself has been carefully optimized for the past decade based, not on the judgment of Facebook programmers and executives, but through endless testing of variations of the algorithm to see which version maximizes the time users spend on Facebook. Thus, the fundamental criterion for judging algorithmic value is not utility, integrity or civility, but increasing screen time. This is not particularly surprising given that Facebook’s income is derived from selling micro-focused advertisement. Facebook is not an essential service, so people who do not agree with this business model are free to not use Facebook. However, there are numerous algorithms that are not so easy to avoid.

<<Edo: correct. An obvious example are the algorithms used by banks to release mortgages. >>

5) An algorithm can also manifest unfairness because it encodes social structures and reflects the at-large culture in which it operates. Algorithms may be more susceptible to this than people because they lack human consciences and cannot counteract biases that they learn from data (Devlin 2017; Caliskan et al. 2017). For instance, a computer scientist found that Google searches involving black-sounding names are more likely to elicit advertisements suggestive of a criminal record than white-sounding names (Sweeney 2013; MIT Technology Review 2013). Subtle biases among humans can be magnified when computerized by algorithms. Algorithms should be developed with more defences and rigorously tested to guard against such outcomes.

<<Dan's comment: Machine learning systems can create black boxes that don't allow for such tweaking.

Scott's response: Unless you had a police force among the algorithms! Wait, is this the plot of The Terminator? Actually, I kind of think it would be smart to set up algorithms to study the performances of other algorithms. Fighting fire with fire. Information with more information. Algorithms with algorithms. Nothing prevents this in theory, but it might be difficult to accomplish.

Dan' response: I just heard an interview with the author of the book, We Are Data, which is about the overwhelming power of internet algorithms. One of the points that he mentions that is important to our discussion is that these algorithms are constantly updated. This gets back to Scott's idea of needing algorithms to assess the algorithms - fighting fire with fire since humans could not possibly keep up with the changes.

https://nyupress.org/books/9781479857593/

>>

6) Moreover, because algorithms must often be rather complex to account for complicated and diverse situations, they can sometimes produce unexpected and unintentional outcomes that turn out to be undesirable in a particular case, or in aggregate create an undesirable pattern. For instance, seemingly reasonable conventions in business may unintentionally and unexpectedly create economic injustice in lending decisions (Rae 2015).

7) Although intentional unfairness is particularly despicable, in some respects it does not matter whether the breach of fairness is on purpose or not; unfairness is still the outcome. The question at hand is what can and should be done about algorithms that create, reflect or magnify unfairness or other errors.

<< Edo: There is a quite stringent regulation about the use of personal data and using of cookies from websites (at least in EU) but not requirements for the fairness of the algorithms. Algorithms should pass a battery of tests (fighting algorithms with algorithms) and they should be certified. We should develop such certification and open a new business :) >>

Transparency is often but not always useful

8) One general approach to remedy algorithmic unfairness is to publish the algorithms so everyone can see how they operate, and predict their implications to check that some groups are not unduly burdened. This is often a very desirable strategy to ensure fairness, as it will make plain the decision that would be reached in any particular case, allowing the public to assess not only the reasonableness of the rules, but also the desirability of the outcomes it will produce. In legislative deliberation, for instance, transparency is often considered an essential feature for the reason that it helps to minimize the improper effects of bias.

<<Dan's comment: I would argue that transparency is a necessary, but not in itself sufficient requirement for algorithmic fairness. However, transparency is the most straight-forward solution to the hidden algorithm problem.

Scott's response: Not always. I could publish the proprietary Risk Calc code in a way that would allow it to be compiled but not understood or reverse engineered. (Just convert all variable names to long strings composed of zeros, capital Os, ones and lower case Ls. Machine interpretable, but impossible for humans.) It's easy to obfuscate any algorithm. Sometimes I think IRS did this with the 1040 form. And predicting global or social-level outcomes from all but the most trivial algorithms is almost impossible, even when people actually understand the algorithm in a local way.

Dan's response: I think we are on the same page, but using different definitions of "transparent." If something is freely available, but not understandable, I don't think it meets the spirit of transparency. One example is the outrageously long legal contracts known as "terms of service" that no one reads, but probably should.>>

9) There are three main problems with insisting the algorithms be public. First, algorithms may be quite complex or difficult to understand. If a person cannot understand the rules encoded in an algorithm or cannot follow their implications in decision making, then the transparency is ineffective and irrelevant. Second, governments should not generally require publication of proprietary algorithms that represent trade secrets because doing so would likely seriously harm economic performance both domestically and internationally. Third, making the rules embedded in a decision algorithm public exposes the decision maker to gaming, in which a person uses the rules and procedures of the algorithm that were intended to identify an optimal result to manipulate the system for the person’s benefit instead, such as when an unscrupulous health care provider adapts a patient’s medical treatment to government payment systems, rather than purely to the patient’s medical needs.

<<Dan's comment: "Difficult to understand" will become an increasingly serious problem as machine learning creates more "black box" expert systems.>>

<<Scott's reply: I completely agree. The point of https://sites.google.com/site/davmarkup/ is trying to imagine how to redress some of this, although I don't think anyone has a prayer against the trends created by machine learning.>>

<<Dan's comment: I'm less concerned about proprietary. Intellectual property issues are a concern, but isn't that the point of a patent? If a company wants to maintain a trade secret, then they should live the consequence of no one trusting their product. There might also be security issues, but how much interaction does the public have with classified algorithms? Maybe a lot (Snowden revelations)?

I'm not concerned about gaming, All systems are susceptible to gaming. The degree of gaming is probably more a function of how well they are constructed rather than their level of transparency. Regarding transparency, the distinction is whether the systems rules must be inferred or whether they are made explicit. If they must be inferred, then the system becomes preferentially gamed by the most sophisticated players (until everyone catches on). Explicit rules at least level the playing field from the beginning.>>

<<Edo: This connect with my point above. Algorithms should be certified. It does not matter if they are proprietary algorithms or not they should be fair. We should concentrate more on the outcomes of the algorithms and not on the way these outcomes are produced. It is like for the automotive, a product needs to be certified and pass some tests (safety/emission). We don't need to know how a car works, we are happy to know that it has passed the certification. I know that this might not be sufficient in same cases. We might risk another diesel-gate. If the algorithm detects that is being tested it might performed in a different way. But this is a problem of design robust and reliable tests.

>>

10) There is a model for transparency and accountability regulation for algorithms in decision making. The European Union adopted a due-process requirement, due to take effect next year, for decisions based “solely on automated processing” that “significantly affect” its citizens (EUR-Lex 2016). The rule gives EU citizens the right to an explanation of the automated decisions and the right to challenge those decisions. The effect of the rule is expected to be narrow, however, because it only applies to cases not involving human judgment “such as automatic refusal of an online credit application or e-recruiting practices without any human intervention” (Angwinaug 2016).

<<Dan's comment: While it is initially narrow, this is one of those regulations that will have increasing power over time as AI systems become more common. I suspect that within five years several Silicon Valley tech companies will be trying to eviscerate it.>> <<Dan's comment: The more I read, the more I think that this is a major point worth emphasizing. It's also likely that Silicon Valley lawyers will be fighing the EU regulation sooner rather than later.>>

Need for algorithms to handle variability and uncertainty

11) There is a growing recognition that many algorithms are developed from data sets that do not include diversity comparable to that of the general population (Kleinman 2017). A significant part of the problem is that the methods to collect appropriately representative data sets, that is, data sets that reflect the variability and diversity of the citizenry, are not widely known within the communities that are responsible for developing algorithms. What is the right level of diversity? How does one characterize it, and how does one demonstrate one has the right level? How much data is enough? Is more data always better, or does it have to be a certain kind of data?

12) The software engineers and other analysts who develop algorithms for uses great and small are often not well equipped by their training to answer these questions. We cannot expect that every team of software developers, every company of bankers, every board of doctors, every panel of domain-specific experts that develops decision algorithms will in all circumstances of note have staff competence in statistics and modern data science. A significant government initiative is needed to support education in statistical science to train the next generation of quantitative analysts and to increase the penetration of statistical thinking and methods into engineering more broadly. Additionally, research is needed to construct accessible software tools that can be widely used by engineers but also lay citizens to assess the representativeness of data sets and the properties of algorithms developed from those data sets. <<Dan: Emphasize this recommendation?>>

13) After an algorithm has been developed, it is employed to make decisions given inputs from particular situations and individuals. Those inputs may not always be perfectly known; there may be some imprecision about one or more inputs. How should that imprecision be handled within the algorithm? In most cases, this kind of uncertainty is simply ignored, but that can lead to decidedly suboptimal outcomes. It would be much better for the algorithm to use a proper uncertainty analysis to make a more informed decision. Uncertainty analysis can be used to ensure compliance with rules in cases where there is doubt about qualifications or the meeting of standards. In other situations, uncertainty analysis can be utilized to give the benefit of the doubt to people in uncertain cases where fairness is a salient issue.

14) It is well known that conscientiously accounting for uncertainty leads to better decisions. This is also true when computers are making the decisions via algorithms without humans. Yet, the use of quantitative methods to take account of uncertainty about inputs in decision-making algorithms is quite rare. The reasons, again, are insufficient human resources, namely, analysts trained in statistical science, and lack of widely accessible high-level software tools with which non-specialists could take account of uncertainty and project it in the calculations within algorithms.

<<Dan: Not to belabor the point, but I think black box uncertainty methods are going to be crucial in the machine learning era. Machine learning is inherently probabilistic, but not necessarily about the data itself...more work needed here.>>

15) Consider another example where uncertainty must be factored into the decision-making process to ensure reasonable decisions and desirable outcomes. In this case, the uncertainty is purely semantic. Suppose that published regulations define the endangerment of a biological species in terms of its abundance being below some threshold, and they likewise provide for special protections for the conservation of the species if it is endangered. In such a situation, a conservationist motivated to protect a species whose abundance is close to but above the threshold might decide to kill a few animals so the abundance falls below the threshold and the species benefits from the protections that would then kick in. This is an example of gaming, but it is also an example of perverse incentivization. An algorithm for defining endangerment that respects the semantic uncertainty of the term ‘endangered’ could prevent this by deploying conservation protections in a graded way. This could minimize the susceptibility to gaming and remove many of the problems of perverse incentivization.

<<Dan: The endangered species example seems a little off-topic, plus it makes me so sad I get distracted. However, I really like the semantic uncertainty discussion. Maybe we could use an example more closely related to the discussion above about bias?>>

<<Edo: a similar example might be a receiver of a social benefits. He/She can decide to not take a job offer to not lose the benefits. In both cases the rules (algorithm) should be design to minimise these possibilities.>>

16) The requisite statistical science methods needed in the design and use of uncertainty-aware algorithms include tools and techniques from probability theory, fuzzy control theory, interval analysis, robust design, and other fields. Improving the standard of algorithm accountability in the nation will require investments perhaps supplementing current initiatives in science, technology, engineering and mathematics.

<<Dan: This will need more specifics and references.>>

Tools for checking algorithm correctness

17) Apart from potentially being unfair, an algorithm may also simply be wrong in that it gives incorrect, improper, or suboptimal results. An unknown but surely very large number of currently used algorithms contain bugs or errant structures that cause them to yield incorrect results often or occasionally. Checking that an algorithm correctly implements the intended method for decision making is called “verification” by engineers. Developing tools for verification is an important area of current research. Industrial and perhaps governmental funding for such research in verification is vital to identifying and correcting algorithms before they reduce efficiency, create unfairness, or result in serious damage to property or human health.

<<Dan: I would like to do more research here regarding the current state of software QA/QC.>>

18) Although research in verification methodologies is a relatively young discipline within engineering, many strategies already known would be very useful if they were more widely applied. Spreadsheets, for example, are notorious for harbouring many and egregious errors (Panko 2000). Software exists to ‘read’ an Excel spreadsheet and convert the formulas and values inside it into a printable script that is much more easily checked than is the original spreadsheet, yet few users make use of this software. There are also software tools that can detect dimensional errors in such scripts automatically without the need for any human review. Although such software could have prevented the crash of NASA’s Mars Climate Orbiter which fell out of the Martian sky when it tried to enter orbit around the planet because of a trivial units incompatibility (Isbell et al. 1999), even today NASA does not employ automated error checking of all mission-critical calculations.

Humane interfaces

19) People interact with algorithms in many ways. Government administrative services often employ online fill-in forms, but commercial businesses and NGOs use e-commerce, automated attendants (“press 1 for sales”), mobile banking, robocalls, social media updating, among many others. Considerable effort has been invested in making these interactions more accessible to everyone through various assistive technologies. However, there has generally been less accommodation for people by the algorithms that use these interface technologies. In fact, in recent years algorithms almost seem to have become more tyrannical in these interactions.

20) This problem is especially serious for government as more and more of its public services are mediated over the Internet. Because circumstances and public policies constantly evolve, the interfaces are often in flux, and it has proven to be difficult to keep websites current and fully functional.

21) Several changes are desirable to make interfaces more accommodating to humans:

• • Inputs are sometimes unnecessarily identified as required (perhaps marked with an asterisk). Declining to answer by leaving a field blank, perhaps because the correct answer is unknown, should not necessarily prevent a well written algorithm from making use of the information that is available. <<Dan: A great idea, but private industry likes to ask for unnecessary personal information in order to sell user data to advertisers. This will require regulatory intervention.>>

  • Even when some inputs are required, it may not be necessary to require they be entered in a particular order. Algorithms should allow people to enter information in any order most convenient to the user.

  • Requiring inputs to have a particular format, or to be expressed on a particular scale or in particular measurement units, is unnecessary and can be inappropriate in many situations. A well designed algorithm should handle inputs in many intelligible formats. It should accept quantities in any appropriate units and check that their inputs have the correct dimensions. For example, an input field expecting a distance should be able to correctly interpret inputs giving kilometres, miles, feet or metres, without forcing a person to make the conversion. An entry without units or with non-conforming units such grams should precipitate follow-up questions.

  • Input fields expecting numerical quantities should be able to accept and process expressions of uncertainty in the inputs, such as ranges or plus-or-minus statements. Interfaces should accept inputs including linguistic hedges such as ‘about’, ‘no more than’, or ‘up to’ to modify numerical quantities (Ferson et al. 2014). People use such hedges when they are trying to be accurate and honest (Prince et al. 1982).

  • Multiple choice input fields should accept multiple, none-of-the-above, mixed, and other where possible. For instance, binary gender assignment can be unnecessarily troubling for intersex people and individuals who self-identify atypically. <<Dan: This is a great example. The discussion starts off talking about algorithm bias and then seems to swerve towards uncertainty expressions. This example ties the ideas together and should be emphasized.>>

  • Annotation or commentary should be allowed for people whose entries may require additional explanation. Inability to document caveats is a common source of user frustration.

• <<Edo: We are not considering the time. Often you are required to fill all the inputs in one shot or you lose everything. However, there are some smart solutions. It is possible to retrieve automatically information from social networks or browsers can store information for some common fields. Of course on social networks you can have fake profiles but it should be possible to create a personal database that can be connected with those algorithms. >>

22) All of these accommodations are entirely possible, but they are rarely employed, which causes frustration, aborted participation, and often error or ambiguity in the collected information. Parliament could promote the use of more humane interfaces by encouraging or requiring their deployment in government websites.

Summary of recommendations

23) To improve the accountability, fairness, correctness, and convenience of algorithms in decision making, it is recommended that Parliament

• • Promote and require transparency wherever appropriate,

  • Prohibit the use of secret algorithms by the government or its contractors,<<Dan: What about the AI inherent black box scenarios?>><<Edo: quite strong recommendation. is the purpose of the algorithm or its implementation that needs to be public?>>

  • Promote standards for monitoring and review of algorithms generally, <<Edo:This is the most important point>><<Dan: I agree.>>

  • Clarify rules on right of appeal in decisions made by algorithms,

  • Promote research in and popularization of methods for the development of uncertainty-aware algorithms,

  • Promote research in and wide application of algorithm verification methods, and

  • Promote the development and deployment of human-algorithm interfaces that are more user-friendly and accommodating for uncertainty, diversity, and convenience.

Notes

[1] Like the words ‘algebra’, ‘chemistry’, ‘zero’ and even ‘cotton’, ‘coffee’ and ‘candy’, the word ‘algorithm’ comes from the Arabic and reflects our rich inheritance from and through the Muslim world that was discovered in Europe during the Renaissance.

References

Angwinaug, J. (2016). Make algorithms accountable. The New York Times, 1 August 2016, http://www.nytimes.com/2016/08/01/opinion/make-algorithms-accountable.html?_r=0.

Caliskan, A., J.J. Bryson, and A. Narayanan (2017). Semantics derived automatically from language corpora contain human-like biases. Science 356 (6334): 183-186. DOI: 10.1126/science.aal4230

Devlin, H. (2017). AI programs exhibit racial and gender biases, research reveals. The Guardian, 13 April 2017, https://www.theguardian.com/technology/2017/apr/13/ai-programs-exhibit-racist-and-sexist-biases-research-reveals.

EUR-Lex (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC. http://eur-lex.europa.eu/legal-content/EN/ALL/?uri=CELEX%3A32016R0679.

Ferson, S., J. O'Rawe, A. Antonenko, J. Siegrist, J. Mickley, C. Luhmann, K. Sentz, and A. Finkel. (2015). Natural language of uncertainty: numeric hedge words. International Journal of Approximate Reasoning 57: 19–39.

Isbell, D., M. Hardin, and J. Underwood (1999). Mars Climate Orbiter team finds likely cause of loss. http://mars.jpl.nasa.gov/msp98/news/mco990930.html.

Kleinman, Z. (2017). Artificial intelligence: How to avoid racist algorithms. BBC News, 14 April 2017, http://www.bbc.co.uk/news/technology-39533308.

Logg, J.M., J.A. Minson and D.A. Moore (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes 151: 90-103.

Manjoo, F. (2017). Can Facebook fix its own worst bug? New York Times, April 25, 2017, https://www.nytimes.com/2017/04/25/magazine/can-facebook-fix-its-own-worst-bug.html?_r=0

MIT Technology Review (2013). Racism is poisoning online ad delivery, says Harvard professor. MIT Technology Review, 4 February 2013, https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/

Panko, R.R. (2000). Spreadsheet risks: what we know. What we think we can do. Proceedings of the Spreadsheet Risk Symposium, European Spreadsheet Risks Interest Group (EuSpRIG), Greenwich, England, 17-18 July 2000.

Prince, E.F., J. Frader, and C. Bock (1982). On hedging in physician−physician discourse. Linguistics and the Professions, R.J. di Pietro (ed.), Ablex Publishing, Norwood, NJ.

Rae, A. (2015). Residential mortgage lending by postcode shows gaps across Britain. The Guardian, UK News, 20 March 2015, https://www.theguardian.com/uk-news/2015/mar/20/residential-mortgage-lending-by-postcode-shows-lending-gaps-across-the-country.

Sandvig, C., K. Hamilton, K. Karahalios, and C. Langbort (2016). When the algorithm itself is a racist: diagnosing ethical harm in the basic components of software. International Journal of Communication 10: 4972–4990, http://ijoc.org/index.php/ijoc/article/view/6182.

Sweeney, L. (2013). Discrimination in online ad delivery. https://arxiv.org/abs/1301.6822.

Woodpile

This is material that may move to the body of the paper. Please feel free to contribute to it, edit it, and—especially—shape it and move it to where it belongs in the body of the paper.

Bill Kahan quotes

"The Adriane 5 disaster was blamed on inadequately tested software. But what I want to know is where is the adequately tested software?"

−William Kahan, at the IFIP Working Conference on Uncertainty Quantification in Scientific Computing, at NIST in Boulder, Colorado

3 August 2011

"In the Airbus disaster, the computer's response to an 'error' event was to abandon control, which caused a stall from which the pilots never recovered. ... Stopping on an execution error is a bad idea. It's a bad policy. The alternative is a bad policy too. ... We need to handle exceptions in a humane way."

−William Kahan, at the IFIP Working Conference on Uncertainty Quantification in Scientific Computing, at NIST in Boulder, Colorado

3 August 2011

Algorithm is not really objective (at best it’s just transparent)

They encode personal opinions and also de facto social racism

Google: ‘black guys’ v. ‘white guys’

Google: ‘hands’

Sweeney’s realization about her name

Should be designed defensively to avoid becoming a tool of racism

Often hard (or painful) to combat de facto racism (e.g., bussing in the US)

De facto is often more insidious (get missing text)

Statistics (a risk-analytic tool) allows us to detect subtle or incipient racism and unfairness

Algorithms mask responsibility

Ronnie’s anecdote about the PM wanting one option rather than multiple ones

Algorithms are appealing to decision makers because they take the decision away

Well, it’s not the decision they don’t like; it’s the responsibility for it that they have

Algorithms allow decision makers to shirk their responsibility

(don’t mind falling; it’s the hitting the ground I don’t like)

There can be heroism can be in bucking the algorithm

Who makes the decision

Risk-based v. risk-informed decision making

Risk-Informed implies using risk assessments as an input to decision-making

Risk-Based implies that risk is the basis for decision-making

If you’re gonna overrule the algorithm if it disagrees with you, why pretend you’re being objective with an algorithm

Tim Barry from EPA ORD says risk analysis has never been the basis of any practical decision

In politics, it’s always horse trading of some kind

In finance, risk analyses are used to support a decision I like, but ignored if they don’t support my decision

Human operator fighting with computer operator was the cause for the airbus forest crash

Paredis’ colleague who spoke at GAFOE on taking pilots out of planes leading to safer flights

Taking the average can hide uncertainty

Averaging conflicting pilot and co-pilot commands led the plane to do nothing in the Air France flight from Brazil

Maybe better to figure out why they are conflicting

Bayesian methods must be applied to human affaires with caution

In some straightforward applications of Bayes’ rule, the use of priors in analysis of human actions and motivations amounts to stereotyping and may constitute improper profiling. <<example; might use the television example of Dr. Green and the disheveled black patient he took to be a crack addict. Even if most of Dr. Green’s patients actually are crack addicts, is it be ethical for him to presume in this way about a new patient?>>

Communicating uncertain information

Should use latest research in psychology and risk communication

Be explicit about sample size (a la EBC’s)

Be explicit about uncertainties (contrast Finkel/RFF ref with Jason’s findings)

Be explicit about age and provenance of information/data

Use natural frequencies for communicating probabilities and proportions

Don’t use percent signs, don’t mention relative risk or conditional risk

Don’t always try to condense multivariate information into a single scalar summary for the sake of simplicity if doing so obscures the tradeoffs of interest

Cannot expect perfect results

Arrow impossibility

Seidenfeld’s rational dictatorship

Privacy

Sweeney/Golle findings

Can use intervalization to anonymize in a way that preserves as much signal as possible consistent with prescribed guarantees about privacy

We have some text from privacy proposals (may need to reword or invite Gang as co-author if we use it)

"[S]aying you don't care about the right to privacy because you have nothing to hide is no different than saying you don't care about freedom of speech because you have nothing to say. It's a fundamentally un-American principle and, more than just nationalism, more than just what this country is about, it's a deeply anti-social principle. Because rights are not just individual; they're collective. What may not have value to you today may have value to an entire population, an entire people, or an entire way of life tomorrow. And if you don't stand up for it, then who will?"

−Edward Snowden, in an interview on Al Jazeera

Possible additional topics:

• 1. an outline of how one might estimate risk of a bad decision per algorithm,

  2. the problem of maintaining user privacy (have text; see above),

  3. the need for ubiquitous uncertainty projection like in Excel and home appliances (have text),

  4. the need for a dubiety campaign to combat indubiety/gullibility (have text), or

  5. the connection to the recent US ban on dangerous "dual-use research of concern" (DURC) in virology.

Don’t think DURC or banned research topic is especially relevant <<is it?>>

Examples where risk science helps:

1. statistical detection of racism

2. tiger example

3. privacy

We would like more!

Modal dialogs and on-line forms that are indeed the boss of you

Would-be patrons of the international science festival Pint of Science are asked when they try to buy tickets to the event a few extra questions in addition to their credit card details. The organisers want this information so that they can better assess the effectiveness of their advertising efforts. Notice that question 3 on their form displayed below is a follow-up to question 2. If the patron answered Yes to question, no answer would be expected to question 3. Yet the bossy form demands an answer anyway, and indeed will not proceed with processing the credit card transaction until all answers are satisfactorily answered.

annoying

delaying

counterproductive