Will impacts of the networked, automated world be mostly positive?
Experts and highly engaged netizens participated in answering a five-question survey fielded by the Imagining the Internet Center and the Pew Internet, Science & Technology Project from July 1-August 12, 2016. This report, issued Feb. 8, 2017, is tied to a survey question asking respondents to share their answer to the following query:
Algorithms will continue to have increasing influence over the next decade, shaping people’s work and personal lives and the ways they interact with information, institutions (banks, health care providers, retailers, governments, education, media and entertainment) and each other. The hope is that algorithms will help people quickly and fairly execute tasks and get the information, products, and services they want. The fear is that algorithms can purposely or inadvertently create discrimination, enable social engineering and have other harmful societal impacts. Will the net overall effect of algorithms be positive for individuals and society or negative for individuals and society? Select from 1)Positives outweigh negatives; 2) Negatives outweigh positives; 3) The overall impact will be about 50-50. Please elaborate on the reasons for your answer.
Among the key themes emerging from 1,302 respondents’ answers in the report released Feb. 8, 2017 were: – Algorithms will continue to spread everywhere. – The benefits, visible and invisible, can lead to greater insight into the world. – The many upsides of algorithms are accompanied by challenges. – Code processes are being refined; ethics and issues are being worked out. – Data-driven approaches achieved through thoughtful design are a plus. – Algorithms don’t have to be perfect; they just have to be better than people. – In the future, the world may be governed by benevolent AI. – Humanity and human agency are lost when data and predictive modeling become paramount. – Programming primarily in pursuit of profits and efficiencies is a threat. – Algorithms manipulate people and outcomes, and even read our minds. – All of this will lead to a flawed yet inescapable logic-driven society. – There will be a loss of complex decision-making capabilities and local intelligence. – Suggested solutions include embedding respect for the individual. – Algorithms reflect the biases of programmers and datasets. – Algorithms depend upon data that is often limited, deficient, or incorrect. – The disadvantaged are likely to be more so. – Algorithms create filter bubbles and silos shaped by corporate data collectors. – Algorithms limit people’s exposure to a wider range of ideas and reliable information and elminate serendipity. – Unemployment numbers will rise as smarter, more-efficient algorithms will take on many work activities. – There is a need for a redefined global economic system to support humanity. – Algorithmic literacy is crucial. – There should be accountability processes, oversight, and transparency. – There is pessimism about the prospects for policy rules and oversight.
This page holds the content of the 87-page survey report. It includes hundreds of respondents’ elaborations – their best estimations of what lies ahead – embedded in an analysis of all 1,302 responses.
To read participants’ for-credit written responses with no analysis, click here.
To read anonymous survey participants’ responses with no analysis, click here.
Summary of Key Findings
Code-Dependent: Algorithm Age Pros and Cons
Algorithms are aimed at optimizing everything. They can save lives, make things easier, and conquer chaos. Still, experts worry they can also put too much control in the hands of corporations and governments, perpetuate bias, create filter bubbles, cut choices, creativity, and serendipity, and could result in greater unemployment.
To illuminate current attitudes about the potential impacts of algorithms in the next decade, Pew Research Center and Elon University’s Imagining the Internet Center conducted a large-scale canvassing of technology experts, scholars, corporate practitioners and government leaders. Respondents to the 2016 Future of the Internet canvassing anticipate that algorithms will expand in their influence over daily life by 2026, with huge implications.
We call this a canvassing because it is not a representative, randomized survey. Its findings emerge from an “opt in” invitation to several thousand people who have been identified by researching those who are widely quoted as technology builders and analysts and those who have made insightful predictions to our previous queries about the future of the Internet.
Some 1,302 experts responded to the following question:
Algorithm Impacts – Algorithms will continue to have increasing influence over the next decade, shaping people’s work and personal lives and the ways they interact with information, institutions (banks, health care providers, retailers, governments, education, media and entertainment) and each other. Will the net overall effect of algorithms be positive for individuals and society or negative for individuals and society? Select from 1) Positives outweigh negatives; 2) Negatives outweigh positives; 3) The overall impact will be about 50-50. Please elaborate on the reasons for your answer.
The non-scientific canvassing found that 38% of these particular respondents predicted that the positive impacts of algorithms will outweigh negatives for individuals and society in general, while 37% said negatives will outweigh positives; 25% said the overall impact of algorithms will be about 50-50, positive-negative. [See the section below titled “About this canvassing of experts” for further details about the limits of this sample.]
Participants were asked to explain their answers, and most wrote detailed elaborations that provide insights about hopeful and concerning trends. Respondents were allowed to respond anonymously; these constitute a slight majority of the written elaborations. These findings do not represent all the points of view that are possible to a question like this, but they do reveal a wide range of valuable observations based on current trends.
Seven key themes found among the responses are illustrated in the following graphic and the massive report that follows it.
In the next section we offer a brief outline of the seven key themes found among the written elaborations. Following that introductory section there is an even-more-in-depth look at additional respondents’ thoughts tied to each of the themes. Some responses are lightly edited for style.
Theme 1: Algorithms will continue to spread everywhere
There is fairly uniform agreement among these respondents that algorithms are generally invisible to the public and there will be an exponential rise in their influence in the next decade. A representative statement of this view came from Barry Chudakov, founder and principal at Sertain Research and StreamFuzion Corp. He replied:
“‘If every algorithm suddenly stopped working, it would be the end of the world as we know it.’ (Pedro Domingo’s The Master Algorithm). Fact: We have already turned our world over to machine learning and algorithms. The question now is, how to better understand and manage what we have done?
“Algorithms are a useful artifact to begin discussing the larger issue of the effects of technology-enabled assists in our lives. Namely, how can we see them at work? Consider and assess their assumptions? And most importantly for those who don’t create algorithms for a living – how do we educate ourselves about the way they work, where they are in operation, what assumptions and biases are inherent in them, and how to keep them transparent? Like fish in a tank, we can see them swimming around and keep an eye on them.
“Algorithms are the new arbiters of human decision-making in almost any area we can imagine, from watching a movie (Affectiva emotion recognition) to buying a house (Zillow.com) to self-driving cars (Google). Deloitte Global predicted more than 80 of the world’s 100 largest enterprise software companies will have cognitive technologies – mediated by algorithms – integrated into their products by the end of 2016. As Brian Christian and Tom Griffiths write in Algorithms to Live By, algorithms provide ‘a better standard against which to compare human cognition itself.’ They are also a goad to consider that same cognition: How are we thinking and what does it mean to think through algorithms to mediate our world?
“The main positive result of this is better understanding of how to make rational decisions, and in this measure a better understanding of ourselves. After all, algorithms are generated by trial and error, by testing, by observing, and coming to certain mathematical formulae regarding choices that have been made again and again – and this can be used for difficult choices and problems, especially when intuitively we cannot readily see an answer or a way to resolve the problem. The 37% Rule, optimal stopping and other algorithmic conclusions are evidence-based guides that enable us to use wisdom and mathematically verified steps to make better decisions.
“The secondary positive result is connectivity. In a technological recapitulation of what spiritual teachers have been saying for centuries, our things are demonstrating that everything is – or can be – connected to everything else. Algorithms with the persistence and ubiquity of insects will automate processes that used to require human manipulation and thinking. These can now manage basic processes of monitoring, measuring, counting or even seeing. Our car can tell us to slow down. Our televisions can suggest movies to watch. A grocery can suggest a healthy combination of meats and vegetables for dinner. Siri reminds you it’s your anniversary.
“The main negative changes come down to a simple but now quite difficult question: How can we see, and fully understand the implications of, the algorithms programmed into everyday actions and decisions? The rub is this: Whose intelligence is it, anyway? … Our systems do not have, and we need to build in, what David Gelernter called ‘topsight,’ the ability to not only create technological solutions but also see and explore their consequences before we build business models, companies and markets on their strengths, and especially on their limitations.”
Chudakov added that this is especially necessary because in the next decade and beyond, “By expanding collection and analysis of data and the resulting application of this information, a layer of intelligence or thinking manipulation is added to processes and objects that previously did not have that layer. So prediction possibilities follow us around like a pet. The result: As information tools and predictive dynamics are more widely adopted, our lives will be increasingly affected by their inherent conclusions and the narratives they spawn.”
“The overall impact of ubiquitous algorithms is presently incalculable because the presence of algorithms in everyday processes and transactions is now so great, and is mostly hidden from public view. All of our extended thinking systems (algorithms fuel the software and connectivity that create extended thinking systems) demand more thinking – not less – and a more global perspective than we have previously managed. The expanding collection and analysis of data and the resulting application of this information can cure diseases, decrease poverty, bring timely solutions to people and places where need is greatest, and dispel millennia of prejudice, ill-founded conclusions, inhumane practice and ignorance of all kinds. Our algorithms are now redefining what we think, how we think and what we know. We need to ask them to think about their thinking – to look out for pitfalls and inherent biases before those are baked in and harder to remove.
“To create oversight that would assess the impact of algorithms, first we need to see and understand them in the context for which they were developed. That, by itself, is a tall order that requires impartial experts backtracking through the technology development process to find the models and formulae that originated the algorithms. Then, keeping all that learning at hand, the experts need to soberly assess the benefits and deficits or risks the algorithms create. Who is prepared to do this? Who has the time, the budget and resources to investigate and recommend useful courses of action? This is a 21st-century job description – and market niche – in search of real people and companies. In order to make algorithms more transparent, products and product information circulars might include an outline of algorithmic assumptions, akin to the nutritional sidebar now found on many packaged food products, that would inform users of how algorithms drive intelligence in a given product and a reasonable outline of the implications inherent in those assumptions.”
Theme 2: Good things lie ahead
A number of respondents noted the many ways in which algorithms will help make sense of massive amounts of data, noting that this will spark breakthroughs in science, new conveniences and human capacities in everyday life, and an ever-better capacity to link people to the information that will help them. They perform seemingly miraculous tasks humans cannot and they will continue to greatly augment human intelligence and assist in accomplishing great things. A representative proponent of this view is Stephen Downes, a researcher at the National Research Council of Canada, who listed the following as positive changes:
“Some examples:
Banks. Today banks provide loans based on very incomplete data. It is true that many people who today qualify for loans would not get them in the future. However, many people – and arguably many more people – will be able to obtain loans in the future, as banks turn away from using such factors as race, socio-economic background, postal code and the like to assess fit. Moreover, with more data (and with a more interactive relationship between bank and client) banks can reduce their risk, thus providing more loans, while at the same time providing a range of services individually directed to actually help a person’s financial state.
“Health care providers. Health care is a significant and growing expense not because people are becoming less healthy (in fact, society-wide, the opposite is true) but because of the significant overhead required to support increasingly complex systems, including prescriptions, insurance, facilities and more. New technologies will enable health providers to shift a significant percentage of that load to the individual, who will (with the aid of personal support systems) manage their health better, coordinate and manage their own care, and create less of a burden on the system. As the overall cost of health care declines, it becomes increasingly feasible to provide single-payer health insurance for the entire population, which has known beneficial health outcomes and efficiencies.
“Governments. A significant proportion of government is based on regulation and monitoring, which will no longer be required with the deployment of automated production and transportation systems, along with sensor networks. This includes many of the daily (and often unpleasant) interactions we have with government today, from traffic offenses, manifestation of civil discontent, unfair treatment in commercial and legal processes, and the like. A simple example: One of the most persistent political problems in the United States is the gerrymandering of political boundaries to benefit incumbents. Electoral divisions created by an algorithm to a large degree eliminate gerrymandering (and when open and debatable, can be modified to improve on that result).”
A sampling of additional examples of positive expectations from anonymous respondents:
• “Algorithms find knowledge in an automated way much faster than traditionally feasible.”
• “Algorithms can crunch databases quickly enough to alleviate some of the red tape and bureaucracy that currently slows progress down.”
• “We will see less pollution, improved human health, less economic waste.”
• “Algorithms have the potential to equalize access to information.”
• “The efficiencies of algorithms will lead to more creativity and self-expression.”
• “Algorithms can diminish transportation issues; they can identify congestion and alternative times and paths.”
• “Self-driving cars could dramatically reduce the number of accidents we have per year, as well as improve quality of life for most people.”
• “Better-targeted delivery of news, services and advertising.”
• “More evidence-based social science using algorithms to collect data from social media and click trails.”
• “Improved and more proactive police work, targeting areas where crime can be prevented.”
• “Fewer underdeveloped areas and more international commercial exchanges.”
• “Algorithms ease the friction in decision-making, purchasing, transportation and a large number of other behaviors.”
• “Bots will follow orders to buy your stocks. Digital agents will find the materials you need.”
• “Any errors could be corrected. This will mean the algorithms only become more efficient to humanity’s desires as time progresses.
Themes illuminating concerns and challenges
Participants in this study were in substantial agreement that the abundant positives of accelerating code-dependency will continue to drive the spread of algorithms; however, as with all great technological revolutions, this trend has a dark side. Most respondents pointed out concerns, chief among them the final five overarching themes of this report; all have subthemes.
Theme 3: Humanity and human agency are lost
when data and predictive modeling become paramount
Advances in algorithms are allowing technology corporations and governments to gather, store, sort and analyze massive data sets. Experts in this canvassing noted that these algorithms are primarily written to optimize efficiency and profitability without much thought about the possible societal impacts of the data modeling and analysis. These respondents argued that humans are considered to be an “input” to the process and they are not seen as real, thinking, feeling, changing beings. They say this is creating a flawed, logic-driven society and that as the process evolves – that is, as algorithms begin to write the algorithms – humans may get left out of the loop, letting “the robots decide.” Representative of this view:
Bart Knijnenburg, assistant professor in human-centered computing at Clemson University, replied, “Algorithms will capitalize on convenience and profit, thereby discriminating certain populations, but also eroding the experience of everyone else. The goal of algorithms is to fit some of our preferences, but not necessarily all of them: They essentially present a caricature of our tastes and preferences. My biggest fear is that, unless we tune our algorithms for self-actualization, it will be simply too convenient for people to follow the advice of an algorithm (or, too difficult to go beyond such advice), turning these algorithms into self-fulfilling prophecies, and users into zombies who exclusively consume easy-to-consume items.”
An anonymous futurist said, “This has been going on since the beginning of the industrial revolution. Every time you design a human system optimized for efficiency or profitability you dehumanize the workforce. That dehumanization has now spread to our health care and social services. When you remove the humanity from a system where people are included, they become victims.”
Another anonymous respondent wrote, “We simply can’t capture every data element that represents the vastness of a person and that person’s needs, wants, hopes, desires. Who is collecting what data points? Do the human beings the data points reflect even know or did they just agree to the terms of service because they had no real choice? Who is making money from the data? How is anyone to know how his/her data is being massaged and for what purposes to justify what ends? There is no transparency, and oversight is a farce. It’s all hidden from view. I will always remain convinced the data will be used to enrich and/or protect others and not the individual. It’s the basic nature of the economic system in which we live.”
A sampling of comments tied to this theme from other respondents (for more thorough details tied to this point read the fuller versions in the full report, below):
• “The potential for good is huge, but the potential for misuse and abuse – intentional, and inadvertent – may be greater.”
• “Companies seek to maximize profit, not maximize societal good. Worse, they repackage profit-seeking as a societal good. We are nearing the crest of a wave, the trough side of which is a new ethics of manipulation, marketing, nearly complete lack of privacy.”
• “What we see already today is that, in practice, stuff like ‘differential pricing’ does not help the consumer; it helps the company that is selling things, etc.”
• “Individual human beings will be herded around like cattle, with predictably destructive results on rule of law, social justice and economics.”
• “There is an incentive only to further obfuscate the presence and operations of algorithmic shaping of communications processes.”
• “Algorithms are … amplifying the negative impacts of data gaps and exclusions.”
• “Algorithms have the capability to shape individuals’ decisions without them even knowing it, giving those who have control of the algorithms an unfair position of power.”
• “The fact the internet can, through algorithms, be used to almost read our minds means [that] those who have access to the algorithms and their databases have a vast opportunity to manipulate large population groups.”
• “The lack of accountability and complete opacity is frightening.”
• “By utilitarian metrics, algorithmic decision-making has no downside; the fact that it results in perpetual injustices toward the very minority classes it creates will be ignored. The Common Good has become a discredited, obsolete relic of The Past.”
• “In an economy increasingly dominated by a tiny, very privileged and insulated portion of the population, it will largely reproduce inequality for their benefit. Criticism will be belittled and dismissed because of the veneer of digital ‘logic’ over the process.”
• “Algorithms are the new gold, and it’s hard to explain why the average ‘good’ is at odds with the individual ‘good.’”
• “We will interpret the negative individual impact as the necessary collateral damage of ‘progress.’”
• “This will kill local intelligence, local skills, minority languages, local entrepreneurship because most of the available resources will be drained out by the global competitors.”
• “Algorithms in the past have been created by a programmer. In the future they will likely be evolved by intelligent/learning machines …. Humans will lose their agency in the world.”
• “It will only get worse because there’s no ‘crisis’ to respond to, and hence, not only no motivation to change, but every reason to keep it going – especially by the powerful interests involved. We are heading for a nightmare.”
• “Web 2.0 provides more convenience for citizens who need to get a ride home, but at the same time – and it’s naive to think this is a coincidence – it’s also a monetized, corporatized, disempowering, cannibalizing harbinger of the End Times. (I exaggerate for effect. But not by much.)”
Theme 4: Biases exist in algorithmically-organized systems,
among programmers and inside datasets
Two strands of thinking tie together here. One is that the algorithm creators (code writers), even if they strive for inclusiveness, objectivity and neutrality, build into their creations their own perspectives and values. The other is that the datasets to which algorithms are applied have their own limits and deficiencies. Even datasets with billions of pieces of information do not capture the fullness of people’s lives and the diversity of their experiences. Moreover, the datasets themselves are imperfect because they do not contain inputs from everyone or a representative sample of everyone. The two themes are advanced in these answers:
Justin Reich, executive director at the MIT Teaching Systems Lab, observed, “The algorithms will be primarily designed by white and Asian men – with data selected by these same privileged actors – for the benefit of consumers like themselves. Most people in positions of privilege will find these new tools convenient, safe and useful. The harms of new technology will be most experienced by those already disadvantaged in society, where advertising algorithms offer bail bondsman ads that assume readers are criminals, loan applications that penalize people for proxies so correlated with race that they effectively penalize people based on race, and similar issues.”
Dudley Irish, a software engineer, observed, “All, let me repeat that, all of the training data contains biases. Much of it either racial- or class-related, with a fair sprinkling of simply punishing people for not using a standard dialect of English. To paraphrase Immanuel Kant, out of the crooked timber of these datasets no straight thing was ever made.”
Following is a sampling of quote excerpts tied to this theme from other respondents (for details, read the fuller versions in the full report, below):
• “Algorithms are, by definition, impersonal and based on gross data and generalized assumptions. The people writing algorithms, even those grounded in data, are a non-representative subset of the population.”
• “If you start at a place of inequality and you use algorithms to decide what is a likely outcome for a person/system, you inevitably reinforce inequalities.”
• “We will all be mistreated as more homogenous than we are.”
• “The result could be the institutionalization of biased and damaging decisions with the excuse of, ‘The computer made the decision, so we have to accept it.’”
• “The algorithms will reflect the biased thinking of people. Garbage in, garbage out. Many dimensions of life will be affected, but few will be helped. Oversight will be very difficult or impossible.”
• “Algorithms value efficiency over correctness or fairness, and over time their evolution will continue the same priorities that initially formulated them.”
• “One of the greatest challenges of the next era will be balancing protection of intellectual property in algorithms with protecting the subjects of those algorithms from unfair discrimination and social engineering.”
• “Algorithms purport to be fair, rational and unbiased but just enforce prejudices with no recourse.”
• “Unless the algorithms are essentially open source and as such can be modified by user feedback in some fair fashion, the power that likely algorithm-producers (corporations and governments) have to make choices favorable to themselves, whether in internet terms of service or adhesion contracts or political biases, will inject both conscious and unconscious bias into algorithms.”
Theme 5: Algorithmic categorizations deepen divides
Two connected ideas about societal divisions were evident in many respondents’ answers. First, they predicted that an algorithm-assisted future will widen the gap between the digitally savvy (predominantly the most well-off, who are the most desired demographic in the new information ecosystem) and those who are not nearly as connected or able to participate. Second, they said social and political divisions will be abetted by algorithms, as algorithm-driven categorizations and classifications steer people into echo chambers of repeated and reinforced media and political content. Two illustrative answers:
Ryan Hayes, owner of Fit to Tweet, commented, “Twenty years ago we talked about the ‘digital divide’ being people who had access to a computer at home vs. those that didn’t, or those who had access to the internet vs. those who didn’t …. Ten years from now, though, the life of someone whose capabilities and perception of the world is augmented by sensors and processed with powerful AI and connected to vast amounts of data is going to be vastly different from that of those who don’t have access to those tools or knowledge of how to utilize them. And that divide will be self-perpetuating, where those with fewer capabilities will be more vulnerable in many ways to those with more.”
Adam Gismondi, a visiting scholar at Boston College, wrote, “I am fearful that as users are quarantined into distinct ideological areas, human capacity for empathy may suffer. Brushing up against contrasting viewpoints challenges us, and if we are able to (actively or passively) avoid others with different perspectives, it will negatively impact our society. It will be telling to see what features our major social media companies add in coming years, as they will have tremendous power over the structure of information flow.”
Following is a sampling of quote excerpts tied to this theme from other respondents (for details, read the fuller versions in the full report, below):
• “If the current economic order remains in place, then I do not see the growth of data-driven algorithms providing much benefit to anyone outside of the richest in society.”
• “Social inequalities will presumably become reified.”
• “The major risk is that less-regular users, especially those who cluster on one or two sites or platforms, won’t develop that navigational and selection facility and will be at a disadvantage.”
• “Algorithms make discrimination more efficient and sanitized. Positive impact will be increased profits for organizations able to avoid risk and costs. Negative impacts will be carried by all deemed by algorithms to be risky or less profitable.”
• “Society will be stratified by which trust/identity provider one can afford/qualify to go with. The level of privacy and protection will vary. Lois McMaster [Bujold]’s Jackson’s Whole suddenly seems a little more chillingly realistic.”
• “We have radically divergent sets of values, political and other, and algos are always rooted in the value systems of their creators. So the scenario is one of a vast opening of opportunity – economic and otherwise – under the control of either the likes of Zuckerberg or the grey-haired movers of global capital or ….”
• “The overall effect will be positive for some individuals. It will be negative for the poor and the uneducated. As a result, the digital divide and wealth disparity will grow. It will be a net negative for society.”
• “Racial exclusion in consumer targeting. Gendered exclusion in consumer targeting. Class exclusion in consumer targeting …. Nationalistic exclusion in consumer targeting.”
• “If the algorithms directing news flow suppress contradictory information – information that challenges the assumptions and values of individuals – we may see increasing extremes of separation in worldviews among rapidly diverging subpopulations.”
• “We may be heading for lowest-common-denominator information flows.”
• “Efficiency and the pleasantness and serotonin that come from prescriptive order are highly overrated. Keeping some chaos in our lives is important.”
A number of participants in this canvassing expressed concerns over the change in the public’s information diets, the “atomization of media,” an over-emphasis of the extreme, ugly, weird news, and the favoring of “truthiness” over more-factual material that may be vital to understanding how to be a responsible citizen of the world.
Theme 6: Unemployment numbers will rise
The spread of artificial intelligence (AI) has the potential to create major unemployment and all the fallout from that.
An anonymous CEO said, “If a task can be effectively represented by an algorithm, then it can be easily performed by a machine. The negative trend I see here is that – with the rise of the algorithm – humans will be replaced by machines/computers for many jobs/tasks. What will then be the fate of Man?”
A sampling of quote excerpts tied to this theme from other respondents (for details, read the fuller versions in the full report, below):
• “AI and robots are likely to disrupt the workforce to a potential 100% human unemployment. They will be smarter more efficient and productive and cost less, so it makes sense for corporations and business to move in this direction.”
• “The massive boosts in productivity due to automation will increase the disparity between workers and owners of capital.”
• “Modern Western society is built on a societal model whereby Capital is exchanged for Labour to provide economic growth. If Labour is no longer part of that exchange, the ramifications will be immense.”
• “No jobs, growing population and less need for the average person to function autonomously. Which part of this is warm and fuzzy?”
• “I foresee algorithms replacing almost all workers with no real options for the replaced humans.”
• “In the long run, it could be a good thing for individuals by doing away with low-value repetitive tasks and motivating them to perform ones that create higher value.”
• “Hopefully, countries will have responded by implementing forms of minimal guaranteed living wages and free education past K-12; otherwise the brightest will use online resources to rapidly surpass average individuals and the wealthiest will use their economic power to gain more political advantages.”
Theme 7: The need grows for algorithmic literacy, transparency, and oversight
The respondents to this canvassing offered a variety of ideas about how individuals and the broader culture might respond to the algorithm-ization of life. They argued for public education to instill literacy about how algorithms function in the general public. They also noted that those who create and evolve algorithms are not held accountable to society and argued there should be some method by which they are. Representative comments:
Susan Etlinger, industry analyst at Altimeter Group, said, “Much like the way we increasingly wish to know the place and under what conditions our food and clothing are made, we should question how our data and decisions are made as well. What is the supply chain for that information? Is there clear stewardship and an audit trail? Were the assumptions based on partial information, flawed sources or irrelevant benchmarks? Did we train our data sufficiently? Were the right stakeholders involved, and did we learn from our mistakes? The upshot of all of this is that our entire way of managing organizations will be upended in the next decade. The power to create and change reality will reside in technology that only a few truly understand. So to ensure that we use algorithms successfully, whether for financial or human benefit or both, we need to have governance and accountability structures in place. Easier said than done, but if there were ever a time to bring the smartest minds in industry together with the smartest minds in academia to solve this problem, this is the time.”
Chris Kutarna, author of Age of Discovery and fellow at the Oxford Martin School, wrote, “Algorithms are an explicit form of heuristic, a way of routinizing certain choices and decisions so that we are not constantly drinking from a fire hydrant of sensory inputs. That coping strategy has always been co-evolving with humanity, and with the complexity of our social systems and data environments. Becoming explicitly aware of our simplifying assumptions and heuristics is an important site at which our intellects and influence mature. What is different now is the increasing power to program these heuristics explicitly, to perform the simplification outside of the human mind and within the machines and platforms that deliver data to billions of individual lives. It will take us some time to develop the wisdom and the ethics to understand and direct this power. In the meantime, we honestly don’t know how well or safely it is being applied. The first and most important step is to develop better social awareness of who, how, and where it is being applied.”
A sampling of quote excerpts tied to this theme from other respondents (for details, read the fuller versions in the full report, below):
• “Who guards the guardians? And, in particular, which ‘guardians’ are doing what, to whom, using the vast collection of information?”
• “There are no incentives in capitalism to fight filter bubbles, profiling, and the negative effects, and governmental/international governance is virtually powerless.”
• “Oversight mechanisms might include stricter access protocols; sign off on ethical codes for digital management and named stewards of information; online tracking of an individual’s reuse of information; opt-out functions; setting timelines on access; no third-party sale without consent.”
• “Unless there is an increased effort to make true information literacy a part of basic education, there will be a class of people who can use algorithms and a class used by algorithms.”
• “Consumers have to be informed, educated, and, indeed, activist in their orientation toward something subtle. This is what computer literacy is about in the 21st century.”
• “Finding a framework to allow for transparency and assess outcomes will be crucial. Also a need to have a broad understanding of the algorithmic ‘value chain’ and that data is the key driver and as valuable as the algorithm which it trains.”
• “Algorithmic accountability is a big-tent project, requiring the skills of theorists and practitioners, lawyers, social scientists, journalists, and others. It’s an urgent, global cause with committed and mobilized experts looking for support.”
• “Eventually, software liability law will be recognized to be in need of reform, since right now, literally, coders can get away with murder.”
• “The Law of Unintended Consequences indicates that the increasing layers of societal and technical complexity encoded in algorithms ensure that unforeseen catastrophic events will occur – probably not the ones we were worrying about.”
• “Eventually we will evolve mechanisms to give consumers greater control that should result in greater understanding and trust …. The pushback will be inevitable but necessary and will, in the long run, result in balances that are more beneficial for all of us.”
• “We need some kind of rainbow coalition to come up with rules to avoid allowing inbuilt bias and groupthink to effect the outcomes.”
• “Algorithms are too complicated to ever be transparent or to ever be completely safe. These factors will continue to influence the direction of our culture.”
• “I expect meta-algorithms will be developed to try to counter the negatives of algorithms.”
Anonymous respondents shared these one-liners on the topic:
• “The golden rule: He who owns the gold makes the rules.”
• “The bad guys appear to be way ahead of the good guys.”
• “Resistance is futile.”
• “Algorithms are defined by people who want to sell you something (goods, services, ideologies) and will twist the results to favor doing so.”
• “Algorithms are surely helpful but likely insufficient unless combined with human knowledge and political will.”
Finally, this prediction from an anonymous participant who sees the likely endpoint to be one of two extremes:
“The overall impact will be utopia or the end of the human race; there is no middle ground foreseeable. I suspect utopia given that we have survived at least one existential crisis (nuclear) in the past and that our track record toward peace, although slow, is solid.”
Key experts’ thinking about the future impacts of algorithms
Following is a brief collection of comments by several of the many top analysts who participated in this canvassing:
‘Steering people to useful information’
Vinton Cerf, Internet Hall of Fame member and vice president and chief internet evangelist at Google: “Algorithms are mostly intended to steer people to useful information and I see this as a net positive.”
Beware ‘unverified, untracked, unrefined models’
Cory Doctorow, writer, computer science activist-in-residence at MIT Media Lab and co-owner of Boing Boing, responded, “The choices in this question are too limited. The right answer is, ‘If we use machine learning models rigorously, they will make things better; if we use them to paper over injustice with the veneer of machine empiricism, it will be worse.’ Amazon uses machine learning to optimize its sales strategies. When they make a change, they make a prediction about its likely outcome on sales, then they use sales data from that prediction to refine the model. Predictive sentencing scoring contractors to America’s prison system use machine learning to optimize sentencing recommendation. Their model also makes predictions about likely outcomes (on reoffending), but there is no tracking of whether their model makes good predictions, and no refinement. This frees them to make terrible predictions without consequence. This characteristic of unverified, untracked, unrefined models is present in many places: terrorist watchlists; drone-killing profiling models; modern redlining/Jim Crow systems that limit credit; predictive policing algorithms; etc. If we mandate, or establish normative limits, on practices that correct this sleazy conduct, then we can use empiricism to correct for bias and improve the fairness and impartiality of firms and the state (and public/private partnerships). If, on the other hand, the practice continues as is, it terminates with a kind of Kafkaesque nightmare where we do things ‘because the computer says so’ and we call them fair ‘because the computer says so.’”
‘A general trend toward positive outcomes will prevail’
Jonathan Grudin, principal researcher at Microsoft, said, “We are finally reaching a state of symbiosis or partnership with technology. The algorithms are not in control; people create and adjust them. However, positive effects for one person can be negative for another, and tracing causes and effects can be difficult, so we will have to continually work to understand and adjust the balance. Ultimately, most key decisions will be political, and I’m optimistic that a general trend toward positive outcomes will prevail, given the tremendous potential upside to technology use. I’m less worried about bad actors prevailing than I am about unintended and unnoticed negative consequences sneaking up on us.”
‘Faceless systems more interested in surveillance and advertising than actual service’
Doc Searls, journalist, speaker and director of Project VRM at Harvard University’s Berkman Center, wrote, “The biggest issue with algorithms today is the black-box nature of some of the largest and most consequential ones. An example is the one used by Dun & Bradstreet to decide credit worthiness. The methods behind the decisions it makes are completely opaque, not only to those whose credit is judged, but to most of the people running the algorithm as well. Only the programmers are in a position to know for sure what the algorithm does, and even they might not be clear about what’s going on. In some cases there is no way to tell exactly why or how a decision by an algorithm is reached. And even if the responsible parties do know exactly how the algorithm works, they will call it a trade secret and keep it hidden. There is already pushback against the opacity of algorithms, and the sometimes vast systems behind them. Many lawmakers and regulators also want to see, for example, Google’s and Facebook’s vast server farms more deeply known and understood. These things have the size, scale, and in some ways the importance of nuclear power plants and oil refineries, yet enjoy almost no regulatory oversight. This will change. At the same time, so will the size of the entities using algorithms. They will get smaller and more numerous, as more responsibility over individual lives moves away from faceless systems more interested in surveillance and advertising than actual service.”
A call for #AlgorithmicTransparency
Marc Rotenberg, executive director of the Electronic Privacy Information Center, observed, “The core problem with algorithmic-based decision making is the lack of accountability. Machines have literally become black boxes – even the developers and operators do not fully understand how outputs are produced. The problem is further exacerbated by ‘digital scientism’ (my phrase) – an unwavering faith in the reliability of big data. ‘Algorithmic transparency’ should be established as a fundamental requirement for all AI-based decision-making. There is a larger problem with the increase of algorithm-based outcomes beyond the risk of error or discrimination – the increasing opacity of decision-making and the growing lack of human accountability. We need to confront the reality that power and authority are moving from people to machines. That is why #AlgorithmicTransparency is one of the great challenges of our era.”
The data ‘will be misused in various ways’
Richard Stallman, Internet Hall of Fame member and president of the Free Software Foundation, said, “People will be pressured to hand over all the personal data that the algorithms would judge. The data, once accumulated, will be misused in various ways – by the companies that collect them, by rogue employees, by crackers that steal the data from the company’s site, and by the state via National Security Letters. I have heard that people who refuse to be used by Facebook are discriminated against in some ways. Perhaps soon they will be denied entry to the U.S., for instance. Even if the U.S. doesn’t actually do that, people will fear that it will. Compare this with China’s social obedience score for internet users.”
People must live with outcomes of algorithms ‘even though they are fearful of the risks’
David Clark, Internet Hall of Fame member and senior research scientist at MIT, replied, “I see the positive outcomes outweighing the negative, but the issue will be that certain people will suffer negative consequences, perhaps very serious, and society will have to decide how deal with these outcomes. These outcomes will probably differ in character, and in our ability to understand why they happened, and this reality will make some people fearful. But as we see today that people feel that they must use the internet to be a part of society, even if they are fearful of the consequences, people will accept that they must live with the outcomes of these algorithms, even though they are fearful of the risks.”
‘EVERY area of life will be affected. Every. Single. One.’
Baratunde Thurston, Director’s Fellow at MIT Media Lab, Fast Company columnist, and former digital director of The Onion, wrote: “”Main positive changes: 1) The excuse of not knowing things will be reduced greatly as information becomes even more connected and complete. 2) Mistakes that result from errors in human judgment, ‘knowledge,’ or reaction time will be greatly reduced. Let’s call this the ‘robots drive better than people’ principle. Today’s drivers will whine, but in 50 years no one will want to drive when they can use that transportation time to experience a reality-indistinguishable immersive virtual environment filled with a bunch of Beyonce bots. 3) Corruption that exists today as a result of human deception will decline significantly—bribes, graft, nepotism. If the algorithms are built well and robustly, the opportunity to insert this inefficiency (e.g., hiring some idiot because he’s your cousin) should go down. 4) In general, we should achieve a much more efficient distribution of resources, including expensive (in dollars or environmental cost) resources like fossil fuels. Basically, algorithmic insight will start to affect the design of our homes, cities, transportation networks, manufacturing levels, waste management processing, and more. There’s a lot of redundancy in a world where every American has a car she never uses. We should become far more energy efficient once we reduce the redundancy of human-drafted processes.
“But there will be negative changes,” he continued “1) There will be an increased speed of interactions and volume of information processed—everything will get faster. None of the efficiency gains brought about by technology has ever lead to more leisure or rest or happiness. We will simply shop more, work more, decide more things because our capacity to do all those will have increased. It’s like adding lanes to the highway as a traffic management solution. When you do that, you just encourage more people to drive. The real trick is to not add more car lanes but build a world in which fewer people need or want to drive.
“2) There will be algorithmic and data-centric oppression. Given that these systems will be designed by demonstrably imperfect and biased human beings, we are likely to create new and far less visible forms of discrimination and oppression. The makers of these algorithms and the collectors of the data used to test and prime them have nowhere near a comprehensive understanding of culture, values, and diversity. They will forget to test their image recognition on dark skin or their medical diagnostic tools on Asian women or their transport models during major sporting events under heavy fog. We will assume the machines are smarter, but we will realize they are just as dumb as we are but better at hiding it.
“3) Entire groups of people will be excluded and they most likely won’t know about the parallel reality they don’t experience.
“Every area of life will be affected. Every. Single. One.”
A call for ‘industry reform’ and ‘more savvy regulatory regimes’
Technologist Anil Dash said, “The best parts of algorithmic influence will make life better for many people, but the worst excesses will truly harm the most marginalized in unpredictable ways. We’ll need both industry reform within the technology companies creating these systems and far more savvy regulatory regimes to handle the complex challenges that arise.”
‘We are a society that takes its life direction from the palm of our hands’
John Markoff, author of Machines of Loving Grace: The Quest for Common Ground Between Humans and Robots and senior writer at The New York Times, observed, “I am most concerned about the lack of algorithmic transparency. Increasingly we are a society that takes its life direction from the palm of our hands – our smartphones. Guidance on everything from what is the best Korean BBQ to who to pick for a spouse is algorithmically generated. There is little insight, however, into the values and motives of the designers of these systems.”
Fix the ‘organizational, societal and political climate we’ve constructed’
danah boyd, founder of Data & Society, commented, “An algorithm means nothing by itself. What’s at stake is how a ‘model’ is created and used. A model is comprised of a set of data (e.g., training data in a machine learning system) alongside an algorithm. The algorithm is nothing without the data. But the model is also nothing without the use case. The same technology can be used to empower people (e.g., identify people at risk) as harm them. It all depends on who is using the information to what ends (e.g., social services vs. police). Because of unhealthy power dynamics in our society, I sadly suspect that the outcomes will be far more problematic – mechanisms to limit people’s opportunities, segment and segregate people into unequal buckets, and leverage surveillance to force people into more oppressive situations. But it doesn’t have to be that way. What’s at stake has little to do with the technology; it has everything to do with the organizational, societal and political climate we’ve constructed.”
We have an algorithmic problem already: Credit scores
Henning Schulzrinne, Internet Hall of Fame member and professor at Columbia University, noted, “We already have had early indicators of the difficulties with algorithmic decision-making, namely credit scores. Their computation is opaque and they were then used for all kinds of purposes far removed from making loans, such as employment decisions or segmenting customers for different treatment. They leak lots of private information and are disclosed, by intent or negligence, to entities that do not act in the best interest of the consumer. Correcting data is difficult and time-consuming, and thus unlikely to be available to individuals with limited resources. It is unclear how the proposed algorithms address these well-known problems, given that they are often subject to no regulations whatsoever. In many areas, the input variables are either crude (and often proxies for race), such as home ZIP code, or extremely invasive, such as monitoring driving behavior minute-by-minute. Given the absence of privacy laws, in general, there is every incentive for entities that can observe our behavior, such as advertising brokers, to monetize behavioral information. At minimum, institutions that have broad societal impact would need to disclose the input variables used, how they influence the outcome and be subject to review, not just individual record corrections. An honest, verifiable cost-benefit analysis, measuring improved efficiency or better outcomes against the loss of privacy or inadvertent discrimination, would avoid the ‘trust us, it will be wonderful and it’s AI!’ decision-making.”
Algorithms ‘create value and cut costs’ and will be improved
Robert Atkinson, president of the Information Technology and Innovation Foundation, said, “Like virtually all past technologies, algorithms will create value and cut costs, far in excess of any costs. Moreover, as organizations and society get more experience with use of algorithms there will be natural forces toward improvement and limiting any potential problems.”
‘The goal should be to help people question authority’
Judith Donath of Harvard Berkman Klein Center for Internet & Society, replied, “Data can be incomplete, or wrong, and algorithms can embed false assumptions. The danger in increased reliance on algorithms is that is that the decision-making process becomes oracular: opaque yet unarguable. The solution is design. The process should not be a black box into which we feed data and out comes an answer, but a transparent process designed not just to produce a result, but to explain how it came up with that result. The systems should be able to produce clear, legible text and graphics that help the users – readers, editors, doctors, patients, loan applicants, voters, etc. – understand how the decision was made. The systems should be interactive, so that people can examine how changing data, assumptions, rules would change outcomes. The algorithm should not be the new authority; the goal should be to help people question authority.”
Do more to train coders with diverse world views
Amy Webb, futurist and CEO at the Future Today Institute, wrote, “In order to make our machines think, we humans need to help them learn. Along with other pre-programmed training datasets, our personal data is being used to help machines make decisions. However, there are no standard ethical requirements or mandate for diversity, and as a result we’re already starting to see a more dystopian future unfold in the present. There are too many examples to cite, but I’ll list a few: would-be borrowers turned away from banks, individuals with black-identifying names seeing themselves in advertisements for criminal background searches, people being denied insurance and health care. Most of the time, these problems arise from a limited worldview, not because coders are inherently racist. Algorithms have a nasty habit of doing exactly what we tell them to do. Now, what happens when we’ve instructed our machines to learn from us? And to begin making decisions on their own? The only way to address algorithmic discrimination in the future is to invest in the present. The overwhelming majority of coders are white and male. Corporations must do more than publish transparency reports about their staff – they must actively invest in women and people of color, who will soon be the next generation of workers. And when the day comes, they must choose new hires both for their skills and their worldview. Universities must redouble their efforts not only to recruit a diverse body of students –administrators and faculty must support them through to graduation. And not just students. Universities must diversify their faculties, to ensure that students see themselves reflected in their teachers.”
The impact in the short term will be negative; in the longer term it will be positive
Jamais Cascio, distinguished fellow at the Institute for the Future, observed, “The impact of algorithms in the early transition era will be overall negative, as we (humans, human society and economy) attempt to learn how to integrate these technologies. Bias, error, corruption and more will make the implementation of algorithmic systems brittle, and make exploiting those failures for malice, political power or lulz comparatively easy. By the time the transition takes hold – probably a good 20 years, maybe a bit less – many of those problems will be overcome, and the ancillary adaptations (e.g., potential rise of universal basic income) will start to have an overall benefit. In other words, shorter term (this decade) negative, longer term (next decade) positive.”
The story will keep shifting
Mike Liebhold, senior researcher and distinguished fellow at the Institute for the Future, commented, “The future effects of algorithms in our lives will shift over time as we master new competencies. The rates of adoption and diffusion will be highly uneven, based on natural variables of geographies, the environment, economies, infrastructure, policies, sociologies, psychology, and – most importantly – education. The growth of human benefits of machine intelligence will be most constrained by our collective competencies to design and interact effectively with machines. At an absolute minimum, we need to learn to form effective questions and tasks for machines, how to interpret responses and how to simply detect and repair a machine mistake.”
Make algorithms ‘comprehensible, predictable and controllable’
Ben Shneiderman, professor of computer science at the University of Maryland, wrote, “When well-designed, algorithms amplify human abilities, but they must be comprehensible, predictable and controllable. This means they must be designed to be transparent so that users can understand the impacts of their use and they must be subject to continuing evaluation so that critics can assess bias and errors. Every system needs a responsible contact person/organization that maintains/updates the algorithm and a social structure so that the community of users can discuss their experiences.”
In key cases, give the user control
David Weinberger, senior researcher at the Harvard Berkman Klein Center for Internet & Society, said, “Algorithmic analysis at scale can turn up relationships that are predictive and helpful even if they are beyond the human capacity to understand them. This is fine where the stakes are low, such as a book recommendation. Where the stakes are high, such as algorithmically filtering a news feed, we need to be far more careful, especially when the incentives for the creators are not aligned with the interests of the individuals or of the broader social goods. In those latter cases, giving more control to the user seems highly advisable.”
Fuller Details:
Predictions for Algorithm Impacts by 2026
Algorithms are instructions for solving a problem or completing a task. Recipes are algorithms. Computer code is algorithmic. The internet runs on algorithms and all online searching is accomplished through them. Email knows where to go thanks to algorithms. Smartphone apps are nothing but algorithms. Computer and video games are algorithmic storytelling. Online dating and book-recommendation and travel websites would not function without algorithms. GPS mapping systems get people from point A to point B via algorithms. Artificial intelligence (AI) is naught but algorithms. The material people see on social media is brought to them by algorithms. In fact, everything people see and do on the web is a product of algorithms. Every time someone sorts a column in a spreadsheet, algorithms are at play, and most financial transactions today are accomplished by algorithms. Algorithms help gadgets respond to voice commands, recognize faces, sort photos and build and drive cars. Hacking, cyberattacks and cryptographic code-breaking exploit algorithms. Self-learning and self-programming algorithms are now emerging, so it is possible that in the future algorithms will write many if not most algorithms.
Algorithms are often elegant and incredibly useful tools used to accomplish tasks. They are mostly invisible aids, augmenting human lives in increasingly incredible ways. However, sometimes the application of algorithms created with good intentions leads to unintended consequences. Recent news items tie to these concerns:
• The British pound dropped 6.1% in value in seconds on Oct. 7, 2016, partly because of currency trades triggered by algorithms.
• Microsoft engineers created a Twitter bot named “Tay” in the spring of 2016 in an attempt to chat with Millennials by responding to their prompts, but within hours it was spouting racist, sexist, Holocaust-denying tweets based on algorithms that had it “learning” how to respond to others based on what was tweeted at it.
• Facebook tried to create a feature to highlight Trending Topics from around the site in people’s feeds. First, it had a team of humans edit the feature, but controversy erupted when some accused the platform of being biased against conservatives. So, Facebook then turned the job over to algorithms only to find that they could not discern real news from fake news.
• Cathy O’Neil, author of Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, pointed out that predictive analytics based on algorithms tend to punish the poor, using algorithmic hiring practices as an example.
• Well-intentioned algorithms can be sabotaged by bad actors. An internet slowdown swept the East Coast of the U.S. on Oct. 21, 2016, after hackers bombarded Dyn DNS, an internet traffic handler, with information that overloaded its circuits, ushering in a new era of internet attacks powered by internet-connected devices. This after internet security expert Bruce Schneier warned in September that “Someone Is Learning How to Take Down the Internet.” And the abuse of Facebook’s News Feed algorithm and general promulgation of fake news online became controversial as the 2016 U.S. presidential election proceeded.
• Researcher Andrew Tutt called for an “FDA for Algorithms,” noting, “The rise of increasingly complex algorithms calls for critical thought about how to best prevent, deter and compensate for the harms that they cause …. Algorithmic regulation will require federal uniformity, expert judgment, political independence and pre-market review to prevent – without stifling innovation – the introduction of unacceptably dangerous algorithms into the market.”
• The White House released two reports in October 2016, one detailing the advance of algorithms and artificial intelligence and another with plans to address issues tied to it, and it issued a December report outlining some of the potential effects of AI-driven automation on the U.S. job market and economy.
• On January 17, 2017, the Future of Life Institute published a list of 23 Principles for Beneficial Artificial Intelligence. The more than 1,600 signatories included Steven Hawking, Elon Musk, Ray Kurzweil and hundreds of the world’s foremost AI researchers and more than 1,600 other endorsers.
The use of algorithms is spreading as massive amounts of data are being created, captured and analyzed by businesses and governments. Some are calling this the Age of Algorithms and predicting that the future of algorithms is tied to machine learning and deep learning that will get better and better at an ever-faster pace.
While many of the 2016 U.S. presidential election post-mortems noted the revolutionary impact of web-based tools in influencing its outcome, XPrize Foundation CEO Peter Diamandis predicted that “five big tech trends will make this election look tame.” He said advances in quantum computing and the rapid evolution of AI and AI agents embedded in systems and devices in the Internet of Things will lead to hyper-stalking, influencing and shaping of voters, and hyper-personalized ads, and will create new ways to misrepresent reality and perpetuate falsehoods.
Analysts like Aneesh Aneesh of Stanford University foresee algorithms taking over public and private activities in a new era of “algocratic governance” that supplants “bureaucratic hierarchies.” Others, like Harvard’s Shoshana Zuboff, describe the emergence of “surveillance capitalism” that organizes economic behavior in an “information civilization.”
The sizeable majority of experts surveyed for this report envision major advances in programmed artificial intelligence and other algorithmic advances in the coming decade.
Following are some themes that emerged.
Theme 1: Algorithms will continue to spread everywhere
Nearly all of these respondents see the great advantages of the algorithms that are already changing how connected institutions and people live and work. A significant majority expects them to continue to proliferate – mostly invisibly – and expects that there will be an exponential rise in their influence. They say this will bring many benefits and some challenges.
Jim Warren, longtime technology entrepreneur and activist, described algorithms: “Any sequence of instructions for how to do something (or how a machine that can understand said instructions can do it) is – by definition – an ‘algorithm.’ All sides – great and small, benevolent and malevolent – have always created and exercised such algorithms (recipes for accomplishing a desired function), and always will. Almost all of the ‘good’ that humankind has created – as well as all the harm (sometimes only in the eye of the beholder) – has been from discovering how to do something, and then repeating that process. And more often than not, sharing it with others. Like all-powerful but double-edged tools, algorithms are. ;-)”
Terry Langendoen, a U.S. National Science Foundation expert whose job is to support research on algorithms, is enthusiastic about what lies ahead. “The technological improvements in the past 50 years in such areas as speech-recognition and synthesis, machine translation and information retrieval have had profound beneficial impacts …,” he said. “The field is poised to make significant advances in the near future.”
The benefits will be visible and invisible and
can lead to greater human insight into the world
Patrick Tucker, author and technology editor at Defense One, pointed out how today’s networked communications amplify the impacts of algorithms. “The internet is turning prediction into an equation,” he commented. “From programs that chart potential flu outbreaks to expensive (yet imperfect) ‘quant’ algorithms that anticipate bursts of stock market volatility, computer-aided prediction is everywhere. As I write in The Naked Future, in the next two decades, as a function of machine learning and big data, we will be able to predict huge areas of the future with far greater accuracy than ever before in human history, including events long thought to be beyond the realm of human inference. That will have an impact in all areas including health care, consumer choice, educational opportunities, etc. The rate by which we can extrapolate meaningful patterns from the data of the present is quickening as rapidly as is the spread of the internet because the two are inexorably linked.”
Paul Jones, clinical professor at the University of North Carolina-Chapel Hill and director of ibiblio.org, was optimistic. “The promise of standardization of best practices into code is a promise of stronger best practices and a hope of larger space for human insight,” he predicted. “Code, flexible and open code, can make you free – or at least a bit freer.”
David Krieger, director of the Institute for Communication & Leadership IKF, predicted, “Data-driven algorithmic cognition and agency will characterize all aspects of society. Humans and non-humans will become partners such that identity(ies) will be distributed and collective. Individualism will become anachronistic. The network is the actor. It is the network that learns, produces, decides, much like the family or clan in collective societies of the past, but now on the basis of big data, AI and transparency. Algorithmic auditing, accountability, benchmarking procedures in machine learning, etc., will play an important role in network governance frameworks that will replace hierarchical, bureaucratic government. Not government, but governance.”
An anonymous software security consultant noted, “There will be many positive impacts that aren’t even noticed. Having an ‘intelligent’ routing system for cars may mean most people won’t notice when everyone gets to their destination as fast as they used to even with twice the traffic. Automated decisions will indeed have significant impacts upon lots of people, most of the time in ways they won’t ever recognize. Already they’re being used heavily in financial situations, but most people don’t see a significant difference between ‘a VP at the bank denied my loan’ and ‘software at the bank denied my loan’ (and in practice, the main difference is an inability to appeal the decision).”
Another anonymous respondent wrote, “Algorithms in general enable people to benefit from the results of the synthesis of large volumes of information where such synthesis was not available in any form before – or at least only to those with significant resources. This will be increasingly positive in terms of enabling better-informed choices. As algorithms scale and become more complex, unintended consequences become harder to predict and harder to fix if they are detected, but the positive benefit above seems so dramatic it should outweigh this effect. Particularly if there are algorithms designed to detect unintended discriminatory or other consequences of other algorithms.”
The many upsides of algorithms are accompanied by serious challenges
Respondents often hailed the positives while noting the need to address the downsides.
Galen Hunt, partner research manager at Microsoft Research NExT, reflected the hopes of many when he wrote, “Algorithms will accelerate in their impact on society. If we guard the core values of civil society (like equality, respect, transparency), the most valuable algorithms will be those that help the greatest numbers of people.”
Alf Rehn, professor and chair of management and organization at Åbo Akademi University in Finland, commented, “New algorithmic thinking will be a great boon for many people. They will make life easier, shopping less arduous, banking a breeze and a hundred other great things besides. But a shaved monkey can see the upsides. The important thing is to realize the threats, major and minor, of a world run by algorithms. They can enhance filter bubbles for both individuals and companies, limit our view of the world, create more passive consumers, and create a new kind of segregation – think algorithmic haves and have-nots. In addition, for an old hacker like me, as algorithmic logics get more and more prevalent in more and more places, they also increase the number of attack vectors for people who want to pervert their logic, for profit, for more nefarious purposes, or just for the lulz.”
Andrew Nachison, founder at We Media, observed, “The positives will be enormous – better shopping experiences, better medical experience, even better experiences with government agencies. Algorithms could even make ‘bureaucrat’ a friendlier word. But the dark sides of the ‘optimized’ culture will be profound, obscure and difficult to regulate – including pervasive surveillance of individuals and predictive analytics that will do some people great harm (‘Sorry, you’re pre-disqualified from a loan.’ ‘Sorry, we’re unable to sell you a train ticket at this time.’). Advances in computing, tracking and embedded technology will herald a quantified culture that will be ever more efficient, magical and terrifying.”
Luis Lach, president of the Sociedad Mexicana de Computación en la Educación, A.C., said, “On the negative side we will see huge threats to security, data privacy and attacks to individuals, by governments, private entities and other social actors. And on the positive we will have the huge opportunity for collective and massive collaboration across the entire planet. Of course the science will rise and we will see marvelous advances. Of course we will have a continuum between positive and negative scenarios. What we will do depends on individuals, governments, private companies, nonprofits, academia, etc.”
Frank Pasquale, author of The Black Box Society: The Secret Algorithms That Control Money and Information and professor of law at the University of Maryland, wrote, “Algorithms are increasingly important because businesses rarely thought of as high-tech have learned the lessons of the internet giants’ successes. Following the advice of Jeff Jarvis’ What Would Google Do, they are collecting data from both workers and customers, using algorithmic tools to make decisions, to sort the desirable from the disposable. Companies may be parsing your voice and credit record when you call them, to determine whether you match up to ‘ideal customer’ status, or are simply ‘waste’ who can be treated with disdain. Epagogix advises movie studios on what scripts to buy based on how closely they match past, successful scripts. Even winemakers make algorithmic judgments, based on statistical analyses of the weather and other characteristics of good and bad vintage years. For wines or films, the stakes are not terribly high. But when algorithms start affecting critical opportunities for employment, career advancement, health, credit and education, they deserve more scrutiny. U.S. hospitals are using big data-driven systems to determine which patients are high-risk – and data far outside traditional health records is informing those determinations. IBM now uses algorithmic assessment tools to sort employees worldwide on criteria of cost-effectiveness, but spares top managers the same invasive surveillance and ranking. In government, too, algorithmic assessments of dangerousness can lead to longer sentences for convicts, or no-fly lists for travelers. Credit scoring drives billions of dollars in lending, but the scorers’ methods remain opaque. The average borrower could lose tens of thousands of dollars over a lifetime, thanks to wrong or unfairly processed data. It took a combination of computational, legal and social scientific skills to unearth each of the examples discussed above – troubling collection, bad or biased analysis, and discriminatory use. Collaboration among experts in different fields is likely to yield even more important work. Grounded in well-established empirical social science methods, their models can and should inform the regulation of firms and governments using algorithms.”
Cindy Cohn, executive director at the Electronic Frontier Foundation, wrote, “The lack of critical thinking among the people embracing these tools is shocking and can lead to some horrible civil liberties outcomes …. I don’t think it’s possible to assign an overall ‘good’ or ‘bad’ to the use of algorithms, honestly. As they say on Facebook, ‘It’s complicated.’”
Bernardo A. Huberman, senior fellow and director of the Mechanisms and Design Lab at HPE Labs, Hewlett Packard Enterprise, said, “Algorithms do lead to the creation of filters through which people see the world and are informed about it. This will continue to increase. If the negative aspects eventually overtake the positive ones, people will stop resorting to interactions with institutions, media, etc. People’s lives are going to continue to be affected by the collection of data about them, but I can also see a future where they won’t care as much or will be compensated every time their data is used for money-making purposes.”
Marcel Bullinga, trend watcher and keynote speaker, commented, “AI will conquer the world, like the internet and the mobile phone once did. It will end the era of apps. Millions of useless apps (because there are way too many for any individual) will become useful on a personal level if they are integrated and handled by AI. For healthy robots/AI, we must have transparent, open source AI. The era of closed is over. If we stick to closed AI, we will see the rise of more and more tech monopolies dominating our world as Facebook and Google and Uber do now.”
Michael Rogers, author and futurist at Practical Futurist, said, “In a sense, we’re building a powerful nervous system for society. Big data, real-time analytics, smart software could add great value to our lives and communities. But at the same time they will be powerful levers of social control, many in corporate hands. In today’s market economy, driven by profit and shareholder value, the possibility of widespread abuse is quite high. Hopefully society as a whole will be able to use these tools to advance more humanistic values. But whether that is the case lies not in the technology, but in the economic system and our politics.”
An anonymous principal engineer commented, “The effect will depend on the situation. In areas where human judgment is required, I foresee negative effects. In areas where human judgment is a hindrance it could be beneficial. For example, I don’t see any reason for there to be train accidents (head-on collisions, speeding around a curve) with the correct design of an intelligent train system. Positive and negative effects will also depend on the perception of the person involved. For example, an intelligent road system could relieve congestion and reduce accidents, but also could restrict freedom of people to drive their cars as they wish (e.g., fast). This could be generalized to a reduction in freedom in general, which could be beneficial to some but detrimental to others.”
Theme 2: Good things lie ahead
Many respondents to the canvassing pointed out that algorithms are already the backbone for most systems and it is quite evident they have been mostly of great benefit and will continue to improve every aspect of life. Their driving idea is that great things will be achieved thanks to recent and coming advances in algorithm-based actions. These respondents said that algorithms will help make sense of massive amounts of data, and this will inspire breakthroughs in science, new conveniences and human capacities in everyday life, and ever-better capacity to link people to the information that will help them. As an anonymous senior researcher employed by Microsoft replied, “They enable us to search the web and sequence genomes. These two activities alone dwarf the negatives.”
Demian Perry, director of mobile at NPR, said algorithmic “helpmates” add efficiencies. “An algorithm is just a way to apply decision-making at scale,” he explained. “Mass-produced decisions are, if nothing else, more consistent. Depending on the algorithm (and whom you ask), that consistency is either less nuanced or more disciplined than you might expect from a human. In the NPR One app, we have yet to find an algorithm that can be trusted to select the most important news and the most engrossing stories that everyone must hear. At the same time, we rely heavily on algorithms to help us make fast, real-time decisions about what a listener’s behavior tells us about their program preferences, and we use these algorithms to pick the best options to present to them at certain points in their listening experience. Thus algorithms are helpmates in the process of curating the news, but they’ll probably never run the show. We believe they will continue to make our drudge work more efficient, so that we have more time to spend on the much more interesting work of telling great stories.”
Stowe Boyd, chief researcher at Gigaom, said, “Algorithms and AI will have an enormous impact on the conduct of business. HR is one enormous area that will be revamped top to bottom by this revolution. Starting at a more fundamental level, education will be recast and AI will be taking a lead role. We will rely on AI to oversee other AIs.”
Data-driven approaches achieved through thoughtful design are a plus
Jason Hong, an associate professor at Carnegie Mellon University, predicted, “On the whole, algorithms will be a net positive for humanity. Any given individual has a large number [of] cognitive biases, limited experiences, and limited information for making a decision. In contrast, an algorithm can be trained on millions or even billions of examples, and can be specifically tuned for fairness, efficiency, speed or other kinds of desired criteria. In practice, an algorithm will be deployed to work autonomously only in cases where the risks are low (e.g., ads, news) or where the certainty is high (e.g., anti-lock brakes, airplane auto-pilot) or good enough (e.g., Uber’s algorithm for allocating passengers to drivers). In most cases, though, it won’t be just a person or just an algorithm, but rather the combination of an expert with an algorithm. For example, rather than just a doctor, it will likely be a doctor working with an AI algorithm that has been trained on millions of electronic health records, their treatments and their outcomes. We have several thousand years of human history showing the severe limitations of human judgment. Data-driven approaches based on careful analysis and thoughtful design can only improve the situation.”
Marti Hearst, a professor at the University of California-Berkeley, said, “For decades computer algorithms have been automating systems in a more-or-less mechanical way for our benefit. For example, a bank customer could set up automated payment for their phone bill. The change we are seeing more recently is that the algorithms are getting increasingly more sophisticated, going from what we might [have] called ‘cut and dried’ decisions like ‘pay the balance of my phone bill’ to much more complex computations resulting in decisions such as ‘show products based on my prior behavior’ or eventually (and menacingly) ‘shut off access to my bank account because of my political posts on social media.’ Every one of these advances is two-sided in terms of potential costs and benefits. The benefits can be truly amazing: automated spoken driving directions that take into account traffic congestion and re-route in real time is stunning – the stuff of science fiction in our own lifetimes. On the other hand, quiet side streets known only to the locals suddenly become full of off-route vehicles from out of town. These new algorithms are successful only because they have access to the data about the activity of large numbers of individual people. And the more reliant we become on them, the fewer options anyone has to go ‘off the grid.’ The rush toward ‘big data’ has not built in adequate protections from harm for individuals and society against potential abuses of this reliance. The bias issue will be worked out relatively quickly, but the excessive reliance on monitoring of every aspect of life appears unavoidable and irreversible.”
Why is the “monitoring of every aspect of life” likely to be “unavoidable and irreversible”? Because all of these improvements are data-dependent. Among the data-reliant innovations expected to rapidly expand are cognitive AI “digital agents” or “assistants.”
Scott Amyx, CEO of Amyx+, commented, “Within the field of artificial intelligence, there has been significant progress on cognitive AI as evidenced by Viv, IBM Watson, Amazon Echo, Alexa, Siri, Cortana and X.ai. Advancement in cognitive AI will usher in a new era of orchestration, coordination and automation that will enable humans to focus on human value-add activities (creativity, friendship, perseverance, resolve, hope, etc.) while systems and machines will manage task orientation. More exciting, in my opinion, is the qualitative, empathetic AI – AI that understands our deep human thoughts, desires and drivers and works to support our psychological, emotional and physical well-being. To that end, we are kicking off a research consortium that will further explore this area of research and development with emphasis on friend AI, empathetic AI, humorous AI and confidant AI. To enable hyper-personalization, these neural network AI agents would have to be at the individual level. All of us at some point in the future will have our own ambient AI virtual assistant and friend to help navigate and orchestrate life. It will coordinate with other people, other AI agents, devices and systems on our behalf. Naturally, concerns of strong AI emerge for some. There is active research, private and public, targeted at friendly AI. We will never know for sure if the failsafe measures that we institute could be broken by self-will.”
Marina Gorbis, executive director at the Institute for the Future, suggested these as “main positive impacts”: “Algorithms will enable each one of us to have a multitude of various types of assistants that would do things on our behalf, amplifying our abilities and reach in ways that we’ve never seen before. Imagine instead of typing search words and getting a list of articles, pushing a button and getting a narrative paper on a specific topic of interest. It’s the equivalent of each one of us having many research and other assistants …. Algorithms also have the potential to uncover current biases in hiring, job descriptions and other text information. Startups like Unitive and Knack show the potential of this.”
Ryan Hayes, owner of Fit to Tweet, said he is looking forward to added algorithmic assistance in his daily life. “There are a lot of ways in which the world is more peaceful and our quality of life better than ever before, but we don’t necessarily feel that way because we’ve been juggling more than ever before, too,” he said. “For example, when I started my career as a [certified public accountant] I could do my job using paper and a 10-key calculator, and when I left for the day I could relax knowing I was done, whereas today I have over 300 applications that I utilize for my work and I can be reached every minute of the day through Slack, texts, several email accounts and a dozen social media accounts. Technology is going to start helping us not just maximize our productivity but shift toward doing those things in ways that make us happier, healthier, less distracted, safer, more peaceful, etc., and that will be a very positive trend. Technology, in other words, will start helping us enjoy being human again rather than burdening us with more abstraction.”
An anonymous deputy CEO wrote, “I hope we will finally see evidence-based medicine and integrated planning in the human habitat. The latter should mean cities developed with appropriate service delivery across a range of infrastructures.”
An anonymous computer security researcher observed, “Algorithms combined with machine learning and data analysis could result in products that predict self-defeating behaviors and react and incentivize in ways that could push users far further than they could go by themselves.”
Code processes are being refined; ethics and issues are being worked out
David Karger, a professor of computer science at MIT, said, “Algorithms are just the latest tools to generate fear as we consider their potential misuse, like the power loom (put manual laborers out of jobs), the car (puts kids beyond the supervision of their parents), and the television (same fears as today’s internet). In all these cases there were downsides but the upsides were greater. The question of algorithmic fairness and discrimination is an important one but it is already being considered. If we want algorithms that don’t discriminate, we will be able to design algorithms that do not discriminate. Of course, there are ethical questions: If we have an algorithm that can very accurately predict whether someone will benefit from a certain expensive medical treatment, is it fair to withhold the treatment from people the algorithm thinks it won’t help? But the issue here is not with the algorithm but with our specification of our ethical principles.”
Respondents predict the development of “ethical machines” and “iteratively improved” code that will diminish the negatives.
Lee McKnight, an associate professor at Syracuse University’s School of Information Studies, wrote, “Algorithms coded in smart service systems will have many positive, life-saving and job-creating impacts in the next decade. Social machines will become much better at understanding your needs, and attempting to help you meet them. Ethical machines – such as drones – will know to sense and avoid collisions with other drones, planes, birds or people, recognize restricted air space, and respect privacy law. Algorithmically driven vehicles will similarly learn to better avoid each other. Health care smart-service systems will be driven by algorithms to recognize human and machine errors and omissions, improving care and lowering costs.”
Jon Lebkowsky, CEO of Polycot Associates, wrote, “I’m personally committed to agile process, through which code is iteratively improved based on practice and feedback. Algorithms can evolve through agile process. So while there may be negative effects from some of the high-impact algorithms we develop, my hope and expectation is that those algorithms will be refined to diminish the negative and enhance the positive impact.”
Edward Friedman, emeritus professor of technology management at the Stevens Institute of Technology, expects more algorithms will be established to evaluate algorithms, writing, “As more algorithms enter the interactive digital world, there will be an increase of Yelp-type evaluation sites that guide users in their most constructive use.”
Ed Dodds, a digital strategist, wrote, “Algorithms will force persons to be more reflective about their own personal ontologies, fixed taxonomies, etc., regarding how they organize their own digital assets or bookmark the assets of others. AI will extrapolate. Users will then be able to run thought experiments such as ‘OK, show the opposite of those assumptions’ and such in natural-language queries. A freemium model will allow whether or not inputting a user’s own preferred filters will be of enough value.”
An anonymous chief scientist observed, “Short-term, the negatives will outweigh the positives, but as we learn and go through various experiences, the balance will eventually go positive. We always need algorithms to be tweakable by humans according to context, creating an environment of IA (intelligent assistants) instead of AI (artificial intelligence).”
Another anonymous respondent agreed, writing, “Algorithms will be improved as a reactive response. So negative results of using them will be complained about loudly at first, word-workers will work on them and identify the language that is at issue, and fine-tune them. At some point it will be 50-50. New ones will always have to be fine-tuned, and it will be the complaining that helps us fine-tune them.”
‘Algorithms don’t have to be perfect; they just have to be better than people’
Some respondents who predicted a mostly positive future said algorithms are unfairly criticized, noting they outperform human capabilities, accomplish great feats and can always be improved.
An anonymous professor who works at New York University said algorithm-based systems are a requirement of our times and mostly work out for the best. “Automated filtering and management of information and decisions is a move forced on us by complexity,” he wrote. “False positives and false negatives will remain a problem, but they will be edge cases.”
An anonymous chief scientist wrote, “Whenever algorithms replace illogical human decision-making, the result is likely to be an improvement.” And an anonymous principal consultant at a top consulting firm wrote, “Fear of algorithms is ridiculously overblown. Algorithms don’t have to be perfect, they just have to be better than people.”
Daniel Berleant, author of The Human Race to the Future, noted, “Algorithms are less subject to hidden agendas than human advisors and managers. Hence the output of these algorithms will be more socially and economically efficient, in the sense that they will be better aligned with their intended goals. Humans are a lot more suspect in their advice and decisions than computers are.”
Avery Holton, an assistant professor and humanities scholar at the University of Utah, got into the details. “In terms of communication across social networks both present and future, algorithms can work quickly to identify our areas of interest as well as others who may share those interests. Yes, this has the potential to create silos and echo chambers, but it also holds the promise of empowerment through engagement encouragement. We can certainly still seek information and relationships by combing through keywords and hashtags, but algorithms can supplement those efforts by showing us not only ‘what’ we might be interested in and ‘what’ we might be missing, but ‘who’ we might be interested in and ‘who’ we might be missing. Further, these algorithms may be able to provide us some insights about others (e.g., their interests, their engagement habits) that help us better approach, develop and sustain relationships.”
David Sarokin, author of Missed Information: Better Information for Building a Wealthier, More Sustainable Future (MIT Press), said algorithms are being applied to identify human bias and discrimination. “Apps/algorithms have a real capability of democratizing information access in important and positive [ways]. For example, phone apps have been developed to collect, collate and combine reports from citizens on their routine interactions – both positive and negative – with police. In widespread use, these can be an effective ‘report card’ for individual officers as well as overall community policing, and help identify potential problems before they get out of hand.”
Dan Ryan, professor of sociology at Mills College in Oakland, California, wrote, “The worry that algorithms might introduce subtle biases strikes me as much social-science ado about very little. No more true than the ways that architecture, cartography, language, organizational rules, credentialing systems, etc., produce these effects.”
An anonymous respondent said, “It would be a fallacy to say that without algorithms our society would be more fair. We can ‘unteach’ discrimination in computers more easily than we can in human beings. The more algorithms are capable of mimicking human behavior, the more we will need to reconsider the implications of what makes us human and how we interact.”
An anonymous principal consultant at a consulting firm wrote, “People often confuse a biased algorithm for an algorithm that doesn’t confirm their biases. If Facebook shows more liberal stories than conservative, that doesn’t mean something is wrong. It could be a reflection of their user base, or of their media sources, or just random chance. What is important is to realize that everything has some bias, intentional or not, and to develop the critical thinking skills to process bias.”
In the future the world may be governed by benevolent AI
An anonymous respondent projected ahead several hundred years, writing:
“Algorithms initially will be an extension of the ‘self’ to help individuals maintain and process the overload of information they have to manage on a daily basis. ‘How’ identities are managed and ‘who’ develops the algorithms will dictate the degree of usefulness and/or exploitation. Fast-forward 200 years – no governments or individuals hold a position of power. The world is governed by a self-aware, ego-less, benevolent AI. A single currency of credit (a la bitcoin) is earned by individuals and distributed by the AI according to the ‘good’ you contribute to society. The algorithm governing the global, collective AI will be optimized toward the common good, maximizing health, safety, happiness, conservation, etc.”
Four themes that concentrate on concerns about algorithms
Participants in this study were in substantial agreement that the abundant positives of accelerating code-dependency will continue to drive the spread of algorithms. However, many argued that – as with all great technological revolutions – algorithms have their dark side. The remaining themes highlight answers that focused on the problems many foresee.
Theme 3: Humanity and human agency are lost when data and predictive modeling become paramount
Many respondents said that as people put too much faith in data, humanity can be lost. Some argued that because technology corporations and, sometimes, governments are most often the agencies behind the code, algorithms are written to optimize efficiency and profitability without much thought about the possible societal impacts of the data modeling and analysis. These respondents said people are considered to be an “input” to the process and they are not seen as real, thinking, feeling, changing beings. Some said that as the process evolves – that is, as algorithms begin to write the algorithms – humans may get left completely out of the loop, letting “the robots decide.”
An anonymous respondent wrote, “We simply can’t capture every data element that represents the vastness of a person and that person’s needs, wants, hopes, desires. Who is collecting what data points? Do the human beings the data points reflect even know, or did they just agree to the terms of service because they had no real choice? Who is making money from the data? How is anyone to know how his/her data is being massaged and for what purposes to justify what ends? There is no transparency, and oversight is a farce. It’s all hidden from view. I will always remain convinced the data will be used to enrich and/or protect others and not the individual. It’s the basic nature of the economic system in which we live.”
Peter Eckart’s comment reflects the attitude of many in this canvassing: “We can create algorithms faster than we can understand or evaluate their impact. The expansion of computer-mediated relationships means that we have less interest in the individual impact of these innovations, and more on the aggregate outcomes. So we will interpret the negative individual impact as the necessary collateral damage of ‘progress.’”
Axel Bruns, a professor at the Digital Media Research Center at Queensland University of Technology, said, “There are competitive, regulatory and legal disadvantages that would result from greater transparency on behalf of the platform operator, and so there is an incentive only to further obfuscate the presence and operations of algorithmic shaping of communications processes. This is not to say that such algorithms are inherently ‘bad,’ in the sense that they undermine effective communication; algorithms such as Google’s PageRank clearly do the job that is asked of them, for instance, and overall have made the web more useful than it would be without them. But without further transparency ordinary users must simply trust that the algorithm does what it is meant to do, and does not inappropriately skew the results it delivers. Such algorithms will continue to be embedded deeply into all aspects of human life, and will also generate increasing volumes of data on their fields. This continues to increase the power that such algorithms already have over how reality is structured, measured and represented, and the potential impact that any inadvertent or deliberate errors could have on user activities, on society’s understanding of itself, and on corporate and government decisions. More fundamentally, the increasing importance of algorithms to such processes also transfers greater importance to the source data they work with, amplifying the negative impacts of data gaps and exclusions.”
An anonymous community advocate said, “There are a lot of places where algorithms are beneficial and helpful, but so far, none of them take into account the actual needs of humans. Human resources are an input in a business equation at the moment, not real, thinking, feeling symbiotes in the eyes of business.”
An anonymous associate professor of political science at a major U.S. university said, “Algorithms are the typecasting of technology. They are a snapshot of behavior influenced by contextual factors that give us a very limited view of an individual. Typecasting is a bad way to be regarded by others and it is a bad way to ‘be.’”
Rebecca MacKinnon, director of the Ranking Digital Rights project at New America, commented, “Algorithms driven by machine learning quickly become opaque even to their creators who no longer understand the logic being followed to make certain decisions or produce certain results. The lack of accountability and complete opacity is frightening. On the other hand, algorithms have revolutionized humans’ relationship with information in ways that have been life-saving and empowering and will continue to do so.”
Programming primarily in pursuit of profits and efficiencies is a threat
A large number of respondents expressed deep concerns about the primary interests being served by networked algorithms. Most kept their comments anonymous, which makes sense since a significant number of the participants in this canvassing are employed by or are funded in some regard by corporate or government interests. As an anonymous chairman and CEO at a nonprofit organization observed, “The potential for good is huge, but the potential for misuse and abuse, intentional and inadvertent, may be greater.” (All respondents not identified by name in this section submitted their comments anonymously.)
One participant described the future this way: “The positives are all pretty straightforward, e.g., you get the answer faster, the product is cheaper/better, the outcome fits the needs more closely. Similarly, the negatives are mostly pretty easy to foresee as well, given that it’s fundamentally people/organizations in positions of power that will end up defining the algorithms. Profit motives, power accumulation, etc., are real forces that we can’t ignore or eliminate. Those who create the algorithms have a stake in the outcome, so they are, by definition, biased. It’s not necessarily bad that this bias is present, but it does have dramatic effects on the outputs, available inputs and various network effects that may be entirely indirect and/or unforeseen by the algorithm developers. As the interconnectedness of our world increases, accurately predicting the negative consequences gets ever harder, so it doesn’t even require a bad actor to create deleterious conditions for groups of people, companies, governments, etc.”
Another respondent said, “The algorithms will serve the needs of powerful interests, and will work against the less-powerful. We are, of course, already seeing this start to happen. Today there is a ton of valuable data being generated about people’s demographics, behaviours, attitudes, preferences, etc. Access to that data (and its implications) is not evenly distributed. It is owned by corporate and governmental interests, and so it will be put to uses that serve those interests. And so what we see already today is that in practice, stuff like ‘differential pricing’ does not help the consumer; it helps the company that is selling things, etc.”
An IT architect at IBM said, “Companies seek to maximize profit, not maximize societal good. Worse, they repackage profit-seeking as a societal good. We are nearing the crest of a wave the trough side of which is a new ethics of manipulation, marketing, nearly complete lack of privacy. All predictive models, whether used for personal convenience or corporate greed, require large amounts of data. The ways to obtain that are at best gently transformative of culture, and on the low side, destructive of privacy. Corporations’ use of big data predicts law enforcement’s use of shady techniques (e.g., Stingrays) to invade privacy. People all too quickly view law-enforcement as ‘getting the bad guys their due’ but plenty of cases show abuse, mistaken identity, human error resulting in police brutality against the innocent, and so on. More data is unlikely to temper the mistakes; instead, it will fuel police overreach, just as it fuels corporate overreach.”
Said another respondent, “Everything will be geared to serve the interests of the corporations and the 1%. Life will become more convenient, but at the cost of discrimination, information compartmentalization and social engineering.”
A professor noted, “If lean, efficient global corporations are the definition of success, the future will be mostly positive. If maintaining a middle class with opportunities for success is the criterion by which the algorithms are judged, this will not be likely. It is difficult to imagine that the algorithms will consider societal benefits when they are produced by corporations focused on short-term fiscal outcomes.”
A senior software developer wrote, “Smart algorithms can be incredibly useful, but smart algorithms typically lack the black-and-white immediacy that the greedy, stupid and short-sighted prefer. They prefer stupid, overly broad algorithms with lower success rates and massive side effects because these tend to be much easier to understand. As a result, individual human beings will be herded around like cattle, with predictably destructive results on rule of law, social justice and economics. For instance, I see algorithmic social data crunching as leading to ‘PreCrime,’ where ordinary, innocent citizens are arrested because they set off one too many flags in a Justice Department data dragnet.”
One business analyst commented, “The outcome will be positive for society on a corporate/governmental basis, and negative on an individual basis.”
A faculty member at a U.S. university said, “Historically, algorithms are inhumane and dehumanizing. They are also irresistible to those in power. By utilitarian metrics, algorithmic decision-making has no downside; the fact that it results in perpetual injustices toward the very minority classes it creates will be ignored. The Common Good has become a discredited, obsolete relic of The Past.”
Another respondent who works for a major global human rights foundation said, “Algorithms are already put in place to control what we see on social media and how content is flagged on the same platforms. That’s dangerous enough – introducing algorithms into policing, health care, educational opportunities can have a much more severe impact on society.”
An anonymous professor of media production and theory warned, “While there is starting to be citizen response to algorithms, they tend to be seen as neutral if they are seen at all. Since algorithms are highly proprietary and highly lucrative, they are highly dangerous. With TV, the U.S. developed public television, what kind of public space for ownership of information will be possible? It is the key question for anyone interested in the future of democratic societies.”
David Golumbia, an associate professor of digital studies at Virginia Commonwealth University, wrote, “The putative benefits of algorithmic processing are wildly overstated and the harms are drastically underappreciated. Algorithmic processing in many ways deprives individuals and groups of the ability to know about, and to manage, their lives and responsibilities. Even when aspects of algorithmic control are exposed to individuals, they typically have nowhere near the knowledge necessary to understand what the consequences are of that control. This is already widely evident in the way credit scoring has been used to shape society for decades, most of which have been extremely harmful despite the credit system having some benefit to individuals and families (although the consistent provision of credit beyond what one’s income can bear remains a persistent and destructive problem). We are going full-bore into territory that we should be approaching hesitantly if at all, and to the degree that they are raised, concerns about these developments are typically dismissed out of hand by those with the most to gain from those developments.”
An anonymous professor at the University of California-Berkeley observed, “Algorithms are being created and used largely by corporations. The interests of the market economy are not the same as those of the people being subjected to algorithmic decision-making. Costs and the romanticization of technology will drive more and more adoption of algorithms in preference to human-situated decision-making. Some will have positive impacts. But the negatives are potentially huge. And I see no kind of oversight mechanism that could possibly work.”
Joseph Turow, a communications professor at the University of Pennsylvania, said, “A problem is that even as they make some tasks easier for individuals, many algorithms will chip away at their autonomy by using the data from the interactions to profile them, score them, and decide what options and opportunities to present them next based on those conclusions. All this will be carried out by proprietary algorithms that will not be open to proper understanding and oversight even by the individuals who are scored.”
Karl M. van Meter, a sociological researcher and director of the Bulletin of Methodological Sociology, Ecole Normale Supérieure de Paris, said, “The question is really, ‘Will the net overall effect of the next decade of bosses be positive for individuals and society or negative for individuals and society?’ Good luck with that one.”
Sometimes algorithms make false assumptions that place people in an echo chamber/ad delivery system that isn’t really a fit for them. An engineer at a U.S. government organization complained, “Some work will become easier, but so will profiling. I, personally, am often misidentified as one racial type, political party, etc., by my gender, address, career, etc., and bombarded with advertising and spam for that person. If I had an open social profile, would I even have that luxury – or would everything now ‘match’ whichever article I most recently read?”
An anonymous respondent wrote, “One major question is: To what extent will the increased use of algorithms encourage a behaviorist way of thinking of humans as creatures of stimulus and response, capable of being gamed and nudged, rather than as complex entities with imagination and thought? It is possible that a wave of algorithm-ization will trigger new debates about what it means to be a person, and how to treat other people. Philip K. Dick has never been more relevant.”
Evan Selinger, professor of philosophy at the Rochester Institute of Technology, wrote, “The more algorithmic advice, algorithmic decision-making and algorithmic action that occurs on our behalf, the more we risk losing something fundamental about our humanity. But because being ‘human’ is a contested concept, it’s hard to make a persuasive case for when and how our humanity is actually diminished, and how much harm each diminishment brings. Only when better research into these questions is available can a solid answer be provided as to whether more positive or negative outcomes arise.”
Algorithms manipulate people and outcomes, and even ‘read our minds’
Respondents registered fears about the ease with which powerful interests can manipulate people and outcomes through the design of networked intelligence and tools.
Michael Kleeman, senior fellow at the University of California-San Diego, observed, “In the hands of those who would use these tools to control, the results can be painful and harmful.”
Peter Levine, professor and associate dean for research at Tisch College of Civic Life, Tufts University, noted, “What concerns me is the ability of governments and big companies to aggregate information and gain insight into individuals that they can use to influence those individuals in ways that are too subtle to be noticed or countered. The threat is to liberty.”
Freelance journalist Mary K. Pratt commented, “Algorithms have the capability to shape individuals’ decisions without them even knowing it, giving those who have control the algorithms (in how they’re built and deployed) an unfair position of power. So while this technology can help in so many areas, it does take away individual decision-making without many even realizing it.”
Martin Shelton, Knight-Mozilla OpenNews Fellow at The Coral Project + New York Times, wrote, “Peoples’ values inform the design of algorithms – what data they will use, and how they will use data. Far too often, we see that algorithms reproduce designers’ biases by reducing complex, creative decisions to simple decisions based on heuristics. Those heuristics do not necessarily favor the person who interacts with them. These decisions typically lead software creators not to optimize for qualitative experiences, but instead, optimizing for click-through rates, page views, time spent on page, or revenue. These design decisions mean that algorithms use (sometimes quite misplaced) heuristics to decide which news articles we might be interested in; people we should connect with; products we should buy.”
Chris Showell, an independent health informatics researcher based in Australia, said, “The organisation developing the algorithm has significant capacity to influence or moderate the behaviour of those who rely on the algorithm’s output. Two current examples: manipulation of the process displayed in online marketplaces, and use of ‘secret’ algorithms in evaluating social welfare recipients. There will be many others in years to come. It will be challenging for even well-educated users to understand how an algorithm might assess them, or manipulate their behaviour. Disadvantaged and poorly educated users are likely to be left completely unprotected.”
Writer James Hinton commented, “The fact the internet can, through algorithms, be used to almost read our minds, means those who have access to the algorithms and their databases have a vast opportunity to manipulate large population groups. The much-talked-about ‘experiment’ conducted by Facebook to determine if it could manipulate people emotionally through deliberate tampering with news feeds is but one example of both the power, and the lack of ethics, that can be displayed.”
An anonymous president of a consulting firm said, “LinkedIn tries to manipulate me to benefit from my contacts’ contacts and much more. If everyone is intentionally using or manipulating each other, is it acceptable? We need to see more-honest, trust-building innovations and fewer snarky corporate manipulative design tricks. Someone told me that someday only rich people will not have smartphones, suggesting that buying back the time in our day will soon become the key to quality lifestyles in our age of information overload. At what cost, and with what ‘best practices’ for the use of our recovered time per day? The overall question is whether good or bad behaviors will predominate globally.” This consultant suggested: “Once people understand which algorithms manipulate them to build corporate revenues without benefiting users, they will be looking for more-honest algorithm systems that share the benefits as fairly as possible. When everyone globally is online, another 4 billion young and poor learners will be coming online. A system could go viral to win trillions in annual revenues based on micropayments due to sheer volume. Example: The Facebook denumerator app removes the manipulative aspects of Facebook, allowing users to return to more typically social behavior.”
Several respondents expressed concerns about a particular industry – insurers. An anonymous respondent commented, “The increasing migration of health data into the realm of ‘big data’ has potential for the nightmare scenario of Gattaca writ real.”
Masha Falkov, artist and glassblower, said, “It is important to moderate algorithms with human judgment and compassion. Already we see every day how insurance companies attempt to wrest themselves out of paying for someone’s medical procedure. The entire health care system in the U.S. is a madhouse presently moderated by individuals who secretly choose to rebel against its tyranny. Doctors who fight for their patients to get the medicine they need, operators within insurance companies who decide to not deny the patient the service, at the risk of their own job. Our artificial intelligence is only as good as we can design it. If the systems we are using presently do not evolve with our needs, algorithms will be useless at best, harmful at worst.”
Systems architect John Sniadowski noted, “Predictive modeling will make life more convenient, but conversely it will narrow choices and confine individuals into classes of people from which there is no escape. Predictive modeling is unstoppable because international business already sees massive financial advantages by using such techniques. An example of this is insurance where risk is now being eliminated in search of profits instead of the original concept of insurance being shared risk. People are now becoming uninsurable either because of their geographic location or social position. Premiums are weighted against individuals on control decisions on which the individual has no control and therefore cannot improve their situation.”
Ryan Sweeney, director of analytics at Ignite Social Media, commented, “Every human is different, so an algorithm surrounding health care could tailor a patient’s treatment plan. It could also have the potential to serve the interests of the insurance company over the patient.”
All of this will lead to a flawed yet inescapable logic-driven society
Some who assessed the impacts of algorithms in the next decade expressed the opinion that they are unreliable, “oversold” and “cold,” saying they “give a false impression” of efficacy and are “not easily subject to critique.” An anonymous respondent said, “It’s not that algorithms are the problem; it’s that we think that with sufficient data we will have wisdom. We will become reliant upon ‘algorithms’ and data and this will lead to problematic expectations. Then that’s when things will go awry.”
Jason Hong, an associate professor at Carnegie Mellon University, said, “People will forget that models are only an approximation of reality. The old adage of garbage in, garbage out still applies, but the sheer quantity of data and the speed of computers might give the false impression of correctness. As a trivial example, there are stories of people following GPS too closely and ending up driving into a river.”
An anonymous computer science PhD noted, “Algorithms typically lack sufficient empirical foundations, but are given higher trust by users. They are oversold and deployed in roles beyond their capacity.”
Bob Frankston, internet pioneer and software innovator, said, “The negatives of algorithms will outweigh the positives. There continues to be magical thinking assuming that if humans don’t intervene the ‘right thing’ will happen. Sort of the modern gold bugs that assume using gold as currency prevents humans from intervening. Algorithms are the new gold, and it’s hard to explain why the average ‘good’ is at odds with the individual ‘good.’”
An anonymous respondent observed, “Algorithms are opaque and not easily subject to critique. People too easily believe that they are scientific. Health care – there is not a single study that shows clinical improvement from the use of the electronic health record, and instead of saving costs, it has increased them. Resources going there are resources not gong into patient care. Consumer choice – we only see what we are allowed to see in whatever markets we’ve been segmented into. As that segmentation increases, our choices decrease. Corporate consolidation also decreases choices. Likewise news, opportunities, access. Big data can be helpful – like tracking epidemics – but it can also be devastating because there is a huge gap between individuals and the statistical person. We should not be constructing social policy just on the basis of the statistical average but, instead, with a view of the whole population. So I am inclined to believe that big data gets us to Jupiter, it may help us cope with climate change but it will not increase justice, fairness, morality and so on.”
B. Remy Cross, an assistant professor of sociology at Webster University in Missouri, said, “Algorithms in particular are prone to a sort of tehcno-fetishism where they are seen as perfectly unbiased and supremely logical, when they are often nothing of the sort.”
An anonymous technology developer commented, “Algorithms will overestimate the certainty with which people hold convictions. Most people are pretty wishy-washy but algorithms try to define you by estimating feelings/beliefs. If I ‘kind of like’ something I am liable to be grouped with fervent lovers of that thing.”
Some said the aura of definitive digital logic is already difficult to overcome. An anonymous software security consultant bemoaned the lack of quick and fair appeals processes for automated decisions. “It’s already nearly impossible to correct an incorrect credit report, despite the existence of clear laws requiring support for doing so. It seems unlikely that similar problems will be easy to correct in the future unless significant regulation is added around such systems. I am hopeful the benefits will be significant, but I expect the downsides to be far more obvious and easy to spot than the upsides.”
Some respondents said human managers will ignore people’s needs or leave them unattended more as machine intelligence takes over more tasks. An anonymous participant commented, “The use of algorithms will create a distance from those who make corporate decisions and the actual decision that gets made. This will result in the plausible deniability that a manager did not actively control the outcome of the algorithm, and as a result, (s)he is not responsible for the outcome when it affects either the public or the employees.”
Another anonymous participant wrote, “The downsides to these are any situations that do not fit a standard set of criteria or involve judgment calls – large systems do not handle exceptional situations well and tend to be fairly inflexible and complicated to navigate. I see a great deal of trouble in terms of connections between service providers and the public they serve because of a lack of empathy and basic interaction. It’s hard to plan for people’s experiences when the lived experience of the people one plans for are alien to one’s own experiential paradigm.”
An anonymous computer scientist wrote, “The tech industry is attuned to computer logic, not feelings or ethical outcomes. The industrial ‘productivity’ paradigm is running out of utility, and we need a new one that is centered on more human concerns.”
James McCarthy, a manager, commented, “Sometimes stuff just happens that can’t be accounted for by even a sufficiently complex rules set, and I worry that increasing our dependency on algorithmic decision-making will also create an increasingly reductive view of society and human behavior.”
An anonymous respondent said, “An algorithm is only as good as the filter it is put through, and the interpretation put upon it. Too often we take algorithms as the basis of fact, or the same as a statistic, which they are not. They are ways of collecting information into subjects. An over-reliance on this and the misinterpretation of what they are created for shall lead to trouble within the next decade.”
Some fear a loss of complex decision-making capabilities and local intelligence
Since the early days of widespread adoption of the internet, some have expressed concerns that the fast-evolving dependence upon intelligence augmentation via algorithms will make humans less capable of thinking for themselves. Some respondents in this canvassing noted this as likely to have a negative impact on the capabilities of the individual.
Amali De Silva-Mitchell, a futurist and consultant, wrote, “Predictive modeling will limit individual self-expression hence innovation and development. It will cultivate a spoon-fed population with those in the elite being the innovators. There will be a loss in complex decision-making skills of the masses. Kings and serfs will be made and the opportunity for diversification lost and then even perhaps global innovative solutions lost. The costs of these systems will be too great to overturn if built at a base level. The current trend toward the uniform will be the undoing rather than building of platforms that can communicate with everything so that innovation is left as key and people can get the best opportunities. Algorithms are not the issue, the issue is a standard algorithm.”
An anonymous respondent said, “Automated decision-making will reduce the perceived need for critical thinking and problem solving. I worry that this will increase trust in authority and make decisions of all kinds more opaque.”
Dave McAllister, director at Philosophy Talk, said, “We will find ourselves automatically grouped into classes (caste system) by algorithms. While it may increase us being effective in finding the information we need while drowning in a world of big data, it will also limit the scope of synthesis and serendipitous discovery.”
Giacomo Mazzone wrote, “Unfortunately most algorithms that will be produced in the next 10 years will be from global companies looking for immediate profits. This will kill local intelligence, local skills, minority languages, local entrepreneurship because most of the available resources will be drained out by the global competitors. The day that a ‘minister for algorithms toward a better living’ will be created is likely to be too late unless new forms of social shared economy emerge, working on ‘algorithms for happiness.’ But this is likely to take longer than 10 years.”
Jesse Drew, a digital media professor at the University of California-Davis, replied, “Certainly algorithms can make life more efficient, but the disadvantage is the weakening of human thought patterns that rely upon serendipity and creativity.”
Ben Railton, a professor of English and American studies at Fitchburg State University in Massachusetts, wrote, “Algorithms are one of the least attractive parts of both our digital culture and 21st-century capitalism. They do not allow for individual identity and perspective. They instead rely on the kinds of categorizations and stereotypings we desperately need to move beyond.”
Miles Fidelman, systems architect, policy analyst and president at the Center for Civic Networking, wrote, “By and large, tools will disproportionally benefit those who have commercial reasons to develop them – as they will have the motivation and resources to develop and deploy tools faster.”
One respondent warned of looming motivations to apply algorithms more vigorously will limit freedom of expression.
Joe McNamee, executive director at European Digital Rights, commented, “The Cambridge/Sanford studies on Facebook likes, the Facebook mood experiment, Facebook’s election turnout experiment and the analysis of Google’s ability to influence elections have added to the demands for online companies to become more involved in policing online speech. All raise existential questions for democracy, free speech and, ultimately, society’s ability to evolve. The range of ‘useful’ benefits is broad and interesting but cannot outweigh this potential cost.”
Suggested solutions include embedding respect for the individual
Algorithms require and create data. Much of the internet economy has been built by groups offering “free” use of online tools and access to knowledge while minimizing or masking the fact that people are actually paying with their attention and/or allegiance – as well as complete algorithmic access to all of their private information plus ever-more-invasive insights into their hopes, fears and other emotions. Some say it has already gotten to the point at which the data collectors behind the algorithms are likely to know more about you than you do yourself.
Rob Smith, software developer and privacy activist, observed, “The major downside is that in order for such algorithms to function, they will need to know a great deal about everyone’s personal lives. In an ecosystem of competing services, this will require sharing lots of information with that marketplace, which could be extremely dangerous. I’m confident that we can in time develop ways to mitigate some of the risk, but it would also require a collective acceptance that some of our data is up for grabs if we want to take advantage of the best services. That brings me to perhaps the biggest downside. It may be that, in time, people are – in practical terms – unable to opt out of such marketplaces. They might have to pay a premium to contract services the old-fashioned way. In summary, such approaches have a potential to improve matters, at least for relatively rich members of society and possibly for the disadvantaged. But the price is high and there’s a danger that we sleepwalk into things without realising what it has cost us.”
David Adams, vice president of product at a new startup, said, “Overreach in intellectual property in general will be a big problem in our future.”
Some respondents suggested that assuring individuals the rights to and control over their identity is crucial.
Garth Graham, board member at Telecommunities Canada, wrote, “The future positives will only outweigh the negatives if the simulation of myself – the anticipation of my behaviours – that the algorithms make possible is owned by me, regardless of who created it.”
Paul Dourish, chancellor’s professor of informatics at the University of California-Irvine, said, “More needs to be done to give people insight into and control over algorithmic processing – which includes having algorithms that work on individuals’ behalf rather than on behalf of corporations.”
Marshall Kirkpatrick, co-founder of Little Bird, previously with ReadWriteWeb and TechCrunch, said, “Most commercial entities will choose to implement algorithms that serve them even at the expense of their constituents. But some will prioritize users and those will be very big. Meeting a fraction of the opportunities that arise will require a tremendous expansion of imagination.”
Susan Price, digital architect and strategist at Continuum Analytics, commented, “The transparent provenance of data and transparent availability of both algorithms and analysis will be crucial to creating the trust and dialog needed to keep these systems fair and relatively free of bias. This necessary transparency is in conflict with the goals of corporations developing unique value in intellectual property and marketing. The biggest challenge is getting humans in alignment on what we collectively hope our data will show in the future – establishing goals that reflect a fair, productive society, and then systems that measure and support those goals.”
One respondent suggested algorithm-writing teams include humanist thinkers. Dana Klisanin, founder and CEO of Evolutionary Guidance Media R&D, wrote, “If we want to weigh the overall impact of the use of algorithms on individuals and society toward ‘positive outweighs negative,’ the major corporations will need to hold themselves accountable through increasing their corporate social responsibility. Rather than revenue being the only return, they need to hire philosophers, ethicists and psychologists to help them create algorithms that provide returns that benefit individuals, society and the planet. Most individuals have never taken a course in ‘Race, Class and Gender,’ and do not recognize discrimination even when it is rampant and visible. The hidden nature of algorithms means that it will take individuals and society that much longer to demand transparency. Or, to say it another way: We don’t know what we don’t know.”
An anonymous respondent who works for the U.S. government cited the difficulties in serving both societal good and the rights of the individual, writing, “There is a tension between the wishes of individuals and the functions of society. Fairness for individuals comes at the expense of some individual choices. It is hard to know how algorithms will end up on the spectrum between favoring individuals over a functioning society because the trend for algorithms is toward artificial intelligence. AI will likely not work the same way that human intelligence does.”
As code takes over complex systems, humans are left out of the loop
As intelligent systems and knowledge networks become more complex and artificial intelligence and quantum computing evolve over the next decade, experts expect that humans will be left more and more “out of the loop” as more and more aspects of code creation and maintenance are taken over by machine intelligence.
The vast majority of comments in this vein came from expert respondents who remained anonymous. A sampling of these statements:
An executive director for an open source software organization commented, “Most people will simply lose agency as they don’t understand how choices are being made for them.”
One respondent said, “Everything will be ‘custom’-tailored based on the groupthink of the algorithms; the destruction of free thought and critical thinking will ensure the best generation is totally subordinate to the ruling class.”
Another respondent wrote, “Current systems are designed to emphasize the collection, concentration and use of data and algorithms by relatively few large institutions that are not accountable to anyone, and/or if they are theoretically accountable are so hard to hold accountable that they are practically unaccountable to anyone. This concentration of data and knowledge creates a new form of surveillance and oppression (writ large). It is antithetical to and undermines the entire underlying fabric of the erstwhile social form enshrined in the U.S. Constitution and our current political-economic-legal system. Just because people don’t see it happening doesn’t mean that it’s not, or that it’s not undermining our social structures. It is. It will only get worse because there’s no ‘crisis’ to respond to, and hence, not only no motivation to change, but every reason to keep it going – especially by the powerful interests involved. We are heading for a nightmare.”
A scientific editor observed, “The system will win; people will lose. Call it ‘The Selfish Algorithm’; algorithms will naturally find and exploit our built-in behavioral compulsions for their own purposes. We’re not even consumers anymore. As if that wasn’t already degrading enough, it’s commonplace to observe that these days people are the product. The increasing use of ‘algorithms’ will only – very rapidly – accelerate that trend. Web 1.0 was actually pretty exciting. Web 2.0 provides more convenience for citizens who need to get a ride home, but at the same time – and it’s naive to think this is a coincidence – it’s also a monetized, corporatized, disempowering, cannibalizing harbinger of the End Times. (I exaggerate for effect. But not by much.)”
A senior IT analyst said, “Most people use and will in the future use the algorithms as a facility, not understanding their internals. We are in danger of losing our understanding and then losing the capability to do without. Then anyone in that situation will let the robots decide.”
What happens when algorithms write algorithms? “Algorithms in the past have been created by a programmer,” explained one anonymous respondent. “In the future they will likely be evolved by intelligent/learning machines. We may not even understand where they came from. This could be positive or negative depending on the application. If machines/programs have autonomy, this will be more negative than positive. Humans will lose their agency in the world.”
And then there is the possibility of an AI takeover.
Seti Gershberg, executive producer and creative director at Arizona Studios, wrote, “At first, the shift will be a net benefit. But as AI begins to pass the Turing test and potentially become sentient and likely super-intelligent, leading to an intelligence explosion as described by Vernor Vinge, it is impossible to say what they will or will not do. If we can develop a symbiotic relationship with AI or merge with them to produce a new man-machine species, it would be likely humans would survive such an event. However, if we do not create a reason for AI to need humans then they would either ignore us or eliminate us or use us for a purpose we cannot imagine. Recently, the CEO of Microsoft put forth a list of 10 rules for AI and humans to follow with regard to their programming and behavior as a method to develop a positive outcome for both man and machines in the future. However, if humans themselves cannot follow the rules set forth for good behavior and a positive society (i.e., the Ten Commandments – not in a religious sense, but one of common sense) I would ask the question, why would or should AI follow rules humans impose on them?”
Theme 4: Biases exist in algorithmically organized systems
There are two strands of thinking that tie together here. One is that the algorithm creators (code writers), even if they strive for inclusiveness, objectivity and neutrality, build into their creations their own perspectives and values. The other is that the datasets to which algorithms are applied have their own limits and deficiencies. Even datasets with millions or billions of pieces of information do not capture the fullness of people’s lives and the diversity of their experiences. Moreover, the datasets themselves are imperfect because they do not contain inputs from everyone, or a representative sample of everyone. This section covers the respondent answers on both those fronts.
Algorithms reflect the biases of programmers and datasets
Bias and poorly developed datasets have been widely recognized as a serious problem that technologists say they are working to address; however, many respondents see this as a problem that will not be remedied anytime soon.
Randy Bush, Internet Hall of Fame member and research fellow at Internet Initiative Japan, wrote, “Algorithmic methods have a very long way to go before they can deal with the needs of individuals. So we will all be mistreated as more homogenous than we are.”
Eugene H. Spafford, a professor at Purdue University, said, “Algorithmic decisions can embody bias and lack of adjustment. The result could be the institutionalization of biased and damaging decisions with the excuse of, ‘The computer made the decision, so we have to accept it.’ If algorithms embody good choices and are based on carefully vetted data, the results could be beneficial. To do that requires time and expense. Will the public/customers demand that?”
Irina Shklovski, an associate professor at the IT University of Copenhagen, said, “Discrimination in algorithms comes from implicit biases and unreflective values embedded in implementations of algorithms for data processing and decision-making. There are many possibilities to data-driven task and information-retrieval support, but the expectation that somehow automatic processing will necessarily be more ‘fair’ makes the assumption that implicit biases and values are not part of system design (and these always are). Thus the question is how much agency will humans retain in the systems that will come to define them through data and how this agency can be actionably implemented to support human rights and values.”
An anonymous freelance consultant observed, “Built-in biases (largely in favour of those born to privilege such as Western Caucasian males, and, to a lesser extent, young south-Asian and east-Asian men) will have profound, largely unintended negative consequences to the detriment of everybody else: women, especially single parents, people of colour (any shade of brown or black), the ‘olds’ over 50, immigrants, Muslims, non-English speakers, etc. This will not end well for most of the people on the planet.”
Marc Brenman, managing partner at IDARE, wrote, “The algorithms will reflect the biased thinking of people. Garbage in, garbage out. Many dimensions of life will be affected, but few will be helped. Oversight will be very difficult or impossible.”
Jenny Korn, a race and media scholar at the University of Illinois at Chicago, noted, “The discussion of algorithms should be tied to the programmers programming those algorithms. Algorithms reflect human creations of normative values around race, gender and other areas related to social justice. For example, searching for images of ‘professor’ will produce pictures of white males (including in cartoon format), but to find representations of women or people of color, the search algorithm requires the user to include ‘woman professor’ or ‘Latina professor,’ which reinforces the belief that a ‘real’ professor is white and male. Problematic! So, we should discuss the (lack of) critical race and feminist training of the people behind the algorithm, not just the people using the algorithm.”
Adrian Hope-Bailie, standards officer at Ripple, noted, “One of the greatest challenges of the next era will be balancing protection of intellectual property in algorithms with protecting the subjects of those algorithms from unfair discrimination and social engineering.”
David Wuertele, a software engineer for a major company innovating autonomous vehicles, noted, “I am optimistic that the services engineers build are capable of being free of discrimination, and engineers will try to achieve that ideal. I expect that we will have some spectacular failures as algorithms get blamed for this or that social tragedy, but I believe that we will have an easier time fixing those services than we will have fixing society.”
Eric Marshall, a systems architect, said, “Algorithms are tweaked or replaced over time. Similar to open source software, the good will outweigh the bad, if the right framework is found.”
Kevin Novak, CEO of 2040 Digital, commented, “Algorithms can lead to filtered results that demonstrate biased or limited information to users. This bias and limitation can lead to opinions or understanding that does not reflect the true nature of a topic, issue or event. Users should have the option to select algorithmic results or natural results.”
Author Paul Lehto observed, “Unless the algorithms are essentially open source and as such can be modified by user feedback in some fair fashion, the power that likely algorithm-producers (corporations and governments) have to make choices favorable to themselves, whether in internet terms of service or adhesion contracts or political biases, will inject both conscious and unconscious bias into algorithms.”
An anonymous respondent commented, “If you start at a place of inequality and you use algorithms to decide what is a likely outcome for a person/system, you inevitably reinforce inequalities. For example, if you were really willing to use the data that exist right now, we would tell African-American men from certain metro areas that they should not even consider going to college – it won’t ‘pay off’ for them because of wage discrimination post-schooling. Is this an ethical position? No. But is it what a computer would determine to be the case based on existing data? Yes.”
And another respondent, Jeff Kaluski, predicted that trying to eliminate all bias may cause new problems, commenting, “New algs will start by being great, then a problem will emerge. The creator will be sued in the U.S. The alg will be corrected. It won’t be good enough for the marginalized group. Someone else will create a better alg that was ‘written in part by marginalized group’ then we’ll have a worse alg than the original+correction.”
Lisa Heinz, a doctoral student at Ohio University, said, “Those of us who learn and work in human-computer areas of study will need to make sure our concerns about discrimination and the exclusionary nature of the filter-bubble are addressed in the oversight mechanisms of algorithm development. This means that all voices, genders and races need to be incorporated into the development of algorithms to prevent even unintentional bias. Algorithms designed and created only by young white men will always benefit young white men to the exclusion of all others.”
Following are additional comments by anonymous respondents regarding bias:
• “The rise of unfounded faith in algorithmic neutrality coupled with spread of big data and AI will enable programmer bias to spread and become harder to detect.”
• “The positives of algorithmic analysis are largely about convenience for the comfortable; the negatives vastly outweigh them in significance.”
• “Bias is inherent in algorithms. This will only function to make humans more mechanical, and those who can rig algorithms to increase inequality and unfairness will, of course, prevail.”
• “Algorithms value efficiency over correctness or fairness, and over time their evolution will continue the same priorities that initially formulated them.”
• “Algorithms can only reflect our society back to us, so in a feedback loop they will also reflect our prejudices and exacerbate inequality. It’s very important that they not be used to determine things like job eligibility, credit reports, etc.”
• “Algorithms purport to be fair, rational and unbiased but just enforce prejudices with no recourse.”
• “Poor algorithms in justice systems actually preserve human bias instead of mitigating it. As long as algorithms are hidden from public view, they can pose a great danger.”
• “How can we expect algorithms designed to maximize ‘efficiency’ (which is an inherently conservative activity) to also push underlying social reform?”
Algorithms depend upon data that is often limited, deficient or incorrect
Some made the case that datasets upon which algorithmic decisions are made may exclude some groups of people, eliminate consumer choice and not recognize exceptions. They may include limited, skewed or incorrect detail, and the analysis based on that can cause harm. An anonymous respondent noted, “Until we begin to measure what we value rather than valuing what we measure, any insights we may gain from algorithms will be canceled out by false positives caused by faulty or incomplete data.”
An anonymous senior program manager at Microsoft observed, “Because of inherent bias (due to a lack of diversity), many algorithms will not fully reflect the complexity of the problems they are trying to address and solutions will tend to sometimes neglect important factors. Unfortunately, it will take time until biases (or simply short-sighted thinking) baked into these algorithms will get detected. By then the American government will have banned innocent people from boarding planes, insurers will have raised premiums for the wrong people, and ‘predictive crime prevention’ will have gotten out of hand.”
An anonymous respondent commented, “Automated systems can never address the complexity of human interaction with the same degree of precision as a person would.” And artist Masha Falkov said, “Real life does not always mimic mathematics. Algorithms have a limited number of variables, and often life shows that it needs to factor in extra variables. There should always be feedback in order to develop better variables and human interaction when someone falls through the cracks of the new normalcy as defined by the latest algorithm. A person may be otherwise a good person in society, but they may be judged for factors over which they do not have any control.”
Randy Albelda, an economics professor at the University of Massachusetts-Boston, replied, “My research is on poor people. I’ve been doing this for a long time (close to 30 years). And no matter how much information, data, empirical evidence that is presented about poor people, we still have horrible anti-poverty policies, remarkable misconceptions about poor people, and lots more poor people. Collecting and analyzing data does not ‘set us free.’ ‘Facts’ are convenient. Political economic forces shape the way we understand and use ‘facts/data’ as well as technology. If we severely underfund health care or much of health care dollars get sucked up by insurance companies, algorithms will be used to allocate insufficient dollars on patients. It will not improve health care.”
Will Kent, an e-resources specialist on the staff at Loyola University-Chicago, observed, “Any amount/type of discrimination could occur. It could be as innocent as a slip-up in the code or a mistranslation. It could be as nefarious as deliberate suppression, obfuscation or lie of omission.”
An anonymous respondent said, “I don’t think we understand intersectionality enough to engineer it in an algorithm. As someone who is LBGTQ, and a member of a small indigenous group who speaks a minority language, I have already encountered so many ‘blind spots’ online – but who do you tell? How do you approach the algorithm? How do you influence it without acquiescing?”
Hume Winzar, an associate professor of business at Macquarie University in Sydney, Australia, wrote, “Banks, governments, insurance companies, and other financial and service providers will use whatever tools they can to focus on who are risks. It’s all about money and power.”
An anonymous lead field technician replied, “Predictive modeling is based on statistical analysis, which by its nature ignores edge cases. It will lead to less freedom of choice in products, more [subtle] coercive advertising and an inability for people to make human mistakes that don’t haunt them for long periods or their whole lives.”
M.E. Kabay, a professor of computer information systems at Norwich University in Vermont, said, “A dictatorship like that in Orwell’s 1984 would love to have control over the algorithms selecting information for the public or for subsectors of the public. If information is power, then information control is supreme power. Warning bells should sound when individualized or group information bubbles generated by the selective algorithms diverge from some definition of reality. Supervisory algorithms should monitor assertions or information flows that deviate from observable reality and documentary evidence; the question remains, however, of whose reality will dominate.”
An anonymous professor at the University of California-Berkeley observed, “Algorithms are, by definition, impersonal and based on gross data and generalized assumptions. The people writing algorithms, even those grounded in data, are a non-representative subset of the population. The result is that algorithms will be biased toward what their designers believe to be ‘normal.’ One simple example is the security questions now used by many online services. E.g., what is your favorite novel? Where did your spouse go to college? What was your first car? What is your favorite vacation spot? What is the name of the street you grew up on?”
An anonymous respondent commented, “There is a lot of potential for abuse here that we have already seen in examples such as sentencing for nonviolent offences. Less-well-off and minority offenders are more likely to serve sentences or longer sentences than others whose actions were the same. We also see that there is a lot of potential for malicious behaviour similar to the abuses corrected previously when nasty neighbours would spread lies and get their victims reclassified for auto or other insurance rates.”
Theme 5: Algorithmic categorizations deepen divides
Two lines of thinking about societal divisions were embodied in many respondents’ answers. First, they predicted that an algorithm-assisted future will widen the gap between the digitally savvy, who are the most desired customers in the new information ecosystem, and disadvantage those who are not nearly as connected or able to participate. The second observation about “divides” is that social and political divisions will be abetted by algorithms, as algorithm-driven insights encourage people to live in echo chambers of repeated and reinforced media and political content. As one respondent put it, “Algorithms risk entrenching people in their own patterns of thought and like-mindedness.”
The disadvantaged are likely to be more so
Some respondents predicted that those individuals who are already being left out or disadvantaged by the digital age will fall even further behind as algorithms become more embedded in society. They noted that the capacity to participate in digital life is not universal because fast-evolving digital tools and connections are costly, complicated, difficult to maintain and sometimes have a steep learning curve. And they said algorithmic tools create databased information that categorizes individuals in ways that are often to their disadvantage.
Pete Cranston of Euroforic Services wrote, “Smart(er) new apps and platforms will require people to learn how to understand the nature of the new experience, learn how it is guided by software, and learn to interact with the new environment. That has tended to be followed by a catch-up by people who learn then to game the system, as well as navigate it more speedily and reject experiences that don’t meet expectations or needs. The major risk is that less-regular users, especially those who cluster on one or two sites or platforms, won’t develop that navigational and selection facility and will be at a disadvantage.”
Christopher Owens, a community college professor, said, “If the current economic order remains in place, then I do not see the growth of data-driven algorithms providing much benefit to anyone outside of the richest in society.”
Tom Vest, a research scientist, commented, “Algorithms will most benefit the minority of individuals who are consistently ‘preferred’ by algorithms, plus those who are sufficiently technically savvy to understand and manipulate them (usually the same group).”
These proponents argued that “upgrades” often do very little to make crucial and necessary improvements in the public’s experiences. Many are incremental and mostly aimed at increasing revenue streams and keeping the public reputations of technology companies – and their shareholder value – high. An anonymous sociologist at the Social Media Research Foundation commented, “Algorithms make discrimination more efficient and sanitized. Positive impact will be increased profits for organizations able to avoid risk and costs. Negative impacts will be carried by all deemed by algorithms to be risky or less profitable.”
Jerry Michalski, founder at REX, commented, “Algorithms are already reshaping – might we say warping? – relationships, citizenship, politics and more. Almost all the algorithms that affect our lives today are opaque, created by data scientists (or similar) behind multiple curtains of privacy and privilege. Worse, the mindset behind most of these algorithms is one of consumerism: How can we get people to want more, buy more, get more? The people designing the algorithms seldom have citizens’ best interests at heart. And that can’t end well. On the positive side, algorithms may help us improve our behavior on many fronts, offsetting our weaknesses and foibles or reminding us just in time of vital things to do. But on the whole, I’m pessimistic about algorithm culture.”
Some of these experts said that – as smart, networked devices and big data combine and allow the creation of highly detailed databased profiles of individuals that follow them everywhere and impact their transactions – people of lesser means and those with some socially questionable acts in their backgrounds will be left out, cheated or forced to come up with alternate methods by which to operate securely, safely and fairly in information networks.
Dave Howell, a senior program manager in telecommunications, said, “Algorithms will identify the humans using connected equipment. Identity will be confirmed through blockchain by comparison to trusted records of patterns, records kept by the likes of [Microsoft], Amazon, Google. But there are weaknesses to any system, and innovative people will work to game a system. Advertising companies will try to identify persons against their records, blockchains can be compromised (given a decade someone will . Government moves too slowly. The Big Five (Microsoft, Google, Apple, Amazon, Facebook) will offer technology for trust and identity; few other companies will be big enough. Scariest to me is Alibaba or China’s state-owned companies with power to essentially declare who is a legal person able to make purchases or enter contracts. Government does not pay well enough to persevere. I bet society will be stratified by which trust/identity provider one can afford/qualify to go with. The level of privacy and protection will vary. Lois McMaster [Bujold]’s Jackson’s Whole suddenly seems a little more chillingly realistic.”
Nigel Cameron, president and CEO of the Center for Policy on Emerging Technologies, observed, “Positives: Enormous convenience/cost-savings/etc. Negatives: Radically de-humanizing potential, and who writes/judges the algos? In a consensus society all would be well. But we have radically divergent sets of values, political and other, and algos are always rooted in the value systems of their creators. So the scenario is one of a vast opening of opportunity – economic and otherwise – under the control of either the likes of Zuckerberg or the grey-haired movers of global capital or ….”
Freelancer Julie Gomoll wrote, “The overall effect will be positive for some individuals. It will be negative for the poor and the uneducated. As a result, the digital divide and wealth disparity will grow. It will be a net negative for society.”
Polina Kolozaridi, a researcher at the Higher School of Economics, Moscow, wrote, “The Digital Gap will extend, as people who are good in automating their labour will be able to have more benefits.”
An anonymous associate professor observed, “Whether algorithms positively or negatively impact people’s lives probably depends on the educational background and technological literacy of the users. I suspect that winners will win big and losers will continue to lose – the Matthew effect. This is likely to occur through access to better, cheaper and more-efficient services for those who understand how to use information, and those who don’t understand it will fall prey to scams, technological rabbit holes and technological exclusion.”
An anonymous respondent wrote, “Algorithms are not neutral, and often privilege some people at the expense of those with certain marginalized identities. As data mining and algorithmic living become more pervasive, I expect these inequalities will continue.”
Another respondent wrote, “The benefits will accrue disproportionately to the parts of society already doing well – the upper middle class and above. Lower down the socioeconomic ladder, algorithmic policy may have the potential to improve some welfare at the expense of personal freedom: for example, via aggressive automated monitoring of food stamp assistance, or mandatory online training. People in these groups will also be most vulnerable to algorithmic biases, which will largely perpetuate the societal biases present in the training data. Since algorithms are increasingly opaque, it will be hard to provide oversight or prove discrimination.”
Algorithms create filter bubbles and silos shaped by corporate data collectors; they limit people’s exposure to a wider range of ideas and reliable information and eliminate serendipity
Code written to make individualized information delivery more accurate (and more monetizable for the creators of the code) also limits what people see, read and understand about the world. It can create “echo chambers” in which people see only what the algorithms determine they want to see. This can limit exposure to opposing views and random, useful information. Among the items mentioned as exemplars in these responses were the United Kingdom’s contentious vote to exit the European Union (which came to be known as “Brexit”) and the 2016 U.S. presidential election cycle. Some respondents also expressed concerns over the public’s switch in news diet from the pre-internet 20th century’s highly edited, in-depth and professionally reported content to the algorithm-driven viewing and sharing of often-less-reliable or fake news via social media outlets such as Facebook and Twitter.
Valerie Bock of VCB Consulting commented, “It has definitely come to pass that it is now more possible than ever before to curate one’s information sources so that they include only those which strike one as pleasurable. That’s a real danger, which we’re seeing the impact of in this time of the Brexit and the 2016 U.S. election season. Our society is as polarized as it has ever been. We are going to need to be disciplined about not surrendering to what the robots think we would like to see. I worry that because it will become a hassle to see stuff we don’t ‘like,’ that gradually, fewer and fewer people will see that which challenges them.”
M.E. Kabay, a professor of computer information systems at Norwich University, said, “We may be heading for lowest-common-denominator information flows. Another issue is the possibility of increasingly isolated information bubbles or echo chambers. If the algorithms directing news flow suppress contradictory information – information that challenges the assumptions and values of individuals – we may see increasing extremes of separation in worldviews among rapidly diverging subpopulations.”
Vance S. Martin, an instructional designer at Parkland College, said, “Algorithms save me time when my phone gets a sense for what I will be typing and offers suggestions, or when Amazon or Netflix recommends something based on my history. However, they also close options for me when Google or Facebook determine that I read or watch a certain type of material and then offer me content exclusively from that point of view. This narrows my field of view, my exposure to other points of view. Using history to predict the future can be useful, but overlooks past reasons, rationales and biases. For example, in the past, the U.S. based its immigration quotas on historical numbers of people who came in the past. So if in the early 1800s there was a large number of Scottish immigrants and few Italian immigrants, they would allow in more Scots, and fewer Italians. So a historical pattern leads to future exclusionary policies. So if an algorithm determines that I am male, white, middle-class and educated, I will get different results and opportunities than a female African-American, lower-class aspirant. So ease of life/time will be increased, but social inequalities will presumably become reified.”
Jan Schaffer, executive director at J-Lab, predicted, “The public will increasingly be creeped out by the nonstop data mining.”
An anonymous assistant professor at a state university said, “I worry that the use of algorithms, while not without its benefits, will do more harm than good by limiting information and opportunities. Algorithms and big data will improve health care decisions, for example, but they will really hurt us in other ways, such as their potential influence on our exposure to ideas, information, opinions and the like.”
Steven Waldman, founder and CEO of LifePosts, said, “Algorithms, of course, are not values-neutral. If Twitter thrives on retweets, that seems neutral but it actually means that ideas that provoke are more likely to succeed; if Facebook prunes your news feed to show you things you like, that means you’ll be less exposed to challenging opinions or boring content, etc. As they are businesses, most large internet platforms will have to emphasize content that prompts the strongest reaction, whether it’s true or not, healthy or not.”
The many acts of exclusion committed in the act of leveraging algorithms was a primary concern expressed by Frank Elavsky, a data and policy analyst at Acumen LLC. Outlining what he perceives to be the potential impacts of algorithmic advances over the next decade, he listed a series of points, writing, “Negative changes? Identity security. Privacy.” He included the following on the list of concerning trends he ties to the “Algorithm Age”:
• “Identity formation – people will become more and more shaped by consumption and desire.
• Racial exclusion in consumer targeting.
• Gendered exclusion in consumer targeting.
• Class exclusion in consumer targeting – see Google’s campaign to educate many in Kansas on the need for a fiber-optic infrastructure.
• Nationalistic exclusion in consumer targeting.
• Monopoly of choice – large companies control the algorithms or results that people see.
• Monopoly of reliable news – already a problem on the internet, but consumer bias will only get worse as algorithms are created to confirm your patterns of interest.”
An anonymous social scientist spoke up for serendipity. “We are mostly unaware of our own internal algorithms, which, well, sort of define us but may also limit our tastes, curiosity and perspectives,” he said. “I’m not sure I’m eager to see powerful algorithms replace the joy of happenstance. What greater joy is there than to walk the stacks in a graduate library looking for that one book I have to read, but finding one I’d rather? I’m a better person to struggle at getting 5 out of 10 New Yorker cartoons than to have an algorithm deliver 10 they’d know I get. I’m comfortable with my own imperfection; that’s part of my humanness. Efficiency and the pleasantness and serotonin that come from prescriptive order are highly overrated. Keeping some chaos in our lives is important.”
An anonymous political science professor took the idea further, posing that a lack of serendipity can kill innovative thinking. He wrote, “The first issue is that randomness in a person’s life is often wonderfully productive, and the whole purpose of algorithms seems to be to squash those opportunities in exchange for entirely different values (such as security and efficiency). A second, related question is whether algorithms kill experimentation (purposely or not); I don’t see how they couldn’t, by definition.”
Several participants in this canvassing expressed concerns over the change in the public’s information diets, the “atomization of media,” an overemphasis of extreme, ugly, weird news, and the favoring of “truthiness” over more-factual material that may be vital to understanding how to be a responsible citizen of the world.
Respondent Noah Grand commented, “Algorithms help create the echo chamber. It doesn’t matter if the algorithm recognizes certain content or not. In politics and news media it is extremely difficult to have facts that everyone agrees on. Audiences may not want facts at all. To borrow from Stephen Colbert, audiences may prefer ‘truthiness’ to ‘truth.’ Algorithms that recognize ‘engagement’ – likes, comments, retweets, etc. – appear to reward truthiness instead of truth.”
An anonymous respondent said, “I am troubled by the way algorithms contribute to the atomization of media through Facebook and the like. We are quite literally losing the discursive framework we need to communicate with people who disagree with us.”
An anonymous technician said online discourse choreographed by algorithms creates a sophomoric atmosphere. “Algorithms are just electronic prejudices, just as the big grown-up world is just high school writ large,” he wrote. “We’ll get the same general sense of everything being kind of okay, kind of sucking, and the same daily outrage story, and the same stupid commentary, except algorithms will be the responsible parties, and not just some random schmuck, and artificial intelligences composed of stacks of algorithms will be writing the stories and being outraged.”
Robert Boatright, professor of political science at Clark University, said algorithms remove necessary cognitive challenges, writing, “The main problem is that we don’t encounter information that conflicts with our prior beliefs or habits, and we’re rarely prompted to confront radically new information or content – whether in news, music, purchasing or any of the other sorts of opportunities that we are provided.”
An anonymous IT analyst noted, “Facebook, for example, only shows topics you’ve previously shown interest in on their platform to show you more of the same. You’re far less likely to expand your worldview if you’re only seeing the same narrow-minded stuff every day. It’s a vast topic to delve into when you consider the circumstances a child is born into and how it will affect individuals’ education.”
Respondents also noted that the savviest tech strategists are able to take advantage of algorithms’ features, foibles and flaws to “game the system” and “get the highest profit out of most people.”
Theme 6: Unemployment numbers will rise
In the mid-1960s an ad hoc committee of 35 scientists and social activists including Linus Pauling and several other Nobel Prize winners sent a letter to President Lyndon B. Johnson warning that in the future a “cybernation revolution” would create a “separate nation of the poor, the unskilled, the jobless.” Is 21st-century technology displacing human workers? Concern over technological employment is nothing new, but it is seen as a much more imminent threat by many experts today. McKinsey, a global consulting company, reports that “as many as 45 percent of the activities that individuals are paid to perform can be automated by adapting currently demonstrated technologies…these activities represent about $2 trillion in annual wages.” The emergence of autonomous vehicles and industrial systems is expected to eliminate many jobs, but the number of white-collar jobs is also expected to decline.
One participant in this canvassing went into detail about ways in which small human teams assisted by algorithms will be able to accomplish much more than large human teams do today, creating efficiencies and eliminating jobs in the process.
Stephen Schultz, an author and editor, wrote, “Algorithms are to the ‘white-collar’ labor force what automation is to the ‘blue-collar’ labor force. Lawyers are especially vulnerable, even more so if those with competency in computer programming start acquiring law degrees and passing legislation and rewriting the syntax of current legal code to be more easily parsed by AI. Another profession that might benefit from algorithmic processing of data is nursing. In the United States, floor nursing is one of the most stressful jobs right now, in part because floor [registered nurses] are being given higher patient loads (up to six) and at the same time being required to enter all assessment data into the EMR (electronic medical record), and then creating/revising care plans based on that data, all of which subsequently leave little time for face-to-face patient care. The nursing process consists of five stages: assessment, diagnosis, planning, implementation and evaluation. Algorithmic information processing would be most helpful in the diagnosis and evaluation stages … with self-reporting monitoring devices directly feeding into the EMR.”
Smarter, more-efficient algorithms will displace many human work activities
A number of respondents focused on the loss of jobs as the primary challenge of the Algorithm Age. They said the spread of artificial intelligence will create significant unemployment, with major social and economic implications. One respondent said that because they are “smarter, more efficient and productive and cost less … algorithms are deadly.” One predicted “potential 100% human unemployment” and another imagined “in some places, a revolution.”
Don Philip, a retired PhD lecturer, commented, “If this is improperly managed we will have a massively unemployed underclass and huge social unrest.”
Peter Brantley, director of online strategy at the University of California-Davis, criticized American capitalism and predicted “significant unrest and upheaval,” commenting, “The trend toward data-backed predictive analytics and decision-making is inevitable. While hypothetically these could positively impact social conditions, opening up new forms of employment and enhanced access and delivery of services, in practice the negative impacts of dissolution of current employment will be an uncarried social burden. Much as the costs of 1960s-80s deindustrialization were externalized to the communities which firms vacated, with no accompanying subvention to support their greater needs, so will technological factors continue to tear at the fabric of our society without effective redress, creating significant unrest and upheaval. Technological innovation is not a challenge well accommodated by the current American capitalist system.”
Seti Gershberg, executive producer and creative director at Arizona Studios, wrote, “AI and robots are likely to disrupt the workforce to a potential 100% human unemployment. They will be smarter, more efficient and productive and cost less, so it makes sense for corporations and business to move in this direction.”
An anonymous respondent wrote, “The big issue in the use of these algorithms is what the function of a ‘job’ is. If it is to keep a person participating in society and earning a living, then algorithms are deadly; they will inevitably reduce the number of people necessary to do a job. If the purpose is to actually accomplish a task (and possibly free up a human to do more-human things), then algorithms will be a boon to that new world. I worry, though, that too many people are invested in the idea that even arbitrary work is important for showing ‘value’ to a society to let that happen.”
Joe Mandese, editor-in-chief of MediaPost, wrote, “Algorithms will replace any manual-labor task that can be done better and more efficiently via an algorithm. In the short term, that means individuals whose work is associated with those tasks will either lose their jobs or will need to be retrained. In the long run, it could be a good thing for individuals by doing away with low-value repetitive tasks and motivating them to perform ones that create higher value.”
Some seek a redefined global economic system to support humanity
While some predict humans might adjust well to a jobless future, others expect that – if steps aren’t taken to adjust – an economic collapse could cause great societal stress and perhaps make the world a much more dangerous place.
Alan Cain commented, “So. No jobs, growing population, and less need for the average person to function autonomously. Which part of this is warm and fuzzy?”
An anonymous PhD candidate predicted, “Without changes in the economic situation, the massive boosts in productivity due to automation will increase the disparity between workers and owners of capital. The increase in automation/use of algorithms leads to fewer people being employed.”
An anonymous director of research at a European futures studies organization commented, “We need to think about how to accommodate the displaced labour.”
Mike Roberts, Internet Hall of Fame member and first president and CEO of ICANN, wrote, “The limits to human displacement by our own smart machines are not known or very predictable at this point. The broader question is how to redefine and reconstruct global economic systems to provide a decent quality of life for humanity.”
Polina Kolozaridi, a researcher at the Higher School of Economics, Moscow, wrote, “It is a big political question, whether different institutions will be able to share their power, not knowing – obviously – how to control the algorithms. Plenty of routine work will be automated. That will lead to a decrease in people’s income unless governments elaborate some way of dealing with it. This might be a reason for big social changes – not always a revolution, but – in some places – a revolution as well. Only regular critical discussion involving more people might give us an opportunity to use this power in a proper way (by proper I mean more equal and empowering).”
A universal basic income – a strings-free stipend awarded to everyone in a community to cover general living expenses – is one potential solution that is often mentioned in discussions of a future with fewer jobs for humans.
Paul Davis, a director who participated in this canvassing, referred to this as a “Living Wage” in his response. He wrote, “The age of the algorithm presents the opportunity to automate bias, and render Labour surplus to requirements in the economic contract with Capital. Modern Western society is built on a societal model whereby Capital is exchanged for Labour to provide economic growth. If Labour is no longer part of that exchange, the ramifications will be immense. So whilst the benefits of algorithms and automation are widespread, it is the underlying social impact that needs to be considered. If Labour is replaced, in a post-growth model, perhaps a ‘Living Wage’ replaces the salary, although this would require Capital to change the social engagement contract.”
Michael Dyer, a computer science professor at the University of California-Los Angeles who specializes in artificial intelligence, commented, “The next 10 years is transitional, but within the next 20 years AI software will have replaced workers’ jobs at all levels of education. Hopefully, countries will have responded by implementing forms of minimal guaranteed living wages and free education past K-12; otherwise the brightest will use online resources to rapidly surpass average individuals and the wealthiest will use their economic power to gain more political advantages.”
An anonymous respondent wrote, “The positives outweigh the negatives, but only if we restructure how society works. We need a societal change that accepts the dwindling availability of traditional work, or we’ll have PhDs rioting because they can’t afford to eat. Something like Basic Income will need to be implemented if increased automation is going to be a good for humanity.”
Another anonymous respondent commented, “We will see less pollution, improved human health, less economic wastage, and fewer human jobs (which must be managed by increasing state-funded welfare).”
An anonymous professor observed, “Unless there is public support for education and continued training, as well as wage and public-service support, automation will expand the number of de-skilled and lower-paying positions paired by a set of highly skilled and highly compensated privileged groups. The benefits of increased productivity will need to be examined closely.”
One respondent said time being freed up by AI doing most of the “work” for humans could be spent addressing oversight of algorithmic systems. Stewart Dickinson, digital sculpture pioneer, said, “Basic Income will reduce beholdenship to corporations and encourage participation in open-source development for social responsibility.”
A theme describing a societal challenge
Theme 7: The need grows for algorithmic literacy, transparency, and oversight
The respondents to this canvassing offered a variety of ideas about how individuals and the broader culture might respond to the algorithm-ization of life. They noted that those who create and evolve algorithms are not held accountable to society and argued there should be some method by which they are. They also argued there is great need for education in algorithm literacy, and that those who design algorithms should be trained in ethics and required to design code that considers societal impacts as it creates efficiencies.
Glenn Ricart, Internet Hall of Fame member, technologist and founder and CTO of US Ignite, commented, “The danger is that algorithms appear as ‘black boxes’ whose authors have already decided upon the balance of positive and negative impacts – or perhaps have not even thought through all the possible negative impacts. This raises the issue of impact without intention. Am I responsible for all the impacts of the algorithm I invoke, or algorithms invoked in my behalf through my choice of services? How can we achieve algorithm transparency, at least at the level needed for responsible invocation? On the positive side, how can we help everyone better understand the algorithms they choose and use? How can we help people personalize the algorithms they choose and use?”
Scott McLeod, an associate professor of educational leadership at the University of Colorado, Denver, is hopeful that the public will gain more control. “While there are dangers in regard to who creates and controls the algorithms,” he said, “eventually we will evolve mechanisms to give consumers greater control that should result in greater understanding and trust. Right now the technologies are far outpacing our individual and societal abilities to make sense of what’s happening and corporate and government entities are taking advantage of these conceptual and control gaps. The pushback will be inevitable but necessary and will, in the long run, result in balances that are more beneficial for all of us.”
It starts with algorithm literacy – this goes beyond basic digital literacy
Because algorithms are generally invisible – even often referred to as “black box” constructs, as they are not evident in user interfaces and their code is usually not made public – most people who use them daily are in the dark about how they work and why they can be a threat. Some respondents said the public should be better educated about them.
David Lankes, a professor and director at the University of South Carolina School of Library and Information Science, wrote, “There is simply no doubt that, on aggregate, automation and large-scale application of algorithms have had a net-positive effect. People can be more productive, know more about more topics than ever before, identify trends in massive piles of data and better understand the world around them. That said, unless there is an increased effort to make true information literacy a part of basic education, there will be a class of people who can use algorithms and a class used by algorithms.”
An anonymous professor at MIT observed, “[The challenge presented by algorithms] is the greatest challenge of all. Greatest because tackling it demands not only technical sophistication but an understanding of and interest in societal impacts. The ‘interest in’ is key. Not only does the corporate world have to be interested in effects, but consumers have to be informed, educated and, indeed, activist in their orientation toward something subtle. This is what computer literacy is about in the 21st century.”
Trevor Owens, senior program officer at the Institute of Museum and Library Services, agreed, writing, “Algorithms all have their own ideologies. As computational methods and data science become more and more a part of every aspect of our lives, it is essential that work begin to ensure there is a broader literacy about these techniques and that there is an expansive and deep engagement in the ethical issues surrounding them.”
Daniel Menasce, a professor of computer science at George Mason University, wrote, “Algorithms have been around for a long time, even before computers were invented. They are just becoming more ubiquitous, which makes individuals and the society at large more aware of their existence in everyday life devices and applications. The big concern is the fact that the algorithms embedded in a multitude of devices and applications are opaque to individuals and society. Consider for example the self-driven cars being currently developed. They certainly have collision-avoidance and risk-mitigation algorithms. Suppose a pedestrian crosses in front of your vehicle. The embedded algorithm may decide to hit the pedestrian as opposed to ramming the vehicle against a tree because the first choice may cause less harm to the vehicle occupants. How does an individual decide if he or she is OK with the myriad decision rules embedded in algorithms that control your life and behavior without knowing what the algorithms will decide? This is a non-trivial problem because many current algorithms are based on machine learning techniques, and the rules they use are learned over time. Therefore, even if the source code of the embedded algorithms were made public, it is very unlikely that an individual would know the decisions that would be made at run time. In summary, algorithms in devices and applications have some obvious advantages but pose some serious risks that have to be mitigated.”
An anonymous policy adviser said, “There is a need for algorithmic literacy, and to critically assess outcomes from, e.g., machine learning, and not least how this relates to biases in the training data. Finding a framework to allow for transparency and assess outcomes will be crucial. Also a need to have a broad understanding of a the algorithmic ‘value chain’ and that data is the key driver and as valuable as the algorithm which it trains.”
Alexander Halavais, director of the master’s program in social technologies at Arizona State University, said teaching these complex concepts will require a “revolutionary” educational effort. “For society as a whole, algorithmic systems are likely to reinforce (and potentially calcify) existing structures of control,” he explained. “While there will be certain sectors of society that will continue to be able to exploit the move toward algorithmic control, it is more likely that such algorithms will continue to inscribe the existing social structure on the future. What that means for American society is that the structures that make Horatio Alger’s stories so unlikely will make them even less so. Those structures will be ‘naturalized’ as just part of the way in which things work. Avoiding that outcome requires a revolutionary sort of educational effort that is extraordinarily difficult to achieve in today’s America; an education that doesn’t just teach kids to ‘code,’ but to think critically about how social and technological structures shape social change and opportunity.”
Justin Reich, executive director at the MIT Teaching Systems Lab, observed, “The advancing impact of algorithms in our society will require new forms and models of oversight. Some of these will need to involve expanded ethics training in computer science training programs to help new programmers better understand the consequences of their decisions in a diverse and pluralistic society. We also need new forms of code review and oversight, that respect company trade secrets but don’t allow corporations to invoke secrecy as a rationale for avoiding all forms of public oversight.”
People call for accountability processes, oversight, and transparency
2016 was a banner year for algorithm accountability activists. Though they had been toiling largely in obscurity, their numbers have begun to grow. Meanwhile, public interest has increased as large investments in AI by every top global technology company and breakthroughs in the design and availability of autonomous vehicles and the burgeoning of big data analytics have raised algorithm issues to a new prominence. Many respondents in this canvassing urged that new algorithm accountability, oversight and transparency initiatives be developed and deployed. After the period in which this question was open for comments by our expert group (July 1-Aug. 12, 2016), three important U.S. Government reports on AI were released and the Partnership on AI, an industry-centered working group including Amazon, Facebook, Google, IBM and Microsoft was announced (Apple joined the partnership in early 2017).
In this canvassing, Frank Pasquale, author of The Black Box Society: The Secret Algorithms That Control Money and Information and professor of law at the University of Maryland, wrote: “Empiricists may be frustrated by the ‘black box’ nature of algorithmic decision-making; they can work with legal scholars and activists to open up certain aspects of it (via freedom of information and fair data practices). Journalists, too, have been teaming up with computer programmers and social scientists to expose new privacy-violating technologies of data collection, analysis and use – and to push regulators to crack down on the worst offenders. Researchers are going beyond the analysis of extant data and joining coalitions of watchdogs, archivists, open data activists and public interest attorneys to assure a more balanced set of ‘raw materials’ for analysis, synthesis and critique. Social scientists and others must commit to the vital, long-term project of assuring that algorithms are producing fair and relevant documentation; otherwise, states, banks, insurance companies and other big, powerful actors will make and own more and more inaccessible data about society and people. Algorithmic accountability is a big-tent project, requiring the skills of theorists and practitioners, lawyers, social scientists, journalists and others. It’s an urgent, global cause with committed and mobilized experts looking for support.”
Several participants in the canvassing said the law must catch up to reality.
Lee McKnight, an associate professor at Syracuse University’s School of Information Studies, said, “Given the wide-ranging impact on all aspects of people’s lives, eventually, software liability law will be recognized to be in need of reform, since right now, literally, coders can get away with murder. Inevitably, regulation of implementation and operation of complex policy models such as [the] Dodd-Frank Volcker Rule capital adequacy standards will themselves be algorithmically driven. Regulatory algorithms, code and standards will be – actually already are – being provided as a service. The Law of Unintended Consequences indicates that the increasing layers of societal and technical complexity encoded in algorithms ensure that unforeseen catastrophic events will occur – probably not the ones we were worrying about.”
Mark Lemley, a professor of law at Stanford Law School, pointed out the urgent need to address new issues arising out of the abundance of previously unavailable data. He explained, “Algorithms will make life and markets more efficient and will lead to significant advances in health. But they will also erode a number of implicit safety nets that the lack of information has made possible. The government will need to step in, either to prevent some uses of information or to compensate for the discrimination that results.”
Tse-Sung Wu, project portfolio manager at Genentech, used emerging concerns tied to autonomous vehicles as a compelling example of the need for legal reform. He wrote, “Perhaps the biggest peril is the dissolution of accountability unless we change our laws. Who will be held to account when these decisions are wrong? Right now, it’s a person – the driver of a vehicle or, in the case of professional services, someone with professional education and/or certification (a doctor making a diagnosis and coming up with a treatment plan; a judge making a ruling; a manager deciding how to allocate resources, etc.). In each of these, there is a person who is the ultimate decision-maker, and, at least at moral level, the person who is accountable (whether they are held to account is a different question). Liability insurance exists in order to manage the risk of poor decision-making by these individuals. How will our legal system of torts deal with technologies that make decisions: Will the creator of the algorithm be the person of ultimate accountability of the tool? Its owner? Who else? The algorithm will be limited by the assumptions, world view/mental model and biases of its creator. Will it be easier to tease these out, will it be harder to hide biases? Perhaps, which would be a good thing. In the end, while technology steadily improves, once again, society will need to catch up. We live in a civilization of tools, but the one thing these tools don’t yet do is make important decisions. The legal concepts around product liability closely define the accountabilities of failure or loss of our tools and consumable products. However, once tools enter the realm of decision-making, we will need to update our societal norms (and thus laws) accordingly. Until we come to a societal consensus, we may inhibit the deployment of these new technologies, and suffer from them inadvertently.”
Patrick Tucker, author of The Naked Future, wrote, “We can create laws that protect people volunteering information such as the Genetic Information Nondiscrimination Act, (Pub.L. 110–233, 122 Stat. 881) that ensures people aren’t punished for data that they share that then makes its way into an algorithm. The current suite of encryption products available to consumers shows that we have the technical means to allow consumers to fully control their own data and share it according to their wants and needs, and the entire FBI vs. Apple debate shows that there is strong public interest and support in preserving the ability of individuals to create and share data in a way that they can control. The worst possible move we, as a society, can make right now is to demand that technological progress reverse itself. This is futile and shortsighted. A better solution is to familiarize ourselves with how these tools work, understand how they can be used legitimately in the service of public and consumer empowerment, better living, learning and loving, and also come to understand how these tools can be abused.”
Many respondents agreed it is necessary to take immediate steps to protect the public’s interests.
Sandi Evans, an assistant professor at California State Polytechnic University, Pomona, said, “We need to ask: How do we evaluate, understand, regulate, improve, make ethical, make fair, build transparency into, etc., algorithms?”
Lilly Irani, an assistant professor at the University of California-San Diego, wrote, “While algorithms have many benefits, their tendency toward centralization needs to be countered with policy. When we talk about algorithms, we sometimes are actually talking about bureaucratic reason embedded in code. The embedding in code, however, powerfully takes the execution of bureaucracy out of specific people’s hands and into a centralized controller – what Aneesh Aneesh has called algocracy. A second issue is that these algorithms produce emergent, probabilistic results that are inappropriate in some domains where we expect accountable decisions, such as jurisprudence.”
Thomas Claburn, editor-at-large at InformationWeek, commented, “Our algorithms, like our laws, need to be open to public scrutiny, to ensure fairness and accuracy.”
One anonymous respondent offered some specific suggestions, “Regarding governance: 1) Let’s start with it being mandatory that all training sets be publicly available. In truth, probably only people well-qualified will review them, but at least vested interests will be scrutinized by diverse researchers whom they cannot control. 2) Before any software is deployed it should be thoroughly tested not just for function but for values.3) No software should be deployed in making decisions that affect benefits to people without a review mechanism and potential to change them if people/patients/students/workers/voters/etc. have a legitimate concern. 4) No lethal software should be deployed without human decision-makers in control. 5) There should be a list of disclosures at least about operative defaults so that mere mortals can learn something about what they are dealing with.”
An anonymous senior fellow at a futures organization studying civil rights observed, “There must be redress procedures since errors will occur.”
Another anonymous respondent wrote, “There are three things that need to happen here: 1) A 21st-century solution to the prehistoric approach to passwords; 2) A means whereby the individual has ultimate control over and responsibility for their information; and 3) Governance and oversight of the way these algorithms can be used for critical things (like health care and finance), coupled with an international (and internationally enforceable) set of laws around their use. Solve these, and the world is your oyster (or, more likely, Google’s oyster).”
Robert Bell, co-founder of the Intelligent Community Forum, commented, “Transparency is the great challenge. As these things exert more and more influence, we want to know how they work, what choices are being made and who is responsible. The irony is that, as the algorithms become more complex, the creators of them increasingly do not know what is going on inside the black box. How, then, can they improve transparency?”
Micah Altman, director of research at MIT Libraries, noted, “The key policy question is: How [will we] choose to hold government and corporate actors responsible for the choices that they delegate to algorithms? There is increasing understanding that each choice of algorithms embody a specific set of choices over what criteria are important to ‘solving’ a problem, and what can be ignored. To incent better choices in algorithms will likely require actors using them to provide more transparency, to explicitly design algorithms with privacy and fairness in mind, and holding actors who use algorithms meaningfully responsible for their consequences.”
Timothy C. Mack, managing principal at AAI Foresight, said, “The use of attention analysis on algorithm dynamics will be a possible technique to pierce the wall of black box decisions, and great progress is being made in that arena.”
Respondents suggested a range of oversight mechanisms, including a “new branch of the [U.S. Federal Communications Commission] made up of coders” and “some kind of a rainbow coalition,” and said it must “legislate humanely the protection of both the individual and society in general.”
Mary Griffiths, an associate professor in media at the University of Adelaide in South Australia, replied, “The most salient question everyone should be asking is the classical one about accountability – ‘quis custodiet ipsos custodes?’ – who guards the guardians? And, in particular, which ‘guardians’ are doing what, to whom, using the vast collection of information? Who has access to health records? Who is selling predictive insights, based on private information, to third parties unbeknown to the owners of that information? Who decides which citizens do and don’t need additional background checks for a range of activities? Will someone with mental health issues be ‘blocked’ invisibly from employment or promotion? The question I’ve been thinking about, following UK scholar [Evelyn] Ruppert, is that data is a collective achievement, so how do societies ensure that the collective will benefit? Oversight mechanisms might include stricter access protocols; sign off on ethical codes for digital management and named stewards of information; online tracking of an individual’s reuse of information; opt-out functions; setting timelines on access; no third-party sale without consent.”
An anonymous cloud-computing architect commented, “Closed algorithms in closed organizations can lead to negative outcomes and large-scale failures. If there is not enough oversight and accountability for organizations and how they use their algorithms, it can lead to scenarios where entire institutions fail, leading to widespread collapse. Nowhere is this more apparent than in critical economic institutions. While many of these institutions are considered ‘too big to fail,’ they operate based on highly secretive and increasingly complex rules with outcomes that are focused on only single factor – short-term economic gains. The consequence is that they can lead to economic disparity, increased long-term financial risk and larger social collapse. The proper response to this risk, though, is to increase scrutiny into algorithms, make them open, and make institutions accountable for the broader social spectrum of impact from algorithmic decisions.”
An anonymous system administrator commented, “We need some kind of rainbow coalition to come up with rules to avoid allowing inbuilt bias and groupthink to effect the outcomes.”
Maria Pranzo, director of development at The Alpha Workshops, wrote, “Perhaps an oversight committee – a new branch of the FCC made up of coders – can monitor new media using algorithms of their own, sussing out suspicious programming – a watchdog group to keep the rest of us safely clicking.”
Fredric Litto, emeritus professor of communications at the University of São Paulo, Brazil, said, “If there is, built-in, a manner of overriding certain classifications into which one falls, that is, if one can opt out of a ‘software-determined’ classification, then I see no reason for society as a whole not taking advantage of it. On the other hand, I have ethical reservations about the European laws that permit individuals to ‘erase’ ‘inconvenient’ entries in their social media accounts. I leave to the political scientists and jurists (like Richard Posner) the question of how to legislate humanely the protection of both the individual and society in general.”
An anonymous postdoctoral fellow in humanities at a major U.S. university commented, “The bias of many, if not most, of the algorithms and databases governing our world are now corporate. The recent debate over whether Facebook’s News Feed algorithm is biased against conservative news in the U.S., for example, does little to address the bias Facebook has in presenting news which is likely to keep users on Facebook, using and producing data for Facebook. A democratic oversight mechanism aimed at addressing the unequal distribution of power between online companies and users could be a system in which algorithms, and the databases they rely upon, are public, legible and editable by the communities they affect.”
Lauren Wagner wrote hopefully about OpenAI, a nonprofit artificial intelligence research agency founded in December 2015 with $1 billion in funding from technologists and entrepreneurs including Sam Altman, Jessica Livingston, Elon Musk, Reid Hoffman and Peter Thiel. “Overall, artificial intelligence holds the most promise and risk in terms of impacting peoples’ lives through the expanding collection and analysis of data. Oversight bodies like OpenAI are emerging to assess the impact of algorithms. OpenAI is a nonprofit artificial intelligence research company. Their goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return.”
Some respondents said any sort of formal regulation of algorithms would not be as effective as allowing the marketplace to initiate debate that inspire improvements.
Writer Richard Oswald commented, “As the service industries use these tools more extensively, they will evolve or face discriminating use by consumers. The secret does not lie in government rules for the algorithms themselves but in competition and free choice allowing consumers to use the best available service and by allowing consumers to openly share their experiences.”
Michael Whitaker, vice president of emerging solutions at ICF International, expects the market to self-correct after public input. He wrote, “Algorithms are delivering and will continue to deliver significant value to individuals and society. However, we are in for a substantial near- to mid-term backlash (some justified, some not) that will make things a bit bumpy on the way to a more transparent future with enhanced trust and understanding of algorithm impacts. Over the next few years, scrutiny over the real-world impacts of algorithms will increase and organizations will need to defend their application. Many will struggle and some are likely to be held accountable (reputation or legal liability). This will lead to increased emphasis on algorithm transparency and bias research.”
Respondents said that, regardless of projected efficacy, attention has to be paid to the long-term consequences of algorithm development.
John B. Keller, director of eLearning at the Metropolitan School District of Warren Township, Indiana, replied, “As algorithms become more complex and move from computational-based operations into predictive operations and perhaps even into decisions requiring moral or ethical judgment, it will become increasingly important that built-in assumptions are transparent to end users and perhaps even configurable. Algorithms are not going to simply use data to make decisions – they are going to make more data about people that will become part of their permanent digital record. We must advocate for benefits of machine-based processes but remain wary, cautious and reflective about the long-term consequences of seemingly innocuous progress of today.”
An anonymous respondent wrote, “A more refined sense of the use of data and algorithms is needed and a critical eye at their outputs to make sure that they are inclusive and relevant to different communities. User testing using different kinds of groups is needed. Furthermore, a more diverse group of creators for these algorithms is needed! If it is all young white men, those who have privilege in this country, then of course the algorithms and data will serve that community. We need awareness of privilege and a more diverse group of creators to be involved.”
Many are pessimistic about the prospects for policy rules and oversight
Is any proposed oversight method really going to be effective? Many have doubts. Their thoughts primarily fall into two categories. There are those who doubt that reliable and effective oversight and regulation can exist in an environment dominated by corporate and government interests, and there are those who believe oversight will not be possible due to the vastness, never-ending growth and complexity of algorithmic systems.
T. Rob Wyatt, an independent network security consultant, wrote, “Algorithms are an expression in code of systemic incentives, and human behavior is driven by incentives. Any overt attempt to manipulate behavior through algorithms is perceived as nefarious, hence the secrecy surrounding AdTech and sousveillance marketing. If they told us what they do with our data we would perceive it as evil. The entire business model is built on data subjects being unaware of the degree of manipulation and privacy invasion. So the yardstick against which we measure the algorithms we do know about is their impartiality. The problem is, no matter how impartial the algorithm, our reactions to it are biased. We favor pattern recognition and danger avoidance over logical, reasoned analysis. To the extent the algorithms are impartial, competition among creators of algorithms will necessarily favor the actions that result in the strongest human response, i.e., act on our danger-avoidance and cognitive biases. We would, as a society, have to collectively choose to favor rational analysis over limbic instinctive response to obtain a net positive impact of algorithms, and the probability of doing so at the height of a decades-long anti-intellectual movement is slim to none.”
An anonymous respondent said, “I expect a weak oversight group, if any, which will include primarily old, rich, white men, who may or may not directly represent vested interests especially in ‘intellectual property’ groups. I also expect all sorts of subtle manipulation by the actual organizations that operate these algorithms as well as single bad actors within them, to basically accomplish propaganda and market manipulation. As well as a further promulgation of the biases that already exist within the analog system of government and commerce as it has existed for years. Any oversight must have the ability to effectively end any bad actors, by which I mean fully and completely dismantle companies, and to remove all senior and any other related staff of government agencies should they be found to be manipulating the system or encouraging/allowing systemic discrimination. There would need to be strong representation of the actual population of whatever area they represent, from socioeconomic, education, racial and cultural viewpoints. All of their proceedings should be held within the public eye.”
Dariusz Jemielniak, a professor of management at Kozminski University in Poland and a Wikimedia Foundation trustee, observed, “There are no incentives in capitalism to fight filter bubbles, profiling and the negative effects, and governmental/international governance is virtually powerless.”
John Sniadowski, a systems architect, noted that oversight is difficult if not impossible in a global setting. He wrote, “The huge problem with oversight mechanisms is that globalisation by the internet removes many geopolitical barriers of control. International companies have the resources to find ways of implementing methods to circumvent controls. The more controls are put in place, the more the probability of unintended consequences and loophole searching, the net result being more complex oversight that becomes unworkable.”
Some respondents said these complex, fast-evolving systems will be quite difficult if not impossible to assess and oversee, now and in the future.
Software engineer Joshua Segall said, “We already have the statistical tools today to assess the impact of algorithms, and this will be aided by better data collection. However, assessment will continue to be difficult regardless of algorithms and data because of the complexity of the systems we aim to study.”
An anonymous senior research scholar at a major university’s digital civil society lab commented, “This is a question of the different paces at which tech (algorithmic) innovation and regulation work. Regulation and governing of algorithms lags way behind writing them and setting them loose on ever-growing (already discriminatory) datasets. As deep learning (machine learning) exponentially increases, the differential between algorithmic capacity and regulatory understanding and its inability to manage the unknown will grow vaster.”
An anonymous respondent warned, “Who are these algorithms accountable for, once they are out in the world and doing their thing? They don’t always behave in the way their creators predicted. Look at the stock market trading algorithms, the ones that have names like ‘The Knife.’ These things move faster than human agents ever could, and collectively, through their interactions with each other, they create a non-random set of behaviors that cannot necessarily be predicted ahead of time, at time zero. How can we possibly know well enough how these interactions among algorithms will all turn out? Can we understand these interactions well enough to correct problems with algorithms when injustice invariably arises?”
Another anonymous respondent noted, “Algorithms affect quantitative factors more than relational factors. This has had a huge effect already on our society in terms of careers and in the shadow work that individuals now have to do. Algorithms are too complicated to ever be transparent or to ever be completely safe. These factors will continue to influence the direction of our culture.”
And an anonymous participant in this canvassing observed that the solution might be more algorithms: “I expect meta-algorithms will be developed to try to counter the negatives of algorithms,” he said. “Until those have been developed and refined, I can’t see there being overall good from this.”
About this Canvassing of Experts
The expert predictions reported here about the impact of the internet over the next 10 years came in response to one of eight questions asked by Pew Research Center and Elon University’s Imagining the Internet Center in an online canvassing conducted between July 1 and Aug. 12, 2016. This is the seventh Future of the Internet study the two organizations have conducted together. For this project, we invited nearly 8,000 experts and members of the interested public to share their opinions on the likely future of the internet, and 1,537 responded to at least one of the questions we asked. This report covers responses to one of five questions in the canvassing. Overall, 1,302 people responded. Some 728 of them gave answers to this follow-up question, which asked them to elaborate on their answers about the future impact of algorithms:
Algorithms will continue to have increasing influence over the next decade, shaping people’s work and personal lives and the ways they interact with information, institutions (banks, health care providers, retailers, governments, education, media and entertainment) and each other. The hope is that algorithms will help people quickly and fairly execute tasks and get the information, products, and services they want. The fear is that algorithms can purposely or inadvertently create discrimination, enable social engineering and have other harmful societal impacts.
Question: Will the net overall effect of algorithms be positive for individuals and society or negative for individuals and society?
The answer options were:
• Positives outweigh negatives
• Negatives outweigh negatives
• The overall impact will be about 50-50
Then we asked:
Please elaborate on your answer and consider addressing these issues in your response: What are the main positive changes you foresee? What are the main negative ones? What dimensions of life will be most affected – health care, consumer choice, the dissemination of news, educational opportunities, others? How will the expanding collection and analysis of data and the resulting applications of this information impact people’s lives? What kinds of predictive modeling will make life more convenient for citizens? What kinds of discrimination might occur? What kind of oversight mechanisms might be used to assess the impact of algorithms?
No matter how they answered the question, nearly all respondents pointed out some negatives of algorithm-based decision-making, sorting, work activities and other applications. Some 38% opted for the prediction that the positive impacts of algorithms will outweigh negatives for individuals and society in general, while 37% said negatives will outweigh positives, and 25% said the overall impact of algorithms will be about 50-50, positive-negative.
While many of these respondents estimate that the impact of algorithms will be negative, most of these experts assume that – no matter what drawbacks may develop – algorithm-based decision-making will continue to expand in influence and impact.
The Web-based instrument was first sent directly to a list of targeted experts identified and accumulated by Pew Research Center and Elon University during the previous six “Future of the Internet” studies, as well as those identified across 12 years of studying the internet realm during its formative years. Among those invited were people who are active in global internet governance and internet research activities, such as the Internet Engineering Task Force (IETF), Internet Corporation for Assigned Names and Numbers (ICANN), Internet Society (ISOC), International Telecommunications Union (ITU), Association of Internet Researchers (AoIR), and the Organization for Economic Cooperation and Development (OECD). We also invited a large number of professionals and policy people from technology businesses; government, including the National Science Foundation, Federal Communications Commission and European Union; think tanks and interest networks (for instance, those that include professionals and academics in anthropology, sociology, psychology, law, political science and communications); globally located people working with communications technologies in government positions; technologists and innovators; top universities’ engineering/computer science, business/entrepreneurship faculty and graduate students and postgraduate researchers; plus many who are active in civil society organizations such as Association for Progressive Communications (APC), Electronic Privacy Information Center (EPIC), Electronic Frontier Foundation (EFF) and Access Now; and those affiliated with newly emerging nonprofits and other research units examining ethics and the digital age. Invitees were encouraged to share the survey link with others they believed would have an interest in participating, thus there was a “snowball” effect as the invitees were joined by those they invited to weigh in.
Since the data are based on a non-random sample, the results are not projectable to any population other than the individuals expressing their points of view in this sample. The respondents’ remarks reflect their personal positions and are not the positions of their employers; the descriptions of their leadership roles help identify their background and the locus of their expertise.
About 80% of respondents identified themselves as being based in North America; the others hail from all corners of the world. When asked about their “primary area of internet interest,” 25% identified themselves as research scientists; 7% as entrepreneurs or business leaders; 8% as authors, editors or journalists; 14% as technology developers or administrators; 10% as advocates or activist users; 9% as futurists or consultants; 2% as legislators, politicians or lawyers; and 2% as pioneers or originators; an additional 25% specified their primary area of interest as “other.”
More than half of the expert respondents elected to remain anonymous. Because people’s level of expertise is an important element of their participation in the conversation, anonymous respondents were given the opportunity to share a description of their internet expertise or background, and this was noted where relevant in this report.
Here are some of the key respondents in this report:
Robert Atkinson, president of the Information Technology and Innovation Foundation; danah boyd, founder of Data & Society; Stowe Boyd, chief researcher at Gigaom; Marcel Bullinga, trend watcher and keynote speaker; Randy Bush, Internet Hall of Fame member and research fellow at Internet Initiative Japan; Jamais Cascio, distinguished fellow at the Institute for the Future; Barry Chudakov, founder and principal at Sertain Research and StreamFuzion Corp.; David Clark, Internet Hall of Fame member and senior research scientist at MIT; Cindy Cohn, executive director at the Electronic Frontier Foundation; Anil Dash, technologist; Cory Doctorow, writer, computer science activist-in-residence at MIT Media Lab and co-owner of Boing Boing; Judith Donath, Harvard University’s Berkman Klein Center for Internet & Society; Stephen Downes, researcher at the National Research Council of Canada; Bob Frankston, Internet pioneer and software innovator; Oscar Gandy, emeritus professor of communication at the University of Pennsylvania; Marina Gorbis, executive director at the Institute for the Future; Jon Lebkowsky, CEO of Polycot Associates; Peter Levine, professor and associate dean for research at Tisch College of Civic Life; Mike Liebhold, senior researcher and distinguished fellow at the Institute for the Future; Rebecca MacKinnon, director of Ranking Digital Rights at New America Foundation; John Markoff, author of Machines of Loving Grace: The Quest for Common Ground Between Humans and Robots and senior writer at The New York Times; Jerry Michalski, founder at REX; Andrew Nachison, founder at We Media; Frank Pasquale, author of The Black Box Society: The Secret Algorithms That Control Money and Information and professor of law at the University of Maryland; Demian Perry, director of mobile at NPR; Justin Reich, executive director at the MIT Teaching Systems Lab; Mike Roberts, Internet Hall of Fame member and first president and CEO of ICANN; Michael Rogers, author and futurist at Practical Futurist; Marc Rotenberg, executive director of the Electronic Privacy Information Center; David Sarokin, author of Missed Information: Better Information for Building a Wealthier, More Sustainable Future; Henning Schulzrinne, Internet Hall of Fame member and professor at Columbia University; Doc Searls, journalist, speaker, and director of Project VRM at Harvard University’s Berkman Klein Center for Internet & Society; Ben Shneiderman, professor of computer science at the University of Maryland; Richard Stallman, Internet Hall of Fame member and president of the Free Software Foundation; Baratunde Thurston, a Director’s Fellow at MIT Media Lab, Fast Company columnist, and former digital director of The Onion; Patrick Tucker, author of The Naked Future and technology editor at Defense One; Steven Waldman, founder and CEO of LifePosts; Jim Warren, longtime technology entrepreneur and activist; Amy Webb, futurist and CEO at the Future Today Institute; and David Weinberger, senior researcher at the Harvard Berkman Klein Center for Internet & Society.
Here is a selection of some of the institutions at which respondents work or have affiliations:
AAI Foresight, Access Now, Adobe, Altimeter, The Aspen Institute, AT&T, Booz Allen Hamilton, California Institute of Technology, Carnegie Mellon University, Center for Digital Education, Center for Policy on Emerging Technologies, Cisco, Computerworld, Craigslist, Cyber Conflict Studies Association, Cyborgology, DareDistrupt, Data & Society, Digital Economy Research Center, Digital Rights Watch, dotTBA, Electronic Frontier Foundation, Electronic Privacy Information Center, Ethics Research Group, European Digital Rights, Farpoint Group, Federal Communications Commission, Flipboard, Free Software Foundation, Future of Humanity Institute, Future of Privacy Forum, Futurewei, Gartner, Genentech, George Washington University, Georgia Tech, Gigaom, Gilder Publishing, Google, Groupon, Hack the Hood, Harvard University’s Berkman Klein Center for Internet & Society, Hewlett Packard, Human Rights Watch, IBM, InformationWeek, Innovation Watch, Institute for Ethics and Emerging Technologies, Institute for the Future, Institute of the Information Society, Intelligent Community Forum, International Association of Privacy Professionals, Internet Corporation for Assigned Names and Numbers, Internet Education Foundation, Internet Engineering Task Force, Internet Initiative Japan, Internet Society, Jet Propulsion Laboratory, Karlsruhe Institute, Kenya ICT Network, KMP Global, The Linux Foundation, Lockheed Martin, Logic Technology, MediaPost, Michigan State University, Michigan State University, Microsoft, MIT, Mozilla, NASA, National Institute of Standards and Technology, National Public Radio, National Science Foundation, Neustar, New America, New Jersey Institute of Technology, The New York Times, Nokia, Nonprofit Technology Network, NYU, OpenMedia, Oxford University’s Martin School, Philosophy Talk, Privacy International, Queensland University of Technology, Raytheon BBN, Red Hat, Rensselaer Polytechnic Institute, Rice University Humanities Research Center, Rochester Institute of Technology, Rose-Hulman Institute of Technology, Semantic Studios, Singularity University, Social Media Research Foundation, Spacetel, Square, Stanford University Digital Civil Society Lab, Syracuse University, Tech Networks of Boston, Telecommunities Canada, Tesla Motors, U.S. Department of Defense, U.S. Ignite, UCLA, UK Government Digital Service, Unisys, United Steelworkers, University of California-Berkeley, University of California-Irvine, University of California-Santa Barbara, University of Copenhagen, University of Michigan, University of Milan, University of Pennsylvania, University of Toronto, Vodaphone, We Media, Wired, Worcester Polytechnic Institute, Yale University, York University.
To read participants’ for-credit written responses with no analysis, click here.
To read anonymous survey participants’ responses with no analysis, click here.