Using Artificial Intelligence and Algorithms

 

Using Artificial Intelligence and Algorithms

 

By: Andrew Smith, Director, FTC Bureau of Consumer Protection

April 8, 2020

Republication

 

AI, Algorithms, Consumer, Privacy, Data, Bogus “Likes”, Chatbot, Facial Recognition, Voice-cloning Technologies, Title VII, Disparate Impact

 

How companies can manage the consumer protection risks of AI and algorithms.

 

 

 

BE TRANSPARENT.

 

Don’t deceive consumers about how you use automated tools. Oftentimes, AI operates in the background, somewhat removed from the consumer experience. But, when using AI tools to interact with customers (think chatbots), be careful not to mislead consumers about the nature of the interaction. The Ashley Madison complaint alleged that the adultery-oriented dating website deceived consumers by using fake “engager profiles” of attractive mates to induce potential customers to sign up for the dating service. And the Devumi complaint alleged that the company sold fake followers, phony subscribers, and bogus “likes” to companies and individuals that wanted to boost their social media presence. The upshot? If a company’s use of doppelgängers – whether a fake dating profile, phony follower, deepfakes, or an AI chatbot – misleads consumers, that company could face an FTC enforcement action.

 

Be transparent when collecting sensitive data. The bigger the data set, the better the algorithm, and the better the product for consumers, end of story…right? Not so fast. Be careful about how you get that data set. Secretly collecting audio or visual data – or any sensitive data – to feed an algorithm could also give rise to an FTC action. Just last year, the FTC alleged that Facebook misled consumers when it told them they could opt in to facial recognition – even though the setting was on by default. As the Facebook case shows, how you get the data may matter a great deal.

 

If you make automated decisions based on information from a third-party vendor, you may be required to provide the consumer with an “adverse action” notice. Under the FCRA, a vendor that assembles consumer information to automate decision-making about eligibility for credit, employment, insurance, housing, or similar benefits and transactions, may be a “consumer reporting agency.” That triggers duties for you, as the user of that information. Specifically, you must provide consumers with certain notices under the FCRA. Say you purchase a report or score from a background check company that uses AI tools to generate a score predicting whether a consumer will be a good tenant. The AI model uses a broad range of inputs about consumers, including public record information, criminal records, credit history, and maybe even data about social media usage, shopping history, or publicly-available photos and videos. If you use the report or score as a basis to deny someone an apartment, or charge them higher rent, you must provide that consumer with an adverse action notice. The adverse action notice tells the consumer about their right to see the information reported about them and to correct inaccurate information.

 

EXPLAIN YOUR DECISION TO THE CONSUMER.

 

If you deny consumers something of value based on algorithmic decision-making, explain why. Some might say that it’s too difficult to explain the multitude of factors that might affect algorithmic decision-making. But, in the credit-granting world, companies are required to disclose to the consumer the principal reasons why they were denied credit, and it’s not good enough simply to say “your score was too low” or “you don’t meet our criteria.” You need to be specific (e.g., “you’ve been delinquent on your credit obligations” or “you have an insufficient number of credit references”). This means that you must know what data is used in your model and how that data is used to arrive at a decision. And you must be able to explain that to the consumer. If you are using AI to make decisions about consumers in any context, consider how you would explain your decision to your customer if asked.

 

If you use algorithms to assign risk scores to consumers, also disclose the key factors that affected the score, rank ordered for importance. Similar to other algorithmic decision-making, scores are based on myriad factors, some of which may be difficult to explain to consumers. For example, if a credit score is used to deny someone credit, or offer them less favorable terms, the law requires that consumers be given notice, a description of the score (its source, the range of scores under that credit model), and at least four key factors that adversely affected the credit score, listed in the order of their importance based on their effect on the credit score.

 

If you might change the terms of a deal based on automated tools, make sure to tell consumers. More than a decade ago, the FTC alleged that subprime credit marketer CompuCredit violated the FTC Act by deceptively failing to disclose that it used a behavioral scoring model to reduce consumers’ credit limits. For example, if cardholders used their credit cards for cash advances or to make payments at certain venues, such as bars, nightclubs, and massage parlors, they might have their credit limit reduced. The company never told consumers that these purchases could reduce their credit limit – neither at the time they signed up nor at the time they reduced the credit limit. That decade-old matter is just as important today. If you’re going to use an algorithm to change the terms of the deal, tell consumers.

 

ENSURE THAT YOUR DECISIONS ARE FAIR.

 

Don’t discriminate based on protected classes. Cavalier use of AI could result in discrimination against a protected class. A number of federal equal opportunity laws, such as ECOA and Title VII of the Civil Rights Act of 1964, may be relevant to such conduct. The FTC enforces ECOA, which prohibits credit discrimination on the basis of race, color, religion, national origin, sex, marital status, age, or because a person receives public assistance. If, for example, a company made credit decisions based on consumers’ Zip Codes, resulting in a “disparate impact” on particular ethnic groups, the FTC could challenge that practice under ECOA. You can save yourself a lot of problems by rigorously testing your algorithm, both before you use it and periodically afterwards, to make sure it doesn’t create a disparate impact on a protected class.

 

Give consumers access and an opportunity to correct information used to make decisions about them. The FCRA regulates data used to make decisions about consumers – such as whether they get a job, get credit, get insurance, or can rent an apartment. Under the FCRA, consumers are entitled to obtain the information on file about them and dispute that information if they believe it to be inaccurate. Moreover, adverse action notices are required to be given to consumers when that information is used to make a decision adverse to the consumer’s interests. That notice must include the source of the information that was used to make the decision and must notify consumers of their access and dispute rights. If you are using data obtained from others – or even obtained directly from the consumer – to make important decisions about the consumer, you should consider providing a copy of that information to the consumer and allowing the consumer to dispute the accuracy of that information.

 

ENSURE THAT YOUR DATA AND MODELS ARE ROBUST AND EMPIRICALLY SOUND.

 

If you provide data about consumers to others to make decisions about consumer access to credit, employment, insurance, housing, government benefits, check-cashing or similar transactions, you may be a consumer reporting agency that must comply with the FCRA, including ensuring that the data is accurate and up to date. You may be thinking: We do AI, not consumer reports, so the FCRA doesn’t apply to us. Well, think again. If you compile and sell consumer information that is used or expected to be used for credit, employment, insurance, housing, or other similar decisions about consumers’ eligibility for certain benefits and transactions, you may indeed be subject to the FCRA. What does that mean? Among other things, you have an obligation to implement reasonable procedures to ensure maximum possible accuracy of consumer reports and provide consumers with access to their own information, along with the ability to correct any errors. RealPage, Inc., a company that deployed software tools to match housing applicants to criminal records in real time or near real time, learned this the hard way. The company ended up paying a $3 million penalty for violating the FCRA by failing to take reasonable steps to ensure the accuracy of the information they provided to landlords and property managers.

 

If you provide data about your customers to others for use in automated decision-making, you may have obligations to ensure that the data is accurate, even if you are not a consumer reporting agency. Companies that provide data about their customers to consumer reporting agencies are referred to as “furnishers” under the FCRA. They may not furnish data that they have reasonable cause to believe may not be accurate. In addition, they must have in place written policies and procedures to ensure that the data they furnish is accurate and has integrity. Furnishers also must investigate disputes from consumers, as well as disputes received from the consumer reporting agency. These requirements are important to ensure that the information used in AI models is as accurate and up to date as it can possibly be. And, the FTC has brought actions, and obtained big fines, against companies that furnished information to consumer reporting agencies but that failed to maintain the required written policies and procedures to ensure that the information that they report is accurate.

 

Make sure that your AI models are validated and revalidated to ensure that they work as intended, and do not illegally discriminate. Again, more lessons from the world of consumer lending, where credit-grantors have been using data and algorithms for decades to automate the credit underwriting process. The lending laws encourage the use of AI tools that are “empirically derived, demonstrably and statistically sound.” This means, among other things, that they are based on data derived from an empirical comparison of sample groups, or the population of creditworthy and noncreditworthy applicants who applied for credit within a reasonable preceding period of time; that they are developed and validated using accepted statistical principles and methodology; and that they are periodically revalidated by the use of appropriate statistical principles and methodology, and adjusted as necessary to maintain predictive ability.

 

HOLD YOURSELF ACCOUNTABLE FOR COMPLIANCE, ETHICS, FAIRNESS, AND NONDISCRIMINATION.

 

Ask questions before you use the algorithm. Going back to the 2016 Big Data report, the Commission warned companies that big data analytics could result in bias or other harm to consumers. To avoid that outcome, any operator of an algorithm should ask four key questions:

• How representative is your data set?
• Does your data model account for biases?
• How accurate are your predictions based on big data?
• Does your reliance on big data raise ethical or fairness concerns?

 

Protect your algorithm from unauthorized use. If you’re in the business of developing AI to sell to other businesses, think about how these tools could be abused and whether access controls and other technologies can prevent the abuse. For instance, just last month, the FTC hosted a workshop on voice-cloning technologies. Thanks to machine learning, these technologies enable companies to use a five-second clip of a person’s actual voice to generate a realistic audio of the voice saying anything. This technology promises to help people who have lost the ability to speak, among other things, but could be easily abused if it falls into the hands of people engaged in imposter schemes. One company that is introducing this cloning technology is vetting users and running the technology on its own servers so that it can stop any abuse that it learns about.

 

Consider your accountability mechanism. Consider how you hold yourself accountable, and whether it would make sense to use independent standards or independent expertise to step back and take stock of your AI. For example, going back to the algorithm that ended up discriminating against black patients, well-intentioned employees were trying to use the algorithm to target medical interventions to the sickest patients. Outside, objective observers who independently tested the algorithm were the ones who discovered the problem. Such outside tools and services are increasingly available as AI is used more frequently, and companies may want to consider using them.

 

Ricci v. DeStefano

Civil Rights Act of 1964, Title VII ; Title VII prohibits intentional acts of employment discrimination based on race, color, religion, sex, and national origin, 42 U. S. C. §2000e–2(a)(1) (disparate treatment), as well as policies or practices that are not intended to discriminate but in fact have a disproportionately adverse effect on minorities, §2000e–2(k)(1)(A)(i) (disparate impact). Once a plaintiff has established a prima facie case of disparate impact, the employer may defend by demonstrating that its policy or practice is “job related for the position in question and consistent with business necessity.” Ibid. If the employer meets that burden, the plaintiff may still succeed by showing that the employer refuses to adopt an available alternative practice that has less disparate impact and serves the employer’s legitimate needs. §§2000e–2(k)(1)(A)(ii) and (C); all the evidence demonstrates that the City rejected the test results because the higher scoring candidates were white. Without some other justification, this express, race-based decisionmaking is prohibited; the Court held that certain government actions to remedy past racial discrimination—actions that are themselves based on race—are constitutional only where there is a “strong basis in evidence” that the remedial actions were necessary. Richmond v. J. A. Croson Co., 488 U. S. 469, 500; see also Wygant v. Jackson Bd. of Ed., 476 U. S. 267, 277; “evidentiary support for the conclusion that remedial action is warranted becomes crucial when the remedial program is challenged in court by nonminority employees; the City’s race-based rejection of the test results cannot satisfy the strong-basis-in-evidence standard; such a prima facie case—essentially, a threshold showing of a significant statistical disparity, Connecticut v. Teal, 457 U. S. 440, 446, and nothing more—is far from a strong basis in evidence that the City would have been liable under Title VII had it certified the test results. That is because the City could be liable for disparate impact discrimination only if the exams at issue were not job related and consistent with business necessity, or if there existed an equally valid, less discriminatory alternative that served the City’s needs but that the City refused to adopt. §§2000e–2(k)(1)(A), (C). Based on the record the parties developed through discovery, there is no substantial basis in evidence that the test was deficient in either respect; fear of litigation alone cannot justify the City’s reliance on race to the detriment of individuals who passed the examinations and qualified for promotions. Discarding the test results was impermissible under Title VII, and summary judgment is appropriate for petitioners on their disparate-treatment claim. If, after it certifies the test results, the City faces a disparate-impact suit, then in light of today’s holding the City can avoid disparate-impact liability based on the strong basis in evidence that, had it not certified the results, it would have been subject to disparate-treatment liability (U.S.S.Ct., 29.06.09, Ricci v. DeStefano, J. Kennedy)

Loi fédérale sur les droits civils de 1964, Titre VII : le Titre VII proscrit les actes intentionnels de discrimination sur le lieu de travail, discrimination fondée sur la race, la couleur, la religion, le sexe, et la nationalité (traitement disparate). Sont également proscrites les politiques et pratiques qui ne visent pas à discriminer mais qui ont en fait un effet adverse sur les minorités, effet adverse qui est disproportionné (impact disparate). Une fois que le demandeur a établi prima facie que son dossier relève de l’impact disparate, l’employeur peut se défendre en démontrant que sa politique ou sa pratique est en lien avec l’emploi en question et relève des nécessités de l’activité professionnelle en question. Si l’employeur apporte cette preuve, le demandeur peut encore l’emporter en démontrant que l’employeur refuse d’adopter une pratique alternative à disposition qui entrainerait un impact disparate moindre et qui servirait les besoins légitimes de l’employeur. L’ensemble des preuves démontre que la ville a rejeté les résultats de l’examen parce que les meilleurs résultats ont été obtenus par des candidats de race blanche. Sans autre justification, cette décision basée sur la race est prohibée. La Cour avait jugé que certaines actions gouvernementales pour remédier à des discriminations raciales passées (actions basées sur la race) étaient constitutionnelles seulement si elles étaient basées sur des preuves solides que les actions correctrices étaient nécessaires. Les preuves à disposition s’agissant de savoir si l’action correctrice était justifiée sont essentielles lorsque dites actions sont contestées en justice par des employés qui n’appartiennent pas à la minorité. Le rejet par la ville, rejet fondé sur des raisons raciales, du résultat des examens ne satisfait pas au critère des preuves solides justifiant l’action ; un cas prima facie comme celui-ci, impliquant essentiellement la preuve d’un seuil d’une disparité statistique significative, et rien de plus, est loin de la notion de « solide preuve » que la ville aurait été responsable de discrimination selon le Titre VII si elle avait certifiée les résultats de l’examen. Cela s’explique du fait que la ville aurait été responsable de discrimination fondée sur un impact disparate seulement si les examens en question n’étaient pas liés à l’emploi et conformes aux nécessités de la profession, ou alors s’il existait une solution alternative également valable, moins discriminatoire, conforme aux besoins de la ville mais que la ville aurait refusé d’adopter. Basé sur le dossier que les parties ont constitué pendant la phase de discovery, aucune preuve solide ne permet de conclure que l’une ou l’autre des deux situations précitées étaient réalisées. La peur d’un procès ne peut à elle seule justifier une décision basée sur la race au détriment des personnes qui ont réussi l’examen et qui sont ainsi qualifiées pour des promotions. Rejeter le résultat des examens n’était pas permis selon le Titre VII, et un summary judgment est approprié en faveur des requérants s’agissant de leur requête fondée sur un traitement disparate. Si, après avoir accepté le résultat des examens, la ville fait face à une action en impact disparate, alors à la lumière des considérants de la présente espèce la ville peut éviter une responsabilité fondée sur un impact disparate en alléguant que la présente espèce reconnaît qu’en n’acceptant pas le résultat de l’examen, elle était responsable d’un traitement disparate.