AI in Finance, Law and Government

Finance, law and government share one feature that shapes everything about how AI is used in them: decisions must be justified to someone else. A lender must tell a rejected applicant why; a lawyer must stand behind every citation; a government agency must be able to explain, and defend in court, why it cut someone's benefits. This lesson covers the main AI applications in each sector, the rules that govern them, the failures that shaped those rules, and the engineering practices that make AI systems auditable.

This lesson is general education, not legal or financial advice. Rules differ by jurisdiction and change; check the sources listed and qualified counsel before acting.

The intuition: show your work

In school, a right answer without working often earned partial credit, because the teacher needed to see that you got there for the right reasons. Regulated sectors apply the same rule to algorithms. A model that predicts well but cannot show why will struggle in lending, litigation or public administration, not because accuracy doesn't matter, but because these settings require contestability: the affected person, an auditor or a judge must be able to examine the reasons and challenge them. Keep that idea in mind; most of the rules below are ways of enforcing it.

Finance

Credit scoring and fair lending

Credit scoring estimates the probability that a borrower will default. Classic scorecards are logistic regression models on binned variables; many lenders now also use gradient-boosted trees and alternative data such as cash-flow information from bank accounts.

In the United States the key legal constraints include:

• Equal Credit Opportunity Act (ECOA) and its implementing Regulation B: lenders may not discriminate on protected characteristics such as race, sex, religion, national origin, marital status or age, and must send applicants who are denied credit an adverse action notice stating the specific principal reasons. In a 2022 circular (2022-03), the Consumer Financial Protection Bureau (CFPB) stated that this duty applies even when a complex algorithm makes the decision: "the model is too complex to explain" is not a defence. The CFPB withdrew that circular, along with dozens of other guidance documents, in May 2025, but the reason-giving requirement itself is in the regulation and still applies whatever technology produces the decision.
• Fair Credit Reporting Act (FCRA): governs consumer report data, its accuracy, and notices when it is used against a consumer.
• Disparate treatment and disparate impact: treating applicants differently because of a protected characteristic is unlawful; a neutral-looking practice with a disproportionate adverse effect on a protected group can also create liability unless it serves a legitimate business need that a less discriminatory alternative could not meet. How far regulators pursue disparate impact varies over time, but private litigation and state law can still apply.

In the EU, the AI Act lists AI used to evaluate the creditworthiness of individuals, or to establish their credit score, as high-risk (fraud detection is excluded). As of November 2026, after the Digital Omnibus amendments (in force since 27 July 2026), high-risk obligations for such stand-alone uses apply from 2 December 2027. The General Data Protection Regulation's Article 22 also gives individuals rights around decisions based solely on automated processing that significantly affect them.

Worked example: reason codes. Suppose a model's log-odds of default is

where $u$ is credit utilisation (0 to 1), $\ell$ the number of late payments in 24 months, $m$ months of credit history, and $d$ the debt-to-income ratio. The probability of default is $p = 1/(1+e^{-z})$.

Applicant: $u=0.95$, $\ell=2$, $m=178$, $d=0.37$.

$z = -3 + 2.375 + 1.4 - 1.068 + 0.74 = 0.447$, so $p = 1/(1+e^{-0.447}) \approx 0.61$. The lender's cut-off is $p < 0.20$, so the application is denied.

To generate reasons, compare each term with a reference profile, here a typical approved applicant with $u=0.30$, $\ell=0$, $m=120$, $d=0.25$ (whose $z = -2.47$, $p \approx 0.08$). The contribution of each feature is coefficient × (applicant value − reference value):

The contributions sum to $0.447 - (-2.47) = 2.917$, exactly the gap in log-odds. The two principal reasons are high utilisation and recent late payments; history length helped the applicant and is not a reason for denial. For linear models this decomposition is exact. For tree ensembles, practitioners use attribution methods such as SHAP (Shapley additive explanations), which approximate the same idea but depend on the choice of reference distribution and can be unstable, so reason codes from them must be tested for consistency.

Worked example: adverse impact ratio. Of 1,000 applicants in group A, 420 are approved (42%); of 1,000 in group B, 600 are approved (60%). The adverse impact ratio is $0.42 / 0.60 = 0.70$. A common screening heuristic, the "four-fifths rule", comes from US employment-selection guidelines and treats ratios below 0.8 as a signal for closer review. It is not a legal safe harbour in lending, but a ratio of 0.70 should trigger analysis: which features drive the gap, are any acting as proxies for a protected characteristic (ZIP code for race is the classic example), and is there a less discriminatory alternative model with similar predictive power?

Code: reasons and disparity checks

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(1)
names = ["utilization", "late_payments_24m", "months_on_file", "dti"]
n = 4000
X = np.column_stack([rng.uniform(0, 1, n), rng.poisson(0.6, n),
                     rng.integers(6, 240, n), rng.uniform(0.05, 0.6, n)])
logit = -3.0 + 2.5 * X[:, 0] + 0.7 * X[:, 1] - 0.006 * X[:, 2] + 2.0 * X[:, 3]
default = rng.random(n) < 1 / (1 + np.exp(-logit))
group = rng.random(n) < 0.4          # protected-class flag, NOT a model input

sc = StandardScaler().fit(X)
m = LogisticRegression().fit(sc.transform(X), default)
p_default = m.predict_proba(sc.transform(X))[:, 1]
approve = p_default < 0.20

def reasons(x, k=2):
    """Features pushing log-odds of default above the approved-applicant mean."""
    ref = sc.transform(X[approve]).mean(axis=0)
    contrib = m.coef_[0] * (sc.transform(x[None])[0] - ref)
    order = np.argsort(-contrib)
    return [(names[i], round(float(contrib[i]), 2)) for i in order[:k] if contrib[i] > 0]

denied = np.where(~approve)[0][0]
print("p(default)=%.2f" % p_default[denied], "reasons:", reasons(X[denied]))
air = approve[group].mean() / approve[~group].mean()
print("adverse impact ratio: %.2f" % air)

The protected attribute is kept out of the model but in the monitoring data: you cannot measure disparity without it. In this synthetic data the group flag is random, so the ratio is close to 1.0. In real data, features correlated with protected characteristics produce gaps that only this kind of check reveals.

Fraud detection

Fraud detection is one of the oldest and most successful ML applications in finance: card transactions, account takeover, money laundering, insurance claims. Its defining features are extreme class imbalance (fraud is a small fraction of transactions), adversaries who adapt, labels that arrive late (a chargeback may come weeks later), and asymmetric costs (a missed fraud loses money; a false decline loses a customer). Systems typically combine rules, gradient-boosted models and graph features (shared devices, addresses, merchants), and are evaluated on precision–recall curves and dollar-weighted metrics rather than accuracy. See Anomaly and Fraud Detection for the techniques.

Algorithmic trading

Most trading volume in major equity markets is now executed by algorithms, from execution algorithms that slice large orders to high-frequency market-making. ML is used for short-horizon prediction, execution optimisation and risk. The defining risks are speed and feedback:

• The 6 May 2010 "Flash Crash", in which US equity indices fell roughly 9% and largely recovered within about half an hour, showed how automated selling and liquidity withdrawal can interact.
• On 1 August 2012, a faulty deployment at Knight Capital activated old code that sent millions of unintended orders; the firm lost about 460 million US dollars in roughly 45 minutes. In 2013 the US Securities and Exchange Commission charged it under the Market Access Rule, which requires pre-trade risk controls, and Knight paid a 12 million dollar penalty.

Regulation therefore focuses on controls: pre-trade limits, kill switches, testing of algorithms before deployment, and (in the EU, under MiFID II) explicit requirements for firms engaged in algorithmic trading. The Knight case is as much a lesson in deployment engineering as in finance.

Model risk management

US banking supervisors set out expectations for model risk management in supervisory guidance issued in April 2011 by the Federal Reserve (SR 11-7) and the Office of the Comptroller of the Currency (Bulletin 2011-12), later adopted by the FDIC. For fifteen years it was the reference text; this section describes its framework because it shaped practice everywhere. It defines a model broadly as a quantitative method that turns inputs into estimates, and model risk as the potential for adverse consequences from decisions based on incorrect or misused models. Its core elements are:

1. Sound development, implementation and use: documented purpose, data, assumptions and limitations.
2. Validation: evaluation of conceptual soundness, ongoing monitoring, and outcomes analysis such as back-testing, performed with effective challenge, meaning critical review by people who are independent, competent and have the influence to force changes.
3. Governance: a model inventory, policies, defined roles, and board and senior management oversight.

In April 2026 the three agencies replaced it with shorter, principles-based Revised Guidance on Model Risk Management (Federal Reserve SR 26-2). The revision narrows the definition of a model, scales oversight to each model's materiality, focuses on larger banks, keeps a reformulated effective challenge, and explicitly leaves generative and agentic AI outside its scope, pointing banks to their broader risk management for those. The SR 11-7 pattern nonetheless remains the template for AI governance in banks worldwide, and it applies naturally to ML and LLM systems: inventory them, tier them by risk, validate independently, and monitor.

Law

E-discovery and technology-assisted review

In litigation, parties must produce relevant documents, often millions of emails and files. Technology-assisted review (TAR), also called predictive coding, trains a classifier on attorney-coded examples and ranks the rest by likely relevance. In Da Silva Moore v. Publicis Groupe (S.D.N.Y., 2012), Magistrate Judge Andrew Peck issued what is widely cited as the first judicial opinion approving TAR in an appropriate case. The central question in TAR protocols is recall: what share of the relevant documents did the process find?

Worked example: estimating recall. A review of a 1,000,000-document collection produces 60,000 documents, of which reviewers confirm 48,000 are responsive. The remaining 940,000 are the "null set". The team draws a random sample of 1,500 from the null set and finds 6 responsive documents, an elusion rate of $6/1{,}500 = 0.004$. Estimated responsive documents missed: $0.004 \times 940{,}000 = 3{,}760$. Estimated recall:

Sampling uncertainty matters: an exact 95% confidence interval for 6 of 1,500 runs from about 0.15% to 0.87%, which translates to recall between about 0.85 and 0.97. Negotiated protocols specify these sampling and validation steps in advance.

Contract review and drafting

ML and LLM tools extract clauses, flag deviations from a playbook, summarise obligations and draft first versions. The main risks are confident errors on unusual clauses, missed cross-references and confidentiality. Good practice is to measure extraction accuracy on the firm's own document types and to keep a lawyer responsible for every output that reaches a client.

Hallucinated citations and court sanctions

LLMs can produce citations to cases that do not exist, with plausible names, courts and quotations. Courts have responded firmly:

• Mata v. Avianca (S.D.N.Y., 2023): lawyers filed a brief citing non-existent cases produced by ChatGPT and, when questioned, asked the chatbot to confirm they were real. Judge P. Kevin Castel sanctioned the lawyers and their firm, including a 5,000 dollar penalty.
• Park v. Kim (2d Cir., 2024): the federal appeals court referred an attorney to its grievance panel after a brief cited a non-existent decision generated by ChatGPT.
• Michael Cohen's case (S.D.N.Y., 2023–2024): fictitious citations generated by Google Bard ended up in a motion filed by Cohen's lawyer; the judge declined to impose sanctions while criticising the failure to check the citations.
• Johnson v. Dunn (N.D. Ala., 2025): lawyers from a large firm who filed motions with fabricated citations generated by ChatGPT were publicly reprimanded, disqualified from the case, and ordered to share the opinion with their clients, opposing counsel and the judges in their other pending cases.

The American Bar Association's Formal Opinion 512 (2024) applies existing duties of competence, confidentiality, communication, candour to the tribunal and reasonable fees to generative AI use, and many courts have issued standing orders requiring disclosure or certification of AI use. The engineering lesson is that retrieval-grounded tools reduce but do not eliminate fabrication: every citation must be checked against an authoritative source before filing.

Government and the public sector

Governments use AI for benefits administration, fraud detection, tax compliance, inspections targeting, translation, call centres and document processing. Public-sector failures are especially damaging because people cannot choose another provider and errors hit those with the least capacity to fight back.

• Australia's "Robodebt" scheme (2016–2019) used an automated rule that averaged annual tax-office income across fortnights to infer overpayments of welfare benefits, then demanded repayment. It was not machine learning, which is the point: simple automation, without legal grounding and human review, produced hundreds of thousands of flawed debts. A Royal Commission reported in 2023 and condemned the scheme as both unfair and unlawful; a federal court had already found in 2019 that debts raised by income averaging alone were not validly made.
• The Netherlands childcare benefits scandal: tax authorities wrongly accused tens of thousands of families of fraud, and a risk-classification model used nationality as a risk factor. The government resigned in January 2021, and the Dutch data protection authority later fined the tax administration for discriminatory processing.
• Michigan's MiDAS unemployment system (2013–2015) automatically issued fraud determinations; a large majority of those later reviewed were found to be wrong, and the state settled a class action.

The common pattern: automated decisions at scale, a presumption of fraud, weak notice and appeal routes, and no one checking error rates.

Transparency and procurement. Responses include public algorithm registers (Amsterdam and Helsinki launched early ones in 2020; the UK publishes records under its Algorithmic Transparency Recording Standard), AI use-case inventories published by US federal agencies, impact assessments before deployment, and procurement terms that require vendors to provide documentation, testing access, audit rights, data portability and incident reporting. In the EU, AI used to evaluate eligibility for essential public benefits and services is high-risk under the AI Act, and public bodies deploying high-risk systems must carry out a fundamental-rights impact assessment.

Auditability and explainability

Across all three sectors, regulators and courts ask for the same things. Treat them as engineering requirements:

Explainability of the model and justification of the decision are different things. A post-hoc attribution explains what the model did; whether that reasoning is lawful and fair is a separate judgement. Where the stakes are highest, many teams choose inherently interpretable models (scorecards, monotonic gradient boosting, generalised additive models) and accept a small accuracy cost.

Common pitfalls

• Proxy discrimination: dropping the protected attribute does not prevent discrimination when other features encode it.
• Unstable explanations: attribution methods that give different reasons for near-identical applicants undermine adverse action notices.
• Validating the model, not the system: Knight Capital's loss came from deployment and configuration, not from a bad prediction.
• Trusting LLM citations or quotations without checking the primary source.
• Automating suspicion: treating a risk score as proof in benefits or fraud settings, and reversing the burden of proof onto citizens.
• No record of what the model saw: without decision logs you cannot answer a complaint, an audit or a court.

In practice

• Keep a model inventory, tier models by impact, and require independent validation before deployment, following the SR 11-7 pattern even outside banking.
• Generate and test reason codes as part of model development, not as an afterthought.
• Monitor approval, error and disparity rates by group, and document the search for less discriminatory alternatives.
• In legal work, check every citation in an authoritative database and follow your court's AI rules.
• In public-sector projects, publish what the system does, keep humans accountable for adverse decisions, measure error rates before scaling, and make appeals easy.
• Build the audit trail first; it is far cheaper than reconstructing decisions after a complaint.