Guilty Until the Algorithm Says Otherwise
How Predictive Governance Outran the Law That Was Built to Contain It
In 2013, a Wisconsin judge sentenced Eric Loomis to six years in prison. Loomis had been convicted of fleeing police and operating a stolen vehicle, serious charges, but not crimes that automatically carry a six-year term. The judge, however, had something on the bench that Loomis’s attorney did not have access to: a risk score generated by a software system called Correctional Offender Management Profiling for Alternative Sanctions, or COMPAS. The score said Loomis was high risk. The judge cited it explicitly in sentencing. When Loomis’s legal team tried to examine what factors had produced the score, what weight each factor carried, and whether the model’s underlying logic was valid, they hit a wall. The software was proprietary. The company behind it declined to disclose its methodology. The state could act on the number. The defendant could not examine it.
This was not a story about a rogue judge or a broken system. The Wisconsin Supreme Court later reviewed the case and found no constitutional violation, though it did require that risk scores not be used as the determinative factor in sentencing and that courts receive a cautionary warning about the limits of such tools. A warning label on a black box, in other words. The system continued to operate. Loomis served his sentence. COMPAS continued to be used in courtrooms across the country.
What the Loomis case exposed was something more structural and harder to name than judicial error. It marked a point where a foundational operating principle of Western governance, the principle that the state may only act on what a specific person has actually done, was quietly displaced by another: that the state may act on what a statistical model predicts a person is likely to do. That shift was not debated in a legislature. It was not announced as a constitutional turning point. It entered through procurement.
To understand why that matters, it helps to understand what was there before. The writ of habeas corpus, codified in English law through the Habeas Corpus Act of 1679, established a principle that later legal scholars would describe as one of the central bulwarks against arbitrary detention: no person should be held by the state without lawful justification for that specific detention. You cannot imprison someone because they seem dangerous. You cannot imprison someone because people like them have historically caused harm. You must show cause specific to that person, grounded in what that person has done or is credibly alleged to have done.
This legal architecture was not accidental. It was a deliberate response to centuries of experience with what happens when states act on probability and suspicion rather than evidence. General warrants, arbitrary confinement, and detention on mere accusation shaped the common law tradition’s deep suspicion of unbounded state discretion. Procedural due process extended the same logic into modern administration. Before the state takes something from you, liberty, property, or a government benefit, it is supposed to give notice, a meaningful opportunity to contest the decision, and a reason that can be explained and challenged.
Those protections were designed for a specific kind of actor: a human official who reasoned their way to a conclusion. The judge, caseworker, or administrator could be wrong, biased, or cruel, but the logic was at least in principle available for inspection. It could be questioned. It could be appealed. It could be reversed. A software system that returns a score without exposing the reasoning that produced it fits awkwardly into that framework. It offers an output, not an account.
Predictive analytics entered American governance not through the front door of legislation but through the side door of vendor contracts, pilot programs, and administrative modernization efforts. No Congress passed a law authorizing algorithmic risk scores as the basis for criminal sentencing. No legislature staged a grand debate over whether Medicaid recipients should have benefits determined by opaque statistical systems. These tools arrived the way most government technology arrives: as things agencies purchased in the name of efficiency, consistency, or modernization.
At the time of the Brennan Center’s review, the New York City Police Department had not maintained audit logs of predictive policing outputs and had not saved individual predictions in a way that allowed outside review. That detail matters because accountability requires records. Oversight is impossible when a system leaves no durable trail of what it predicted, who it flagged, or how its outputs shaped decisions. The accountability structure needs something concrete to inspect. When the system runs and vanishes behind the dashboard, accountability has nothing to reach.
The legal system has struggled to know what to do with this. Courts are designed to evaluate discrete acts: a search, an arrest, a sentencing decision, a benefits termination. They are not designed to determine whether the statistical assumptions inside a proprietary model are valid, whether the training data embeds old discrimination, or whether the tool performs materially better than simpler alternatives. In Loomis, the Wisconsin Supreme Court acknowledged those concerns and still allowed the tool’s use under caution. The court did not so much resolve the tension as sidestep it.
That is what makes the governance problem deeper than a debate over accuracy. The gap is not simply that oversight has not caught up. The gap is that the accountability structure was built to evaluate reasoning, while predictive systems often offer correlation. The law asks for explanation. The software offers a score. Those are not the same thing.
History has seen this logic before. The language was different, the institutions were different, and the machinery was far cruder, but the underlying confidence was familiar: that population-level statistical patterns can justify state action against individuals before harm has occurred. In the early twentieth century, the American eugenics movement argued that science had made it possible to identify people likely to become social burdens. Legislatures responded by authorizing forced sterilization laws in dozens of states. The mechanism was actuarial. Individuals were treated as carriers of group risk.
When Buck v. Bell reached the Supreme Court in 1927, Justice Oliver Wendell Holmes Jr. wrote the most infamous line in the case: ‘three generations of imbeciles are enough.’ The opinion did not read like the rant of a crank. It read like the work of a jurist persuaded that technical expertise and public welfare justified pre-emptive state action. The danger did not announce itself as lawless excess. It arrived dressed as rational administration.
That history matters not because predictive governance today is identical to eugenics. It is not. The institutional forms differ, the legal environment differs, and the ambitions differ. The point is narrower and more uncomfortable. American governance has repeatedly shown a willingness to treat statistical classification as a sufficient basis for acting on individuals, especially when the classification arrives under the banner of expertise. The accountability structure has often responded after harm accumulated, not before deployment.
A more recent example makes the pattern easier to see. In 2016, Arkansas deployed an algorithmic system based on Resource Utilization Groups, or RUGs, to determine how many home care hours Medicaid recipients with disabilities would receive. The system replaced individualized human assessment with a statistical model. For many recipients, the cuts were immediate and severe. Donna Ray, a woman with cerebral palsy, had her in-home care hours reduced from 56 per week to 32. Others saw similar reductions, often with little or no explanation.
When the system was challenged in court, the state’s own witnesses could not fully explain why the algorithm generated the outputs it did. That point deserves more attention than it usually gets. The state had delegated a consequential judgment to a model it could not meaningfully explain. A federal court found that the process violated due process. Arkansas later paid a settlement, and more than eleven thousand people had been affected. Once again, the accountability gap closed retrospectively, after the system had already acted on thousands of lives.
Seen together, these cases reveal a recurring pattern. The tool enters through procurement. It operates with limited transparency. It is defended as more consistent than human discretion. Legal scrutiny arrives only after concrete harm appears, and even then the scrutiny is usually applied through doctrines designed for a different kind of decision-maker. The result is not merely an oversight lag. It is a mismatch between what predictive systems do and what legal accountability was built to evaluate.
That mismatch now stretches across criminal justice, public benefits, and policing. Risk assessment tools influence bail, sentencing, parole, and supervision decisions. Algorithmic systems help determine disability support, unemployment eligibility, and nutrition assistance. Predictive policing systems and risk lists flag individuals or neighborhoods for heightened attention before a new offense has occurred. In each domain, probability begins to function as a practical substitute for evidence.
The appeal of this shift is easy to understand. Human decision-makers are inconsistent. They carry personal prejudice, institutional bias, selective memory, and fatigue. A model seems, at least at first glance, like an antidote to that arbitrariness. If the same inputs produce the same outputs every time, perhaps governance becomes less capricious. That hope is not irrational. In some narrow settings, it may even be partly true.
But the stronger consistency of a model does not answer the harder question. The question is not only whether the system is more accurate than the humans it supplements or replaces. The question is what kind of justification the state owes the people it acts upon. Insurance companies have used actuarial logic for generations. That makes sense when the consequence is pricing a policy. It is a very different matter when the consequence is assigning prison time, reducing home care, or increasing police scrutiny. The mathematics may rhyme. The moral status of the decision does not.
This is where the current debate often becomes evasive. Arguments about bias, calibration, false positives, and model transparency are all real and necessary. Yet they can obscure the more basic issue. Even a perfectly calibrated predictive model would not answer the foundational political question of whether statistical resemblance is a legitimate basis for coercive state action. A highly accurate prediction is still a prediction. It is not evidence that a person did the thing the system anticipates.
Researchers examining litigation over algorithmic governance have found that courts frequently apply pre-algorithmic due process standards to algorithmic decisions. That is understandable. Courts work with the doctrinal tools they have. But it also reveals the shape of the problem. The law is being applied faithfully to a class of systems it was not built to understand. The formal checklist, notice, hearing, explanation, is often satisfied in thin ways even when the person subject to the decision still has no meaningful access to the logic that produced it.
The procurement pathway intensifies that problem. When a legislature writes a statute, at least in theory it identifies the authority being granted and the limits attached to it. Procurement often does neither. Agencies purchase capabilities that seem operationally useful, then fold those capabilities into decision-making routines before the legal and democratic implications are fully understood. By the time public scrutiny arrives, the system is already embedded in workflows, defended as necessary, and buffered by contractual claims of proprietary secrecy.
In that sense, procurement does more than speed adoption. It changes the politics of adoption. It allows a government to alter the basis of its own decision-making without having to declare that alteration openly. What looks like software acquisition can amount to a quiet revision of governing philosophy. The state moves from asking what happened to asking what is likely to happen next. It moves from adjudicating conduct to managing anticipated risk. And it does so without ever quite saying that the older standard has been displaced.
That quietness helps explain why predictive governance can expand with relatively little public debate. Most people do not experience it first as a constitutional issue. They experience it as administrative modernization, a smarter allocation system, a better dashboard, a more consistent set of outcomes. The rhetoric is managerial. The consequences are political. A society can therefore drift into pre-emptive governance while still speaking the language of due process and fairness, much as earlier forms of technocratic administration claimed neutrality while reorganizing power underneath familiar words.
The comparison to eugenics is useful here only if it is handled carefully. The point is not that every predictive system is a prelude to atrocity. The point is that technical methods have often been used to launder contested social judgments into objective-looking classifications. Dangerousness, dependency, deviance, burden, risk: these are not purely scientific facts waiting to be measured. They are political judgments that become harder to contest once a system translates them into numbers. Objectivity, in such cases, can function less as a safeguard than as a shield.
That is why the most revealing question is not whether predictive systems remove bias from governance. It is whether they remove the governed person’s ability to challenge the basis on which governance acts. The systems most loudly described as reducing arbitrary human discretion are often the ones least available for human confrontation. A judge can be questioned. A caseworker can be cross-examined. A model score, especially one wrapped in trade secrecy or institutional opacity, cannot be confronted in the same way. It is presented as neutral precisely where it is least open to challenge.
None of this means predictive analytics will disappear from governance. The capabilities are too attractive, the bureaucratic incentives too strong, and the language of efficiency too persuasive. Agencies under pressure to do more with less will continue to reach for tools that promise faster triage and more consistent decisions. Courts will continue to improvise. Vendors will continue to market opacity as intellectual property. The issue is not whether these systems will remain part of public administration. The issue is what kinds of political and legal assumptions their continued use quietly normalizes.
The older legal architecture rested on a hard premise: if the state is going to act against you, it must justify itself by reference to something you specifically did, or are specifically alleged to have done. Predictive governance does not exactly violate that premise. It works by replacing it. It asks whether risk can stand in for cause, whether probability can stand in for evidence, and whether the authority of a model can stand in for the explainability that due process once demanded.
That substitution has happened piecemeal, through software procurement, pilot deployments, and administrative routine rather than constitutional confrontation. Perhaps that is why it has drawn so little sustained political attention. A public trained to notice dramatic legal rupture may not notice a philosophical shift when it arrives disguised as a dashboard upgrade. Yet the shift is real. The systems designed to make governance more rational have also made one of the oldest questions in political life harder to ask: on what basis does the state owe you an account of what it is doing to you?
Predictive systems cannot answer that question because they were not built to ask it. They were built to rank, sort, flag, and forecast. Whether those functions are compatible with the moral logic of reactive governance is still unresolved. The deeper problem may be that by the time the question becomes impossible to ignore, the answer will already be built into the machinery.
Sources and References
· Brandon L. Garrett. (2012). Habeas corpus, due process and the Suspension Clause. Virginia Law Review, 98(1). https://virginialawreview.org/articles/habeas-corpus-due-process-and-suspension-clause-study-foundations-american/ [Tier 1]
· Brennan Center for Justice. (2020, March 31). Predictive policing explained. https://www.brennancenter.org/our-work/research-reports/predictive-policing-explained [Tier 2]
· Colorado Technology Law Journal. (2019). Lessons from the COMPAS-ProPublica debate. http://ctlj.colorado.edu/wp-content/uploads/2021/02/17.1_4-Washington_3.18.19.pdf [Tier 1]
· Moss, K. et al. (2023, August 6). Arkansas Medicaid home and community based services hours cuts. Benefits Tech Advocacy Hub. https://btah.org/case-study/arkansas-medicaid-home-and-community-based-services-hours-cuts.html [Tier 2]
· Ohio State University Glenn College of Public Affairs. (2025, September 24). How do algorithmic decision-making systems used in public benefits determinations affect due process? https://glenn.osu.edu/research-and-impact/how-do-algorithmic-decision-making-systems-used-public-benefits-determinations [Tier 1]
· ProPublica. (2016). Machine bias: There’s software used across the country to predict future criminals. And it’s biased against Blacks. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing [Tier 2]
· ProPublica. (2016, July 13). Warning labels are needed for scores rating defendants’ risk of future crime. https://www.propublica.org/article/wisconsin-court-warning-labels-needed-scores-rating-risk-future-crime [Tier 2]
· Rutherford, A. (1990). Eugenics: Past, present, and the future. BMJ / PubMed. https://pubmed.ncbi.nlm.nih.gov/1928094/ [Tier 1]
· The Heritage Foundation. (n.d.). The Habeas Corpus Clause. https://constitution.heritage.org/essays/a1-s9-c2/ [Tier 1 for constitutional summary; interpretive secondary source]
· TechTonic Justice. (n.d.). Victories. https://www.techtonicjustice.org/victories [Tier 2]
· University repository source on U.S. eugenic legislation and its model effects. https://repository.gchumanrights.org/bitstreams/4d6298c3-6150-4e78-b1d8-6b63dd065103/download [Tier 1]
Limitations and Gaps
· The essay relies on a mix of Tier 1 academic/legal sources and Tier 2 policy and investigative reporting.
· The New York City Police Department predictive policing example is anchored to the Brennan Center’s 2020 review and should be understood as a period-specific accountability example rather than a blanket present-tense claim about all NYPD systems in 2026.
· Some claims about procurement pathways are analytical inferences drawn from documented deployments rather than from a single consolidated primary source describing the full procurement history of each tool.
· The essay is intentionally scoped to a narrow tension and does not attempt a comprehensive survey of all predictive analytics vendors, all jurisdictions, or all legal scholarship on algorithmic governance.
Disclaimer
AI Disclosure: AI tools were used in research support, drafting support, and editorial organization. All claims, structure, and final language were reviewed and shaped for publication quality.
Related Reading
· Virginia Law Review, ‘Habeas Corpus, Due Process and the Suspension Clause.’
· ProPublica, ‘Machine Bias.’
· Brennan Center for Justice, ‘Predictive Policing Explained.’
· Benefits Tech Advocacy Hub, ‘Arkansas Medicaid Home and Community Based Services Hours Cuts.’
· Ohio State University Glenn College of Public Affairs, research on algorithmic decision-making and due process.