A computer can calculate a lending risk score in milliseconds. It can search a medical record faster than a physician, compare insurance options faster than an agent, and scan financial transactions at a scale no compliance team could match.

Then something goes wrong.

A patient is denied treatment. An insurance applicant is placed in the wrong risk category. A family receives an unexplained credit rejection. At that moment, the first question is rarely, “Which model architecture produced this result?”

People ask, “Who made this decision?”

That question reveals the behavioral puzzle at the center of automation. Artificial intelligence (AI) may perform much of the technical work, but human cooperation has always required more than technical performance. It requires identifiable responsibility, trusted relationships, social standing, judgment, and some way to challenge a harmful decision.

Direct answer: Automation can replace calculations, classification, document review, and other tasks. It does not remove the need for humans because high-stakes systems still require identifiable accountability, contextual judgment, trusted appeals, and someone with the authority to repair harm. These functions depend on social and institutional relationships that machines do not independently possess.

Automation removes tasks from people more easily than it removes responsibility from human relationships and institutions.

What Does “the Tribe” Mean in an Automated Society?

The word tribe can invite romantic images of campfires and tightly bonded ancestral groups. Here it means something narrower: a network of people linked by reciprocal obligations, reputations, shared rules, and consequences.

Human sociality has deep evolutionary roots. Robert Foley argued that anthropoid primates are distinguished by sustained relationships, patterned interaction, and flexible social organization. Humans inherited and expanded this capacity, building larger networks around kinship, exchange, alliance, reputation, and cultural institutions (Foley, 1995).

John Q. Patton’s research on meat sharing in Conambo, an Indigenous community in the Ecuadorian Amazon, offers a useful example. Hunters did not distribute valuable meat through an anonymous allocation mechanism. Transfers were shaped by kinship, reciprocity, political alliance, and the expectation of future support. People tracked exchanges, remembered loyalties, and directed resources toward relationships that could matter later (Patton, 2005).

A piece of meat and an insurance decision appear to belong to different worlds. The underlying coordination problem is recognizable. Valuable resources must be allocated under uncertainty. People watch who benefits, who carries the cost, whether the process is fair, and who will stand with them after an adverse outcome.

Modern institutions scale these relationships through licenses, contracts, professional standards, audit records, appeal rights, and regulatory enforcement. These are culturally evolved methods for managing the same recurring problem: cooperation becomes fragile when nobody can be held responsible.

To understand why accountability survives automation, it helps to begin with the deeper evolutionary problem of human cooperation: people cooperate selectively when trust, reciprocity, reputation, and enforcement make cooperation sustainable.

Key terms

• Automation: The use of machines or software to complete work with reduced direct human involvement.
• Artificial intelligence (AI): Computational systems that perform tasks involving prediction, classification, language, pattern recognition, or decision support.
• Human oversight: A process in which a person has enough information, authority, time, and competence to review or change an automated outcome.
• Automation bias: The tendency to accept an automated recommendation too readily, including when contradictory evidence is available.
• Algorithm aversion: The tendency to reject an algorithm after seeing it make an error, even when it performs better than available human alternatives.
• Accountability: The obligation to explain a decision, accept consequences, and take corrective action when harm occurs.

Competing Hypotheses

Three hypotheses compete to explain why humans remain inside automated workflows.

The efficiency-substitution hypothesis: Humans remain in automated workflows because the technology is still immature. As accuracy, reliability, and explainability improve, people will gradually disappear from most decisions.

The social-accountability hypothesis: Humans will remain at consequential decision points even when machines outperform them technically, because people and institutions need an accountable agent who can interpret context, hear an appeal, repair harm, and face consequences.

The institutional-persistence hypothesis: Human involvement survives mainly because laws, professions, and organizational habits change slowly. Once regulation and culture adjust, many oversight roles will fade.

These hypotheses make different predictions. Pure substitution predicts steady movement toward machine-only decisions as performance improves. Social accountability predicts persistent human involvement around exceptions, disputes, explanations, and harm. Institutional persistence predicts major differences across jurisdictions and industries, even where the underlying technology is similar.

At the proximate level, people may resist automated decisions because they fear error, loss of control, or unfair treatment. At the evolutionary level, human social systems repeatedly rewarded attention to reputation, responsibility, reciprocity, and coalition membership.

Human demand for accountable decision-makers may also reflect culturally transmitted institutions rather than a fixed psychological requirement. Legal systems, professions, and regulatory traditions teach people to expect named responsibility, explanation, and appeal.

If the need for human involvement is mainly technological, it should decline as systems become more accurate. If it is mainly social, humans should remain concentrated around adverse decisions, exceptions, appeals, and repair even after machine performance improves.

Why Does Cooperation Require Accountability?

Reciprocal cooperation works when participants can distinguish reliable partners from unreliable ones. They need memory, reputation, and some response to defection or neglect. A person who repeatedly takes benefits while avoiding obligations becomes less attractive as a partner.

Human cooperation also depends on alliances and group boundaries, a pattern explored in The Coalitional Brain: Why Humans Default to “Us vs. Them”. The coalitional structure of human groups is especially relevant to reputation, punishment, and stabilizing cooperation inside organizations.

Insurance itself can be understood as a scaled cultural system for pooling risk, building on older patterns of food sharing and mutual support. Read The Birth and Evolution of Insurance in Human Societies for a full account of how these institutions developed.

Licenses, appeal rights, audits, and professional standards are culturally transmitted institutions that extend cooperation beyond face-to-face groups, a process that can be examined through dual inheritance theory.

Can Artificial Intelligence Be Held Accountable?

Algorithms have performance records, but they do not have reputations in the human social sense. They cannot feel shame, lose standing among peers, sacrifice personal interests to restore a relationship, or demonstrate loyalty under pressure. A company can lose trust because of an algorithm. A professional can lose a license for relying on one irresponsibly. The model itself remains outside the social exchange.

This does not make the technology useless. It explains why responsibility moves upward rather than disappearing. The person approving the system, the institution deploying it, and the professionals acting on its recommendations become part of the accountable chain.

Current governance frameworks reflect this structure. The National Institute of Standards and Technology (NIST) identifies accountability, transparency, reliability, safety, explainability, privacy, and fairness as components of trustworthy AI. Its risk framework also calls for clear human roles and responsibilities across the design, deployment, and management of AI systems (National Institute of Standards and Technology, 2023).

The World Health Organization (WHO) similarly places autonomy, safety, transparency, responsibility, and accountability among the principles governing AI in health. These requirements exist because clinical accuracy alone does not settle questions about consent, acceptable risk, access, competing values, or responsibility for harm (World Health Organization, 2021).

These are contemporary regulatory and ethical choices, rather than direct products of natural selection. Their persistence across different institutions suggests that they solve a recurring social problem. High-stakes cooperation works better when decision authority is connected to answerability.

What Are Automation Bias and Algorithm Aversion?

Automation creates two distinct human problems. The first risk is that people reject useful automation. The second is that they trust it too much.

Research on algorithm aversion shows that people may abandon an algorithm after observing it make an error, even after seeing that it performs better than a human forecaster. Human mistakes are often treated as understandable. Machine mistakes can be interpreted as evidence that the entire system is untrustworthy (Dietvorst et al., 2015).

The opposite pattern is automation bias. People may accept automated recommendations without adequately searching for contradictory information. A systematic review of clinical decision support research found automation bias across multiple settings, even though decision-support systems often improved overall performance (Goddard et al., 2012).

This creates an awkward design problem. Human involvement does not automatically produce human judgment. A reviewer who approves hundreds of machine recommendations without sufficient time, authority, or independent evidence functions as a rubber stamp.

Recent evidence reinforces that concern. A 2024 systematic review and meta-analysis examined 106 experiments comparing humans alone, AI alone, and human-AI combinations. The combined systems improved performance relative to humans working alone, on average, but performed worse than the better of the human or AI acting alone. Decision tasks showed particular difficulty. Simply putting a person and a model together did not reliably create superior judgment (Vaccaro et al., 2024).

Evidence: Human-AI combinations can improve average human performance, while still falling short of the strongest independent performer.

Interpretation: Oversight must be designed around complementary capabilities. Adding a final approval box to an automated process creates the appearance of accountability without necessarily improving the decision.

Speculation: Organizations may preserve ceremonial human involvement because it distributes blame more comfortably. The employee remains visible, while authority has quietly shifted to a model, vendor, or optimization target that the employee cannot meaningfully challenge.

Why Do Healthcare, Insurance, and Finance Still Need Human Judgment?

High-stakes systems distribute scarce resources. Healthcare allocates treatments, appointments, and clinical attention. Insurance classifies risk, determines coverage, processes claims, and affects access to care or financial protection. Finance allocates credit, prices capital, detects fraud, and influences household opportunity.

These decisions contain predictable patterns that machines can evaluate effectively. They also contain incomplete information, exceptional cases, conflicting goals, and consequences that extend beyond the variables available to the model.

Consider a prior-authorization system that flags a treatment as inconsistent with standard criteria. The machine may have correctly identified the usual rule. A clinician may know that the patient’s history makes the standard alternative dangerous. Someone must determine whether the exception is legitimate, document the reasoning, communicate with the patient and provider, and accept responsibility for the outcome.

The same structure appears in insurance. A model can identify application inconsistencies or predict claim severity. It cannot decide by itself whether the available data reflect fraud, clerical error, unusual life circumstances, or a biased data source. Those distinctions require investigation, procedural fairness, and an escalation path.

The National Association of Insurance Commissioners (NAIC) adopted a model bulletin directing insurers toward written governance, risk management, testing, documentation, and accountability for AI-supported decisions. The insurer remains responsible for outcomes, including inaccurate or unfairly discriminatory results, even when a third-party system contributed to the decision (National Association of Insurance Commissioners, 2023).

Finance provides an equally clear example. The Consumer Financial Protection Bureau (CFPB) has stated that creditors using complex algorithms must still give applicants accurate and specific reasons for adverse actions. A black-box model does not remove the obligation to explain why credit was denied (Consumer Financial Protection Bureau, 2022).

These rules preserve several features of human cooperation: a named party carries responsibility; the affected person can request an explanation; a decision can be reviewed or contested; the institution can be sanctioned or required to repair harm; and future trust depends on how the institution responds.

The machine may produce the recommendation. The social relationship remains between people and institutions.

How Does Automation Change Human Status?

Automation also changes status.

Human groups allocate prestige partly through visible competence, contribution, reliability, and control over valued resources. In Patton’s Conambo research, hunting success had potential status value, while meat transfers could help reinforce alliances. Skill mattered, but the social use of the output mattered as well (Patton, 2005).

Modern professions follow different cultural rules, but a related dynamic remains. Physicians, underwriters, financial advisers, analysts, and compliance officers gain status through knowledge and judgment. Automation can reduce the value of producing routine outputs. It may increase the value of knowing when the output is unreliable, how to investigate an exception, and how to defend a decision under review.

A professional who once gained standing by remembering more facts may increasingly gain standing by asking better questions, detecting flawed assumptions, explaining tradeoffs, and taking responsibility when the evidence is incomplete.

This transition can create resistance. Some opposition to automation may protect income or occupational territory. Some may reflect a legitimate concern that expertise will decay when people stop practicing foundational tasks. Those explanations can coexist.

A well-designed institution should therefore treat status as part of implementation. Employees asked to supervise AI need real authority, training, and recognition for identifying errors. If speed and approval volume receive rewards while overrides create inconvenience, the organization has built an incentive system for automation bias.

What Does Meaningful Human Oversight Require?

Human oversight becomes meaningful through specific decision rights and working conditions.

First, tasks should be divided by risk. Low-consequence, reversible, high-volume work can tolerate greater automation. Decisions involving health, eligibility, major financial loss, discrimination risk, or legal rights require stronger review and escalation.

Second, every consequential workflow needs an accountable owner. Responsibility should include authority to stop the process, request additional evidence, override the model, and document why.

Third, the institution should preserve a path for appeal. Trust depends less on the promise of perfect decisions than on what happens after an error. People need a credible way to be heard by someone with the power to change the outcome.

Fourth, organizations should measure the quality of human-AI interaction. Useful metrics include override rates, overturned decisions, false positives, false negatives, escalation time, repeat complaints, subgroup outcomes, reviewer disagreement, and the frequency with which employees identify model failures.

Fifth, human competence must be maintained. A reviewer who no longer understands the underlying task cannot supervise it. Periodic independent decisions, scenario testing, error review, and continued professional training reduce the risk that the human role becomes decorative.

Finally, accountability must extend to leadership and vendors. Frontline employees should not carry responsibility for models they cannot inspect, objectives they did not choose, or production pressures they cannot control.

What Should Be Automated, Reviewed, or Reserved for Humans?

Decision type	Recommended control
Repetitive, reversible, low-consequence task	High automation
Pattern detection or preliminary classification	Automation with human review
Decision affecting health, coverage, credit, or legal rights	Accountable human decision owner
Unusual case with missing or conflicting evidence	Human investigation
Adverse decision contested by the affected person	Meaningful human appeal
Systemic error or subgroup disparity	Governance and compliance escalation

What Would Change My Mind?

The social-accountability hypothesis would become weaker if several patterns emerged:

• People consistently preferred fully automated decisions in healthcare, insurance, and finance after controlling for speed, price, and accuracy.
• Removing identifiable human responsibility had no effect on trust, acceptance, appeal behavior, or willingness to continue the relationship.
• Machine-only systems handled unusual cases and contested decisions as effectively as empowered human reviewers.
• Human oversight continued to add cost without improving error detection, fairness, explanation, or recovery after harm.
• Societies broadly reassigned moral and legal responsibility to autonomous systems rather than to their owners, developers, and deployers.

We do not yet have enough evidence to assume any of these outcomes.

Key Takeaways

• Automation can remove tasks without removing social responsibility.
• Humans remain valuable in high-stakes systems when they have authority to investigate, override, explain, and repair decisions.
• Human involvement can create new risks through automation bias, complacency, and ceremonial review.
• Trust depends on identifiable responsibility, credible appeals, and visible consequences for failure.
• Status will increasingly attach to judgment, exception handling, and accountability rather than routine information production.
• The performance of the full human-AI workflow should be measured, rather than assuming that either automation or human review is inherently safer.

Automation may eventually become capable of producing more accurate recommendations than people across many domains. Accuracy will still leave a social question unanswered.

Who stands behind the decision?

Until machines can participate in relationships of obligation, reputation, repair, and consequence, humans will remain inside the circle. Their work may move away from calculation and toward judgment. Their presence may become less frequent but more consequential.

Automation removes tasks from people more easily than it removes responsibility from human relationships and institutions. The tribe will use better tools. It will still need members who can answer when those tools affect another person’s life.

References and Further Reading

Consumer Financial Protection Bureau. (2022). Consumer Financial Protection Circular 2022-03: Adverse action notification requirements in connection with credit decisions based on complex algorithms.

Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126.

Foley, R. A. (1995). The adaptive legacy of human evolution: A search for the environment of evolutionary adaptedness. Evolutionary Anthropology, 4(6), 194–203.

Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127.

National Association of Insurance Commissioners. (2023). Model bulletin on the use of artificial intelligence systems by insurers.

National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). U.S. Department of Commerce.

Patton, J. Q. (2005). Meat sharing for coalitional support. Evolution and Human Behavior, 26(2), 137–157.

Trivers, R. L. (1971). The evolution of reciprocal altruism. The Quarterly Review of Biology, 46(1), 35–57.

Vaccaro, M., Almaatouq, A., & Malone, T. W. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour, 8, 2293–2303.

World Health Organization. (2021). Ethics and governance of artificial intelligence for health: WHO guidance. World Health Organization.