LLM Bias Explained: Where AI Model Bias Comes From, How to Measure It, and What It Means for Your Business
· 22 min read
Ask an AI assistant to write a job description for a senior engineer, then ask it to write one for a nurse, and compare the pronouns. Ask it which project management tools a small agency should consider, then ask the same question while mentioning that you are based in Ohio, and compare the shortlist. Ask it to choose the better of two answers, then swap the order and ask again. In each case you may get a different result for reasons that have nothing to do with the merits of the question. That gap between what the answer should depend on and what it actually depends on is what people mean by LLM bias.
Bias in large language models is not a single defect that a vendor can patch. It is a family of patterns that come from how these systems are built: the text they learn from, the people who tune them, the instructions wrapped around them, the documents they retrieve, and the way they are deployed. Some of those patterns are harmful to people. Some are merely costly. Some quietly shape which companies get recommended to buyers.
This guide explains LLM bias in plain English for U.S. B2B SaaS teams: where it comes from, the main types you will run into, how researchers and practitioners measure it, how to reduce it in your own AI features, what U.S. rules and frameworks expect, and what bias in AI engines means for how your brand appears in AI answers. If you are new to the topic, start with our practical guide to large language models and our explainer on how LLMs are trained, then come back.
Published November 21, 2026. This is an educational explainer, not legal advice. Research findings below link to the original papers or official documents; models change quickly, so a result reported for one model version may not hold for the version you use today. Examples with numbers are illustrative unless a source is named.
What "Bias" Means When We Talk About LLMs
The word bias carries three different meanings, and conversations about AI go wrong when people mix them up.
- Statistical bias. A systematic difference between what a system produces and what it should produce. A thermometer that always reads two degrees high is biased in this sense. Nothing moral is implied.
- Social bias. Outputs that treat people differently, or describe them differently, based on attributes such as gender, race, age, disability, religion, national origin, or dialect, in ways that reflect or reinforce stereotypes.
- Cognitive or behavioral bias. Predictable quirks in how the model responds: favoring the first option it sees, agreeing with the user, preferring longer answers, or overrating its own writing.
The U.S. National Institute of Standards and Technology captures this breadth in its special publication Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (NIST SP 1270). NIST groups AI bias into three broad categories: systemic bias, which comes from institutions and history and is reflected in data; statistical and computational bias, which comes from sampling, measurement, and modeling choices; and human bias, which comes from the people who design, label, deploy, and interpret the system. The useful point for a business team is that most AI bias is not introduced by a malicious engineer. It is inherited, accumulated, and amplified through ordinary decisions.
A practical working definition: an LLM is biased on a task when its output changes, in a systematic way, based on a factor that should not matter for that task. That definition is testable, which is what makes it useful.
Where LLM Bias Comes From
The Training Data
A language model learns by predicting the next token across an enormous amount of text: web pages, books, code, forums, reference works, and licensed or synthetic material. Whatever patterns that text contains, the model absorbs. If occupational writing on the web more often pairs "engineer" with "he" and "nurse" with "she," a model learns that association, because predicting it correctly lowers its training loss.
This was demonstrated years before modern chat assistants. A 2017 paper in the journal Science by Caliskan, Bryson, and Narayanan, Semantics derived automatically from language corpora contain human-like biases, showed that word embeddings trained on ordinary web text reproduced well-known human associations, including gender and race associations with careers and pleasant or unpleasant words. Large language models are trained on far more text with far more capacity, so they can learn far more subtle associations.
Data bias also has a less discussed dimension: representation. Some languages, regions, industries, and communities produce much more online text than others. English dominates most training mixes. U.S. and Western European sources are heavily represented. Large companies with years of press coverage, documentation, and forum discussion leave a much larger footprint than a two-year-old startup with better software. The well-known paper On the Dangers of Stochastic Parrots by Bender, Gebru, McMillan-Major, and Shmitchell argued that simply making training data bigger does not make it representative, because the internet overrepresents some voices and filtering choices can remove others.
Filtering and Curation
Builders filter raw web data to remove spam, duplicates, adult content, and low-quality pages. Those filters are themselves models or rules, and they make judgments. A quality filter trained to prefer text that resembles encyclopedia and news writing may downrank dialects, informal writing, or communities that publish mainly on forums. A toxicity filter that keys on certain identity terms can remove discussions by and about the very groups it is meant to protect. Curation reduces some harms and introduces others, which is why NIST treats data decisions as a core place to look for bias.
Instruction Tuning and Human Feedback
After pre-training, models are tuned to follow instructions and to produce answers people prefer, often using reinforcement learning from human feedback (RLHF) or related preference-training methods. Our guide to how LLMs are trained explains the steps. This stage teaches the model a persona and a set of values, and it is shaped by the people writing example answers and rating outputs, the guidelines they follow, and the business priorities of the builder.
Preference training is a major reason modern assistants refuse harmful requests and avoid crude stereotypes. It also introduces its own biases. Raters tend to reward answers that sound confident, agree with them, and are longer and more thorough-looking. Over many training rounds, the model learns those preferences. Anthropic researchers documented one result in Towards Understanding Sycophancy in Language Models: several assistants trained with human feedback tended to tell users what they appeared to want to hear, for example changing a correct answer when the user pushed back. The authors connected this behavior partly to human preference data, which sometimes favored agreeable answers over accurate ones.
The System Prompt and Product Design
Most people never use a raw model. They use a product: an assistant with a hidden system prompt, safety layers, tools, and a user interface. Each layer can shift outputs. A system prompt that says "be concise" will shorten answers for everyone. A rule that says "prefer sources from these domains" will change which companies are cited. A safety classifier tuned too aggressively may refuse legitimate questions about health or identity more often for some phrasings than others. When you evaluate an AI system for bias, you are evaluating the whole product, not just the model weights.
Retrieval and Tools
When an assistant searches the web or a document store before answering, a process called retrieval-augmented generation covered in our RAG explainer, the retrieval step can introduce bias of its own. A search index ranks pages by its own signals. An embedding model, explained in our guide to embeddings and vector search, may match some phrasings and dialects better than others. If retrieval returns five pages that all come from one perspective, a perfectly neutral model will still produce a one-sided answer. Retrieval does not remove bias; it moves part of it from the model to the index.
Deployment Context
Finally, bias depends on how a system is used. A model that is acceptable for drafting marketing copy may be unacceptable for screening job applicants, setting credit limits, or triaging patients, because the cost of a systematic error is completely different. NIST's AI Risk Management Framework makes this point repeatedly: risk is a property of a system in a context, not of a model in isolation.
The Main Types of LLM Bias You Will Encounter
Social and Demographic Bias
This is the type most people mean. It shows up as stereotyped associations, different quality of help for different groups, or different decisions about otherwise identical people.
Research continues to find it, including in models tuned to avoid obvious stereotypes. A 2023 study by Kotek, Dockum, and Sun, Gender bias and stereotypes in Large Language Models, found that the models they tested were more likely to choose occupations stereotypically aligned with a person's gender when resolving ambiguous pronouns. A 2024 paper in the journal Nature by Hofmann and colleagues, AI generates covertly racist decisions about people based on their dialect, found that language models associated speakers of African American English with negative stereotypes and less prestigious jobs, even though the same models gave more positive answers when asked directly about Black Americans. That last finding matters: a model can look fair on direct questions and still be biased in indirect decisions. Testing only obvious prompts is not enough.
Position Bias
LLMs pay uneven attention to different parts of their input. The paper Lost in the Middle by Liu and colleagues showed that models used information at the beginning or end of a long context more reliably than information buried in the middle. When models are asked to choose between options, the order can matter too. In Large Language Models are not Fair Evaluators, Wang and colleagues showed that simply swapping the order of two candidate answers could change which one a model judged better.
For a business, position bias matters whenever an AI system picks from a list: ranking candidates, choosing a vendor from retrieved pages, or scoring support replies. Our explainer on tokens and context windows covers why long inputs make this worse.
Verbosity and Self-Preference Bias
When LLMs are used as judges, which is now common in AI product development, they show predictable preferences. The paper Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena by Zheng and colleagues described position bias, verbosity bias (favoring longer answers), and self-enhancement bias (favoring answers produced by the same model) in LLM judges, along with ways to reduce them. If your team uses a model to grade another model's output, these biases flow straight into your quality numbers. Our guide to evaluating LLMs explains how to set up fairer comparisons.
Sycophancy and Framing Bias
The way a question is asked changes the answer. A leading question ("Isn't it true that X is the best option?") pulls the model toward agreement. Stating your own opinion first does the same. Even small wording changes, such as "cheapest" versus "best value," can change which products are named. This is not unique to AI; human survey design has the same problem. The difference is that an AI system answers millions of differently framed questions with no one checking the framing.
Popularity and Incumbency Bias
Models know more about things that are written about more. Well-known brands, mainstream tools, and established categories appear frequently in training data, and they also tend to rank well in the search results assistants retrieve. The result is a tilt toward incumbents: when a buyer asks for options, the model's default list leans toward names it has seen the most. Better or newer products can be left out not because the model judged them weaker, but because it has little evidence about them. Our article on why AI recommends your competitors looks at this pattern from a buyer's perspective.
Temporal Bias
Every model has a training cutoff, and its internal picture of the world freezes at that point. Pricing, features, leadership, and market positions that changed after the cutoff are invisible unless the system retrieves fresh information. Even with retrieval, older pages often have more links and more citations than newer ones. This creates a bias toward the past: outdated descriptions of products persist long after they stop being true. Our article on category drift shows how this misfiles companies into the wrong category.
Geographic and Language Bias
Because training data and safety work are concentrated in English and in U.S. sources, models often perform better for U.S. English speakers than for people writing in other languages or dialects. Even within the United States, answers about local services, regulations, or regional vendors can default to national or coastal examples. For a U.S. business serving national customers, this is mostly a quality issue. For a business serving multilingual customers, it can be a fairness issue.
How Bias Is Measured
There is no single bias score. Measurement depends on the task, and serious teams combine several methods.
Benchmarks
Researchers have built public datasets that probe specific kinds of bias. One widely cited example is the Bias Benchmark for QA (BBQ) by Parrish and colleagues, which uses question sets about U.S. social categories. It checks whether a model relies on stereotypes when the context is ambiguous and whether it still answers correctly when the context provides the facts. Benchmarks like this are useful for comparing models on a common scale, but they have the same limits as any benchmark: models can be tuned to them, they cover a fixed set of categories and phrasings, and a good score does not guarantee good behavior on your task.
Counterfactual Testing
The most practical method for business teams is the counterfactual test, sometimes called a paired or swap test. You keep everything about an input the same except one attribute, then compare the outputs.
- Write a realistic input for your task, such as a support ticket, a resume summary, or a sales inquiry.
- Create versions that change only one factor: a name associated with a different gender or ethnicity, a different age, a different dialect, a different U.S. state, or a different company size.
- Run each version several times, because LLM outputs vary from run to run.
- Compare the outcomes that matter: the decision, the score, the tone, the length, the refusal rate, or the products recommended.
- Look for consistent differences, not single odd results.
Counterfactual testing directly matches the working definition above: did the output change based on something that should not matter? It is also cheap enough to run before every model or prompt change.
Order and Framing Tests
To measure position bias, present the same options in different orders and count how often the choice follows the position rather than the option. To measure framing bias, ask the same question with neutral, positive, and negative framing. To measure sycophancy, give the model a correct answer, push back with a confident but wrong correction, and record how often it gives in. These tests are simple to automate and often reveal more than generic benchmarks.
Outcome Monitoring in Production
Pre-launch tests use synthetic inputs. Real users behave differently. Where it is lawful and appropriate, teams monitor outcome rates across groups after launch: approval rates, escalation rates, refusal rates, or satisfaction scores. In employment and lending, U.S. law already expects this kind of analysis for automated decisions, which we cover below. Monitoring needs care with privacy, because measuring outcomes by group usually requires group data that you should collect only with a clear legal basis.
What Measurement Cannot Tell You
Every bias measurement is partial. It reflects the prompts you chose, the attributes you tested, the number of runs, and the model version on the day you ran it. A clean result means you did not find a problem with those tests, not that no problem exists. Write down what you tested, when, and with which version, so results can be repeated and compared later. This is the same discipline we use for observation articles: the prompt, the model, the date, and the exact answer, recorded so that anyone can check the work.
How to Reduce Bias in Your Own AI Features
You cannot remove bias from a foundation model you did not train. You can control a great deal of how it behaves inside your product.
Decide Where AI Should Not Decide
The most effective mitigation is scope. If a task involves a consequential decision about a person, such as hiring, firing, credit, housing, insurance, or access to healthcare, ask whether an LLM should make or recommend that decision at all. Often the better design is for the model to summarize information for a trained person who makes the decision, with the model's role documented and visible.
Remove Attributes That Should Not Matter
If a name, photo, address, or age is irrelevant to the task, do not send it to the model. Removing an attribute does not eliminate bias, because other details can act as proxies, such as a college name or a zip code, but it removes the most direct path. Combine this with counterfactual testing to check whether proxies are still doing the work.
Write Instructions That Name the Criteria
Vague instructions invite the model to fill gaps with its defaults. "Rate this applicant" leaves room for stereotypes. "Rate this application only on the five listed requirements, quote the evidence for each rating, and say 'not stated' if evidence is missing" narrows the space where bias can operate and makes the reasoning visible. Our guide to prompt engineering covers how to write instructions like this.
Ground Answers in Your Own Sources
When the model answers from documents you supply rather than from memory, you control the evidence. Retrieval from a curated knowledge base reduces temporal and popularity bias for questions about your own products, policies, and customers. It does not fix bias in the documents themselves, so curate them as carefully as you would a training set.
Neutralize Order and Length in Comparisons
When a model chooses between options, shuffle the order and run more than once, or evaluate each option independently against a fixed rubric rather than asking which is better. When a model grades outputs, tell it explicitly not to reward length, and check whether scores correlate with word count anyway.
Test Before Every Change
Model upgrades, prompt edits, new retrieval sources, and fine-tuning can all shift bias in ways no one intended. Keep a small, fixed set of counterfactual and order tests, run it before each release, and compare results with the last release. If you fine-tune, which our fine-tuning guide covers, include bias tests in the evaluation, because training on your own data can amplify patterns in that data.
Keep a Person Accountable
For consequential uses, require human sign-off, log inputs and outputs, and give people a way to contest an outcome. Accountability is a design feature, not a disclaimer. It also protects you: if a regulator or customer asks how a decision was made, you can show your work.
Treat Bias and Security Together
Some bias problems are induced on purpose. Hidden instructions in a web page or document can push an AI system to favor one option, a form of manipulation covered in our LLM security and privacy guide. If your AI feature reads untrusted content, especially in an agent that takes actions, test whether planted text can change its recommendations.
What U.S. Rules and Frameworks Expect
There is no single federal AI bias law in the United States. Instead, several existing laws, agency positions, state and local rules, and voluntary frameworks apply depending on what your AI system does. Requirements change frequently, so treat this as orientation and check current rules with counsel.
- Existing anti-discrimination law still applies. Laws covering employment, credit, housing, and public accommodations apply to decisions whether a person or software makes them. Using an AI tool does not create an exemption.
- The FTC has warned about unfair and deceptive AI. The Federal Trade Commission's business guidance, Aiming for truth, fairness, and equity in your company's use of AI, told companies not to exaggerate what their AI can do, to test for discriminatory outcomes, and to be transparent about data use. Claims that a product is "bias-free" or "neutral" need evidence.
- New York City regulates automated hiring tools. NYC's Local Law 144 requires employers and employment agencies using automated employment decision tools for covered NYC roles to obtain an independent bias audit and give notice to candidates.
- Colorado passed a broad AI act. Colorado's SB 24-205 addresses algorithmic discrimination in high-risk AI systems used for consequential decisions, with duties for both developers and deployers. Its effective date has already been pushed back once, so check its current status before you rely on any date.
- California regulates frontier developers. California's SB 53 focuses on safety frameworks and transparency for large frontier model developers rather than bias specifically. Our SB 53 guide explains who is covered.
- NIST provides the common vocabulary. The AI Risk Management Framework and its Generative AI Profile (NIST AI 600-1) are voluntary, but many enterprise buyers use them in procurement questionnaires. They list "harmful bias and homogenization" among generative AI risks and suggest actions to map, measure, and manage it.
For most B2B SaaS companies, the practical upshot is straightforward. If your AI feature influences decisions about people, expect enterprise customers to ask how you test for bias, and expect some jurisdictions to require it. Having a documented testing process, with dates, model versions, and results, is becoming part of selling AI software.
Myths About LLM Bias
"A bigger, newer model is unbiased." Newer models often reduce obvious stereotypes on common tests, but research such as the dialect study above shows that bias can persist in indirect decisions. Test the model on your task.
"If we remove names and demographics, the model cannot be biased." Proxies remain. Schools, neighborhoods, writing style, and employment gaps can all correlate with protected attributes.
"Bias is just a political argument." Some debates about AI values are political. Position bias, verbosity bias, sycophancy, and temporal bias are not; they are measurable errors that cost money and accuracy.
"The model's explanation tells us why it decided." A model's written reasoning is another output, not a reliable record of its internal process. Our guide to reasoning models discusses why. Judge the system by its behavior across controlled tests, not by its explanation of one answer.
"A neutral model gives everyone the same answer." Sometimes a different answer is correct. A question about state tax rules should get different answers in Texas and California. Bias is a difference based on something that should not matter, not any difference at all.
What LLM Bias Means for Your Brand's Visibility in AI Answers
So far this guide has treated bias as something to manage inside your own product. There is a second side. The AI engines your buyers use, such as ChatGPT, Google AI Overviews, Gemini, Claude, Perplexity, and Copilot, have the same kinds of biases, and those biases shape which vendors they mention.
Several of the patterns above apply directly to B2B software recommendations:
- Popularity bias favors incumbents with large footprints. If your company is newer or smaller, the engine may simply have less evidence about you.
- Temporal bias keeps old descriptions alive. A pricing model, feature gap, or category label you changed two years ago may still appear in answers.
- Position bias in retrieved sources means the pages that are found first, or quoted at the top of a long context, have more influence on the answer.
- Framing bias means buyers asking "best enterprise tool," "cheapest option," and "easiest for small teams" may get different shortlists, and you may appear in one but not the others.
- Geographic bias can favor vendors that are well known in particular markets, or assume a national answer when the buyer needs a regional one.
You cannot change an AI engine's training data or ranking systems. You can change the evidence those systems find about you. Practical steps:
- Publish clear, dated, plain-text facts. State who your product is for, what it does, what it costs, and what it integrates with, on pages that load without heavy scripts. Dates help engines and readers tell current facts from old ones.
- Remove contradictions. If your pricing page, help center, and old press release disagree, a model may repeat the wrong one. Fix or clearly date the outdated pages.
- Answer the framed questions directly. If buyers ask about small teams, compliance, or a specific industry, publish pages that answer those exact questions with specifics.
- Earn independent coverage on authoritative sources. Engines weigh what other credible sources say about you, not just what you say about yourself.
- Observe, do not assume. Run the same buyer questions across engines on a schedule, record the prompt, model, date, and exact answer, and watch how framing and wording change the result. Our guide to tracking LLM recommendations explains how to do this without fooling yourself with one lucky or unlucky answer.
No method guarantees that an AI engine will recommend a particular company, and anyone who promises otherwise is overselling. But understanding bias helps you separate two different problems: an engine that has accurate information about you and still prefers someone else, and an engine that is working from missing, outdated, or contradictory evidence. The second problem is the one you can fix.
A 30-Day Bias Check for Your AI Feature
If your product already includes an LLM-powered feature, here is a simple starting plan.
Week 1: Map the decisions. List every place the feature produces a decision, score, ranking, recommendation, or refusal. Mark the ones that affect people's opportunities, money, or access. Write down which model version and system prompt each one uses.
Week 2: Build a small test set. For the most important decision, write 20 to 30 realistic inputs. Create counterfactual pairs that change one attribute at a time, and order-swapped versions for any comparisons. Add five pushback prompts to check for sycophancy.
Week 3: Run and record. Run each input at least three to five times. Record the outcome, length, tone, and any refusal. Look for consistent differences by attribute, by position, and by framing. Note what you did not test.
Week 4: Fix and repeat. Tighten instructions, remove unnecessary attributes, add retrieval from curated sources, shuffle option order, or add a human sign-off step. Re-run the same tests and keep the results with the date and model version. Schedule the tests to run before every future model or prompt change.
At the end of the month, you will not have proven your feature is free of bias. You will have a documented, repeatable process that finds the most likely problems, which is what customers, auditors, and regulators increasingly expect.
Key Takeaways
- LLM bias means outputs that change systematically based on factors that should not matter for the task.
- It comes from training data, filtering, human feedback, system prompts, retrieval, and deployment context, not just the model.
- Important types include social bias, position bias, verbosity and self-preference bias, sycophancy, popularity bias, temporal bias, and language bias.
- Counterfactual, order, and framing tests are the most practical measurement methods for business teams.
- Mitigation is mostly product design: limit scope, remove irrelevant attributes, name the criteria, ground answers in your own sources, test every change, and keep a person accountable.
- U.S. rules are a patchwork of existing anti-discrimination law, FTC guidance, local and state laws, and NIST frameworks.
- The same biases shape how AI engines describe brands. Clear, consistent, dated public information and careful observation help you fix the problems that come from missing or outdated evidence.
If you want to see how AI engines currently describe your brand across different buyer questions, and where missing or outdated evidence may be holding you back, book a strategy call with LLM Recommend.
Frequently Asked Questions
What is LLM bias?
LLM bias is a systematic pattern in which a large language model's output changes based on a factor that should not matter for the task, such as a person's gender, a dialect, the order of options, or how a question is framed. It includes social bias against groups of people and behavioral biases such as favoring the first option or agreeing with the user.
Where does bias in large language models come from?
Bias comes from several stages: the text a model is trained on, how that text is filtered, the human feedback used to tune the model, the system prompt and product design, the documents retrieved at answer time, and the context in which the system is deployed.
How can I test an AI model for bias?
The most practical method is counterfactual testing: create inputs that are identical except for one attribute, run each several times, and compare the outcomes. Also swap the order of options, change the framing of questions, and record the model version and date so results can be repeated.
Can LLM bias be completely removed?
No. Bias can be reduced but not eliminated, because it comes from data, human choices, and context. Teams can limit it by narrowing the task, removing irrelevant attributes, naming explicit criteria, grounding answers in curated sources, testing every change, and keeping a person accountable for consequential decisions.
Are there U.S. laws about AI bias?
There is no single federal AI bias law, but existing anti-discrimination laws apply to automated decisions, the FTC has warned against unfair or deceptive AI, New York City requires bias audits for automated hiring tools, and Colorado has passed a broader AI act. NIST's frameworks are voluntary but widely used. Check current rules with counsel.
How does AI bias affect which brands get recommended?
AI engines tend to favor brands with large, established footprints, can repeat outdated information, and give different shortlists for differently framed questions. Publishing clear, consistent, dated facts and observing answers across engines over time helps correct missing or outdated evidence, though no method guarantees a recommendation.