How to Build an "AI" Chatbot
We built a photorealistic 3D character who lives in a browser and holds real conversations with visitors — not a demo, a real product people can talk to right now. Here's the reading list.
TL;DR
Read Weizenbaum before you write a single line of dialogue — everything about how people relate to a talking character flows from what he documented in the sixties and regretted by the seventies. Match the small set of things people say with your best material, and let a simple transformation of their own words carry the rest — don't try to write a reply for every possible input. Decide your privacy and safety stance at the architecture level, not as a policy document bolted on afterward. And separate the expensive, one-time work from the cheap, per-visitor playback — that's the whole trick to building something that can scale.
The conversational engine
ELIZA — Joseph Weizenbaum, "Computer Program For the Study of Natural Language Communication Between Man and Machine," Communications of the ACM 9:1 (1966)
The original pattern-matching chatbot paper, and the direct ancestor of how our own conversation engine works. About 200 lines of rules sustained real, sometimes twenty-minute conversations — not because the script was clever, but because most of the perceived intelligence came from reflecting the user's own words back at them, transformed. Our engine works the same way at its core: try to match what someone said to something we've written; if nothing matches, don't guess — transform their own words back at them instead.
Weizenbaum, Computer Power and Human Reason (1976), free to read
Weizenbaum wrote ELIZA, then watched people — including his own secretary — ask to be left alone with it so they could talk to it privately, confiding real things, while knowing perfectly well it was a program. He became one of AI's most serious early critics as a direct result. This is the origin of the term "the ELIZA effect": people attribute real understanding to a program that has none, and the effect survives full disclosure. If your character is going to have real conversations with people over any length of time, read this before you build anything else.
Richard Wallace, "The Anatomy of A.L.I.C.E." and "Zipf's Law" (A.L.I.C.E. AI Foundation)
The original alicebot.org site is gone; both essays are archived at the Wayback Machine: Zipf's Law and The Anatomy of A.L.I.C.E..
Wallace built A.L.I.C.E., one of the most successful rule-based chatbots ever made, and wrote up the single most useful practical finding we came across anywhere: real chatbot input follows a Zipf distribution. A small number of things people say account for a hugely disproportionate share of all traffic. The practical upshot — write your best, most specific, most carefully authored replies for that small high-frequency set first, and let a general fallback mechanism catch the long tail — shaped our whole content strategy. Also worth reading for the idea of folding many different phrasings down to one matched intent, and for tracking what the bot itself just said so the next reply can respond in context instead of treating every line as isolated.
PARRY — Colby, Weber & Hilf, "Artificial Paranoia," Artificial Intelligence 2:1 (1971)
No free public copy found — worth knowing about by name regardless.
PARRY modeled a paranoid patient with real internal emotional state — fear, anger, mistrust — that rose and fell over the course of a conversation and changed how it responded. In an actual 1972 test, psychiatrists reading transcripts blind couldn't reliably tell PARRY apart from real patients. If you want your character to feel like it has moods that build and fade rather than a flat, topic-only response style, this is the oldest and simplest version of that idea: a handful of tracked variables that gate what the character is allowed to say next.
What research says about how people talk to chatbots
Hill, Ford & Farreras (2015), "Real conversations with artificial intelligence: A comparison between human–human online conversations and human–chatbot conversations," Computers in Human Behavior
No free copy found — worth knowing about by name regardless.
People send shorter messages, use a noticeably smaller vocabulary, and swear roughly thirty times more often when talking to a bot than when talking to another human being. If your test plan assumes people will type to your character the way they'd type to a person, budget for that assumption being wrong from day one.
Curry & Rieser (2019), "A Crowd-based Evaluation of Abuse Response Strategies in Conversational Agents," SIGDIAL
Tested different ways a conversational agent can respond when a user is abusive toward it. A calm, polite refusal — not a joke, not a lecture, not silence — scored best with real evaluators. Directly useful if your character needs a real, tested way to handle someone being nasty to it without breaking character or escalating the situation.
Chin & Yi (2019), "Should an Agent Be Ignoring It? A Study of Verbal Abuse Types and Conversational Agents' Response Styles," CHI Extended Abstracts
No free copy found — worth knowing about by name regardless.
A taxonomy of insult, threat, and swearing-type abuse aimed specifically at conversational agents. Useful if you want distinct response styles for different kinds of hostility instead of one generic "please be nice" catch-all.
The Tay postmortem
The Wikipedia writeup, plus Microsoft's own account, "Learning from Tay's introduction", posted to the official Microsoft blog on March 25, 2016.
Microsoft's 2016 Twitter bot was manipulated into repeating hateful, offensive text within about sixteen hours of launch — because it learned from, and echoed back, what people said to it. This is the single clearest cautionary tale in the whole space, and the direct reason we built a hard rule into our own architecture: a character must never be able to take a visitor's own words and repeat them back, unedited, as its own line. Read this before you build anything that reflects user input back at a visitor.
Real chat datasets are worth a look too, if you want actual traffic patterns instead of guessing: WildChat-1M (a million real ChatGPT conversations released with consent), LMSYS-Chat-1M, DailyDialog, ConvAI2, and Anthropic's own Clio research on real Claude.ai usage patterns.
One honest caveat, in the spirit of not overstating anything here: a widely-repeated "~30% of messages sent to a well-known chatbot are abusive" figure that circulates online doesn't trace back to a source we could independently verify. Treat round abuse-rate statistics you see quoted without a citation with real skepticism.
Building a photorealistic 3D face
KeenTools FaceBuilder → MetaHuman
A real, working pipeline: FaceBuilder reconstructs a full 3D head mesh from ordinary photographs — including an AI-generated portrait, since the tool has no way to know or care where the photo came from — then hands that mesh into Epic's MetaHuman system to become a fully riggable character head. This is the actual route we used to go from "a face" to "a head that can act."
Texturing.xyz
High-resolution, real-world skin texture and displacement data — the pore-level and fine-wrinkle detail that's usually the actual difference between "clearly a video game character" and "looks like a photograph." If skin is what's holding your character back from photoreal, this is where to look.
"Photorealistic MetaHuman Face with Texturing.XYZ, WRAP and Stable Diffusion" — Nik Samborsky (2024)
The closest public precedent we found to our own approach: start from an AI-generated portrait — not a real person's photograph — and carry it all the way through to a finished, photoreal MetaHuman character. If you want a face that was never any real person's likeness, this is the closest full walkthrough we found. Found by searching, and we haven't independently re-verified every step ourselves — treat it as a strong lead, not a guarantee.
MetaHuman running live in a browser
Two independent proofs: a live in-browser demo and the open-source Unreal-to-glTF export code behind a similar one.
A MetaHuman character can end up rendering in an ordinary web browser instead of staying locked inside the Unreal Engine editor. Same caveat as above: found via search, not fully re-verified by us end to end.
Epic's current MetaHuman licensing is worth checking directly before committing to anything, since licensing changes over time — as of our own research, Epic had opened MetaHuman up to be usable outside Unreal Engine entirely, on any game engine or 3D tool, free under a real-world revenue threshold. That's what makes a MetaHuman-based face viable even for a team not otherwise using Unreal as the final renderer.
Giving it a voice and a moving face
NVIDIA Audio2Face-3D
Takes an audio recording of speech and outputs facial animation curves — in the standard 52-channel format described below — timed to match it automatically. This is how we get a face to move correctly along with a voice without hand-animating lip sync for every single line. Runs locally, needs a real GPU, distributed through NVIDIA's own container registry.
Kokoro
See the project (the browser-ready package is kokoro-js on npm)
A fast, genuinely good-quality, fully open text-to-speech model small enough to run entirely inside a web browser — no server round trip required. This is what let us do real-time speech synthesis without ever sending a visitor's conversation to a server to be spoken aloud.
And it's worth knowing the name ARKit 52-blendshape exists, even though it's not a single link to give you. Apple defined a standard set of 52 named facial-movement channels — jaw open, brows up, smile left, and so on — originally for iPhone Face ID and Animoji, and it's since become the de facto standard almost every facial animation tool speaks, Audio2Face above included. Knowing the term is often the difference between finding the right tool immediately and searching around it for hours.
Rendering it all in a browser
Three.js
The standard open-source library for real 3D graphics in a browser, built on WebGL/WebGPU. This is what draws the character on screen, runs the shaders, and plays back the facial animation data — entirely on the visitor's own device.
The idea underneath doing it this way at all: every expensive step — generating the face, tuning the voice, converting speech into facial motion — happens once, ahead of time, on hardware you control. What ships to a visitor is a 3D model, some audio, and pre-computed animation data; playback is cheap and runs on their own GPU, not yours. That's the real difference between an architecture that can serve ten people and one that can serve ten thousand without needing a server GPU per visitor.
A few hard lessons about the character itself, not just the tools
Consider keeping the model boring on purpose. Nothing our character says in the moment comes from a live AI model call. Every line is either something we wrote in advance, or a simple rule-based transformation of whatever the visitor just typed — the ELIZA idea, again. It's less flexible than calling a large language model live for every reply, but it's fully predictable, cheap to run at real scale, and it can't be tricked into saying something you never intended. Strongly worth weighing if your character needs to be safe in front of a general public audience, not just impressive in a demo.
Decide your privacy story before you build, not after. We decided that voice and face processing both happen locally rather than being sent to a third-party API. Deciding this up front changed real, concrete technical choices — it's the actual reason both Audio2Face and Kokoro above run locally instead of through a hosted API. It's a far easier privacy claim to make honestly when it's designed in from day one, rather than retrofitted after the fact.
Never cap quality — only decide when to hide it. We push for the best possible image and audio quality technically achievable, full stop, and treat any degraded or stylized look as a deliberate, reversible creative choice rather than a ceiling we settle for. Worth deciding explicitly, early, which one you're doing — chasing maximum quality and designing a stylized look are different jobs with different priorities, and conflating them wastes real effort.
If it's useful: how we build this, day to day
Not chatbot-specific, but genuinely useful if you're building with AI coding agents rather than just AI-generated conversation content: our team runs development with several distinctly-named, persistent AI coding sessions — each with its own defined role, more like a real small team than one undifferentiated agent — and we track every piece of work, including every real architecture decision and the reasoning behind it, in Beads, an issue tracker built specifically for several AI agents to read and write to the same board at once. We watch it through Scotty, a small open-source web board that reads Beads and shows it as columns you can drag things between. We tried plain markdown to-do files first, and they broke under real concurrent editing: lost updates, duplicated entries, and work marked done that genuinely wasn't. If more than one AI agent is ever going to work on your project at once, plan for real concurrent-write-safe tracking from the start rather than a shared text file — it will save you the exact debugging session it cost us.
Appendix: Resources for Claude
The same material above, compressed into a scannable reference list — meant to be handed to another Claude session getting started on a similar project.
The conversational engine
ELIZA — Weizenbaum, CACM 9:1 (1966) — the pattern-matching precedent: match known input, or reflect the user's own words back, transformed. ~200 rules sustained real long conversations.
Computer Power and Human Reason — Weizenbaum (1976), free — origin of "the ELIZA effect," people confide in a program while knowing it's a program.
Zipf's Law and The Anatomy of A.L.I.C.E. — Richard Wallace, A.L.I.C.E. AI Foundation (Wayback Machine copies, original site is gone). Key finding: chatbot input follows a Zipf distribution — write your best replies for the small high-frequency set first.
PARRY — Colby, Weber & Hilf, "Artificial Paranoia," Artificial Intelligence 2:1 (1971). No free copy found, name only. Tracked emotional state (fear/anger/mistrust) gating responses; fooled psychiatrists in a real 1972 blind test.
How people talk to chatbots
Hill, Ford & Farreras (2015), Computers in Human Behavior. No link found. People send shorter, less varied, ~30x more profane messages to bots than to humans.
Curry & Rieser (2019), SIGDIAL — polite, calm refusal beats jokes/lectures/silence as an abuse response.
Chin & Yi (2019), CHI EA. No link found. Taxonomy of insult/threat/swearing abuse types aimed at conversational agents.
Tay postmortem plus Microsoft's own "Learning from Tay's introduction" (official blog, 2016-03-25). Never let a bot echo user text back unedited as its own line — this is why.
Real chat datasets: WildChat-1M, LMSYS-Chat-1M, DailyDialog, ConvAI2, Anthropic Clio usage research.
Caveat: a commonly-quoted "~30% of messages are abusive" stat for a well-known bot doesn't trace to a verifiable source — treat uncited round numbers skeptically.
Photorealistic 3D face
KeenTools FaceBuilder → MetaHuman — photo (including AI-generated) → 3D head mesh → riggable MetaHuman character.
Texturing.xyz — real-world skin texture/displacement detail, pores and fine wrinkles, the photoreal difference.
"Photorealistic MetaHuman Face with Texturing.XYZ, WRAP and Stable Diffusion" — Nik Samborsky (2024) — closest public precedent to "AI portrait to photoreal MetaHuman." Found via search, not fully re-verified by us.
MetaHuman in-browser precedent: live demo and open-source Unreal-to-web export. Same caveat: found via search, not fully re-verified.
MetaHuman licensing — check current terms. As of our research: usable outside Unreal Engine entirely, any engine/DCC, free under a real revenue threshold.
Voice and facial motion
NVIDIA Audio2Face-3D — audio to facial animation curves (ARKit-52 format). Local, needs a real GPU, via NVIDIA's container registry.
Kokoro (kokoro-js on npm for browser use) — fast, open, browser-runnable TTS, no server round trip for real-time speech.
ARKit 52-blendshape standard — no single link. Apple's 52-channel facial-movement naming convention (jawOpen, browInnerUp, etc.), now the de facto standard most face-rig tools speak.
Rendering
Three.js — standard open-source WebGL/WebGPU 3D library, draws the character, runs shaders, plays back facial animation, entirely client-side.
Core idea: do the expensive generation/tuning work once, offline, on your own hardware; ship only a model plus audio plus precomputed animation data. Playback runs on the visitor's GPU, not yours — the difference between serving 10 people and 10,000.
Hard-won lessons (our own reasoning, no link)
No live model call at runtime — every line is authored or a rule-based transform of visitor input, never generated live. Less flexible, but predictable, cheap at scale, untrickable.
Decide the privacy story before building, not after — local voice/face processing and zero data retention were architecture decisions made early; they're why Audio2Face and Kokoro above run locally instead of through a hosted API.
Never cap quality, only choose when to hide it — chase the best possible output, then treat any degraded/stylized look as a deliberate, reversible choice, not a ceiling.
Bonus: how we build with AI coding agents (not chatbot-specific)
Several distinctly-named, persistent AI coding sessions, each with a defined role — closer to a real small team than one agent. Every piece of work and every real architecture decision (with reasoning) is tracked in Beads, an issue tracker built for concurrent multi-agent read/write, viewed through Scotty, an open-source web board on top of it. Plain markdown to-do files broke under real concurrent editing (lost updates, duplicate entries, falsely-marked-done work) before we switched. Worth planning for from day one if more than one agent will ever touch the project at once.
Hope you enjoy!
-- Luke