Tap8: Interactive video as the interface between humans and AI

Today, we're sharing the technical vision behind Tap8: interactive video that is generated, watchable, and operable in real time. We think the interface to AI should scale alongside intelligence; how models hand information back to humans should not be treated as an afterthought. Interactive video lets a model answer with a small runnable world — a chart you can interrogate, a mechanism you can take apart, a lesson that reshapes itself around what you ask — instead of another wall of prose.

Tap8's generation engine is deployed today, producing information-dense video whose facts are rendered as live code rather than hallucinated pixels. In early benchmark testing across seven systems, it places first on overall quality for information-dense work. A research preview of the interactive system is coming soon.

The bandwidth bottleneck

The real constraint on AI's usefulness is not intelligence — it is bandwidth between model and human. A capable adult reads at roughly 240 words per minute, about 5 tokens per second of intake. Frontier models already emit 80–240 tokens per second, and specialised inference runs into the thousands. Even at the conservative end — one model, one attentive reader — that is a ~16× mismatch, and agent teams widen it by orders of magnitude.¹ Agents spend most of their output budget producing prose the user will never finish reading; the user makes decisions from a skimmed fraction of what was generated. This is not an edge case — it is the default.

Text is the wrong substrate for this mismatch. Humans do not process text and visual information at the same rate: the same facts rendered as a chart, an annotated diagram, or a spatial layout are absorbed in a fraction of the time, because the eye parallelises in ways reading does not.

Audio is the human-preferred input to AIs, but vision is the preferred output from them. Around a third of our brains is a massively parallel processor dedicated to vision — the 10-lane superhighway of information into the brain. As AI improves we'll see a progression: raw text → markdown (today's default) → HTML (early, forming) → … interactive videos / simulations, ending in something generated directly by a diffusion neural net. — Andrej Karpathy

Yet we still pour every reply through a textarea and a scrolling transcript. Chat UIs are the new command line: the same paradigm as the 1980s terminal, chosen when graphics were hard and memory expensive, now wearing better fonts over an infinitely more capable model. The GUI revolution of the 1980s did not make computers smarter; it made the intelligence they already had legible and steerable.

The bottleneck runs the other way too. As models begin improving models — the loops the field now calls recursive self-improvement — humans must supervise systems that emit decisions faster than any transcript can be read. The leaders of the frontier labs are publicly converging on the same concern, and at research workshops the question is asked directly: how do we keep humans meaningfully in the loop as these loops accelerate?² We regard high-throughput interfaces for human oversight not as a tooling optimisation but as a defining safety mechanism for that era. Both directions point at the same missing artifact: an interface that uses the full visual bandwidth of the human.

Information should run, not display

Human interaction with the information world is moving from seeing information, to running it, to eventually entering a simulable, actionable world. Almost every digital product today lives at the first layer: text, pages, apps, dashboards — fixed containers designed in advance, into which information is poured. The human states a need, the system retrieves or computes, and the result lands in a pre-designed interface. Search for a pair of earbuds and you get a results page, a spec table, a review video — all prepared beforehand; the user only browses.

At the far end sits the fully learned world model: a system that has internalised how objects move, what actions cause, and which states are possible, and can simulate consequences directly. It is the right long-term destination — and today it is compute-heavy, weakly controllable, hard to edit, and poor at executing precise logic.

At this layer, the system generates, in real time, a small runnable world from the user's intent, the current context, and the environment state. Ask how noise cancellation works and it doesn't open a page — it generates a 3D model, an exploded structure, a sound-wave animation, clickable parameters. Code here is not a tool for presenting information but a world-state descriptor: current state, plus conditions, plus evolution rules, yields the next state.

Seen as routes to that missing layer, the field offers three:

• Language & GUI. The mainstream agent path: language understanding → tool calls → results poured into fixed interfaces. Output remains text-centric, visuals decorative; it struggles to express dynamic space and complex process.
• Pixel world models. Video-generation systems predict the next frame in pixel space. Rich, but compute-heavy, weakly controllable, hard to edit — and trained to imitate appearance, not to carry correct information. There is no element to click, no equation kept exact.
• Code + pixel diffusion — our route. Understand the user and environment → generate structure, objects, and rules as code → execute in the browser → render a visual experience → keep revising from visual feedback. The logical expressiveness of language models, the determinism of software, the immediacy of video.

The route is not "simulate the whole world in pixels" but "generate a runnable world in code first, then use generative models to fill in texture and open-ended visuals where needed." Code handles structure, causality, interaction, and state — the low-entropy representation. Pixel diffusion handles texture, detail, and high-entropy open-world content. Not either/or, but a division of labour: tokens for what is true, pixel diffusion for what it looks like. This phrase recurs through the rest of this post, because it is the design rule the whole system is built on.

The industry is already moving in this direction — confirmation that text and chat alone are no longer enough. Google's Generative UI generates interfaces and simulations on the fly for a prompt; Anthropic's custom visuals build diagrams and charts inline in chat.³ But each, by its maker's own description, ships a one-shot artifact: whatever interactivity it has — a button, a slider — is baked into a single generated snapshot that never regenerates itself in response to what the user does. There is no S(t+1) loop, and no video. The direction is validated; the commitment so far is a feature, not a foundation. Tap8 goes the distance the artifacts stop short of: it generates the next state on the fly, so every interaction reshapes the medium itself rather than exercising pre-written controls.

What Tap8 does

The first turn generates an interactive video with a main storyline — an explanation, a lesson, a walkthrough, grounded in real source material. From then on, nothing about it is fixed. Books and web pages made information readable; video made it watchable; software made it operable. Tap8 makes it all three at once, generated in real time.

• Every element is a handle. A chart is not a picture of a chart — it stays a chart. Hover a term and it expands; drag a parameter and the explanation reshapes; zoom into a region of a diagram and get the diagram underneath it. Because fact-bearing content is live code rather than baked pixels, interactivity is a property of the medium, not a feature bolted on.
• Interaction branches the storyline. Point at a term and it becomes its own sub-story. Ask "wait — why?" and the storyline bends around the question. Each detour becomes a node in a personal exploration graph; come back tomorrow and the environment picks up where you left off, shaped by what you explored and what you skipped.
• How-to that adapts. Traditional how-to video is linear and one-size-fits-all, while every learner is non-linear and unique. An interactive video can pause when you need to act, branch when your setup differs, answer when you ask, and verify when you finish — turning a 20-minute passive watch into a 5-minute successful completion. How-to is also where the stakes are highest: 86% of YouTube's users say the platform matters for figuring out how to do things they haven't done before.⁴
• Personalised and proactive. Delivery is tuned to what you already understand, what you're working toward, and what you keep circling back to — and the system surfaces what you should know next before you know to ask. Search answers the question you had the vocabulary to form; a proactive interface suggests the question you didn't know was available.

We optimise for information delivery and task completion — the functional end of the video spectrum: education, professional training, deep research, how-to, news. That is where correctness matters most and where interactivity pays off most. We start functional and keep expanding towards the lighter end of the spectrum — lifestyle, shopping, product launches, travel — where we expect to meet those working their way over from the entertainment side.

Our approach

Informative video carries information whose correctness matters, and whose correctness is determined by something other than how the video looks. That single property is what makes the problem hard. A model trained to reconstruct pixels learns what a whiteboard looks like, never whether the mathematics on it is right — its loss was never a function of the mathematics. More data and more scale make the whiteboard more convincing, not more correct. So we architect for correctness and appearance as different problems needing different machinery, and we build toward one unified interactive video model in deliberate stages — with a deployed, state-of-the-art system at every stage along the way.

Route facts and pixels differently

Not all content in an explanatory scene relates to the world the same way, so forcing all of it through one rendering path mishandles some of it. We split every scene into three tiers, defined by two questions: is the content fact-bearing, and must it be physically fused into the scene?

Tier B is the interesting one: it is the most correctness-critical content and the content that least needs to be visually fused into the scene, which is exactly what lets us keep it exact. The result is a system that is fail-safe on facts — when in doubt, content routes to the exact layer, where the worst case is that it looks composited rather than embedded, never that it is wrong. And because Tier B stays live code rather than baked pixels, the output is not a video in the ordinary sense: it is a timeline-synchronized interactive document, which is what makes every-element-is-a-handle possible at all.

One backbone, two heads

A naive way to build this would be to wire a reasoning model to a video model and let them talk in text: the reasoner compresses everything it wants into a prompt, the video model decompresses the prompt into pixels, and neither the full internal state nor the training gradient ever crosses the gap. A shared latent removes that boundary. When one backbone produces both the plan and the perceptual content in a single attention context, the renderer is not told what the plan is — it attends to the same hidden states that produced it. Communication becomes attention, not serialization.

On top of that shared context sit two heads, each confined to the subspace where its inductive bias is right. A discrete token head produces what must be correct — code, structure, symbols — trained with a token-level loss that sharply penalizes the plausible-but-wrong continuation. (This is exactly why language models can do algebra: getting the maths wrong is punished token by token.) A continuous-diffusion pixel head produces what must merely look right — the perceptual detail — where continuous diffusion wins on quality per token. Tokens for what is true, pixel diffusion for what it looks like, in one model rather than two.

We are not betting on this backbone alone — the industry is building it with us. A single transformer carrying a discrete loss and a continuous diffusion loss in one shared context is now multi-lab consensus: Meta, DeepSeek, and ByteDance have all published variants, and Google DeepMind runs it in production — its Nano Banana image family generates images natively inside the Gemini model rather than calling out to a separate specialist, positioned for information-bearing visuals while Google's specialist pixel-diffusion models (Imagen, Veo) carry photoreal and cinematic content.⁵ That product split is the sharpest external evidence for our thesis — the appearance-versus-process division, chosen at flagship scale. We build on a well-tested backbone the industry is converging on, and concentrate our contribution and optimisation on the fidelity-and-interaction layer that sits on top of it — the layer no general-purpose model is optimizing for.

Visual quality: recursive self-improvement

Our engine's visual quality comes from a loop our CTO published at the ICLR 2026 Workshop on Recursive Self-Improvement.⁶ A coding agent writes visual code; an evaluation agent renders that code and inspects the actual output — a visual critic, not a syntax checker — and feeds descriptive, vision-based feedback back to the coder. Traces that pass the evaluator's highest bar are distilled into training data; the fine-tuned model replaces the one powering both agents, and the loop begins again with a natively better designer and a natively better critic.

On top of the paper, we run the same trick one level up. A meta-coding agent analyses batches of evaluator feedback to find systematic errors in the coder, proposes changes, and backtests candidates before promoting them. A meta-evaluation agent watches where the evaluator and human graders disagree, and runs automated experiments to re-tune the evaluator's rubrics until a candidate demonstrates higher correlation with human judgment. Human evaluators — a large pool of expert annotators — audit samples throughout, so the loop optimizes toward people, not toward itself.

The meta-loop is not a luxury; it is forced by the domain. In code and mathematics, correctness never changes. Visual taste changes constantly, and faster as AI output proliferates: styles that read as state-of-the-art in early 2025 read as slop today — not because the code became less correct, but because taste moved, and it moves fastest against exactly the patterns a reward model most recently reinforced. A static judge would train the model into yesterday's taste. A self-evolving judge is the only scalable way to keep up — and because "wow factor" is, mathematically, the ability to sample a low-probability yet surprisingly suitable path, keeping current rarely requires retraining the model. An up-to-date evaluator periodically nudges the policy toward or away from paths, cheaply and continuously.

Two pieces of machinery make the loop trainable at all. We developed a credit-assignment method that propagates a holistic visual judgment — "this looks good / bad" — back to the specific tokens of code responsible; without it, the visual signal is too coarse to train on efficiently. And because scoring visual code means rendering it, including deliberately risky rollouts that can and do crash rendering environments, we built proprietary training infrastructure that renders, captures, and grades unstable generated code — including animated and 3D scenes — safely and without blocking the training cluster.

Latency: diffusion language models

After solving visual information delivery, the gap is the clock. Autoregressive code models emit on the order of 100 tokens per second, which turns a single rich scene into minutes of generation — good enough for an offline render, fatal for interaction. Real-time generation is the last gate between the deployed engine and the interactive product, and our answer is the diffusion language model (DLM): instead of generating one token at a time, each conditioned on the last, a DLM instantiates a canvas of tokens and iteratively refines every position in parallel over a small number of denoising steps. Because many positions resolve per forward pass, decoding becomes compute-bound rather than memory-bound — the regime modern accelerators are built for. Open-sourced DLMs demonstrate the step-change: 2,146 tokens per second on code generation, roughly an order of magnitude above typical autoregressive decoding.⁷ That is the difference between a scene that takes a minute and a scene that takes a few seconds; between an offline render and a live surface.

Two properties of DLMs are usually liabilities, and both turn in our favor on this task. First, masked diffusion LMs run on a fixed-length canvas, and allocating that length efficiently is an open problem for general models. Our output is always frontend code for one animated scene, and its length is predictable from the input — 3D and shader scenes run long, diagrams medium, simple layouts short. Across a small, known set of task shapes we can pursue canvas-length optimization a general model cannot: length adaptation learned during training, allocating the canvas dynamically per shape. Second, diffusion LMs still lag autoregression on hard sequential reasoning.⁷ That gap would bite if we were asking the model to solve hard problems. We are not: we are asking it to translate and polish grounded source material into visual code that looks right — with sufficient training data, a distribution-translation problem, not a reasoning one, sitting squarely in the half of the capability space where diffusion LMs are already strong and fast.

The cost of the bet is real: DLMs do not expose the per-token likelihoods that every standard SFT, RLHF, and distillation recipe is built on, so the post-training stack has to be rebuilt on sequence-level objectives appropriate to diffusion LMs. We are building it, and early internal results are encouraging. Speed, when it lands, buys more than interaction: our quality already comes from a render-grade-regenerate refinement loop that is too expensive to run to convergence at autoregressive speed. At DLM speed the same loop runs many more passes in the same wall-clock budget — at runtime, live. The throughput gain converts directly into visual quality.

Present in real time, deep when it needs to be

Deep grounding — retrieval, tool use, browsing, planning — cannot finish in the time it takes to respond to a tap. Thinking Machines named this tension for real-time audio-video collaboration and split their system into an ever-present interaction model and an asynchronous background reasoner;⁸ we adopt the same split for interactive video, and simplify it. The interaction tier is the fast model itself: it takes every gesture, renders code and video, and holds the thread. Easy interactions — "slow down," "zoom in" — never leave it. When a gesture genuinely needs new facts, it delegates a derived sub-query to a reasoning tier that runs asynchronously, and stays present — answering follow-ups, taking new input — weaving the grounded result back in when it arrives. While the reasoner works, the interaction tier only re-presents what is already grounded or shows scaffolding; it never invents a fact to fill the gap.

What makes this split clean rather than a step backward is what crosses the boundary: only discrete, symbolic content — a sub-query going out, a grounded answer coming back. The shared latent that actually matters, semantic state to pixels, stays entirely inside the interaction tier. Reasoning becomes a separate, swappable component — always the best available frontier reasoner — and each paradigm sits where it is strongest: a fast diffusion-language-model renderer at the front, a frontier autoregressive reasoner behind, each robust to how the other evolves.

A loop that improves with use

Because the artifact is interactive, evaluation and improvement close into a loop. Our vision-language-model judges are not a research aspiration — they are the working reward signal behind today's quality. And because every element of the fact-bearing surface is addressable, real sessions produce graded attention on exactly the content whose correctness matters: which term gets expanded, what gets scrubbed back to, which chart gets interrogated. The judges are calibrated against that signal and self-evolve to stay aligned with what real viewers actually found correct and clear. A better judge yields a better model, which yields better video, which yields sharper calibration. Most systems learn from passive watch-through; we learn from people reaching into the surface and using it.

Real-time interactivity is the destination, but the engine underneath it can be measured today. We benchmarked Tap8's deployed generation engine against six systems spanning both paradigms: three code-render systems and three pixel-diffusion systems.⁹ Twenty-five intent-driven scenes across five use cases — Professional, News, How-To, Explainer, Marketing — each grounded in real-world source material, with the brief specifying the facts a viewer should walk away with. Scenes were sampled under rules fixed before generation.

Every system ran the latest flagship release in its series, frozen as of July 6, 2026, on its official platform at maximum quality settings. Every system received the identical fully-detailed brief verbatim — runtime, art direction, per-scene shot list, and the facts to deliver — with zero per-system prompt engineering.

Twelve independent annotators rated all videos blind and unlabeled, in randomized order, fully crossed — every annotator rated every system, two raters per video — producing 6,760 annotations across five metric protocols: three adopted from published benchmarks, two benchmark-native.¹¹ An automated pass corroborated the panel with more than 3.7 million frame-level model evaluations under VBench++'s long-video pipeline.

Because Marketing behaves as a persuasion task rather than an information-transfer one, every result is reported in two frames: all 25 scenes, and the information-dense frame (Professional, News, How-To, Explainer) the benchmark targets. Headline claims below use the information-dense frame and say so; the all-25 figure is given wherever the ranking changes.

Tap8 tops Overall Quality on information-dense work: 75.2 T, first of seven. The mechanism is that Tap8 is the only system that never collapses on either axis — second on Informative Quality while holding enough aesthetic ground that the blend comes out first.

Factual Coverage is the binding constraint, and Tap8 leads it outright. Coverage — the share of the brief's required facts that appear on screen, verbatim and legible — is the benchmark's most discriminating metric, and it causes comprehension: pooled across 173 videos, the fit is delivery = 0.36 + 0.60 × coverage (r = 0.70). A fact that never reaches the screen still gets answered right 36% of the time by guessing; one that does reaches 96%. Getting a fact on screen is worth roughly 60 points of comprehension — so coverage is the game, and Tap8 plays it best: 83.6% on information-dense scenes, significantly ahead of five of six competitors, with only Fable 5 within statistical reach. Downstream, viewers learn as much from Tap8 as from any system in the field — 84.5% comprehension correct-rate, statistically tied for first — and Tap8 leads the field outright on rendering on-screen facts correctly.

Shown side-by-side against Fable 5 — the field's informative leader — viewers prefer Tap8. Tap8's win-rate is 56.9% on information-dense scenes and 58.7% across all 25. Both intervals cross 50%, so the statistically supported claim is that Tap8 significantly does-not-lose to Fable 5 — a non-inferiority result with the point estimate on Tap8's side, not a significant win. It still closes the loop on the frontier: the one system ahead of Tap8 on content composites does not beat it in front of viewers.

The aesthetic trade-off is not fully closed. Across all 25 scenes, the pixel-diffusion systems are preferred aesthetically over Tap8 — its overall side-by-side win-rate is 40.7%, significantly below the no-preference line, driven by exactly the systems at the bottom of the informative ranking. On the T2VWorldBench visual dimensions, Tap8 is statistically tied with the leader on Realism, Relevance, and Consistency. And the human panel's ordering is not an artifact of our raters: the VBench++ long-video pipeline, scored with no access to the human labels, recovers the same preference ranking of the six competitor systems (Spearman ρ = +0.89).

Green = best in field per row. † Composites: Informative Quality = (Content Fidelity + Visual Quality)/2; Overall Quality adds Aesthetic Preference as a third term. On the all-25 frame the top of several rows shifts toward Fable 5; confidence intervals, per-use-case splits, and significance verdicts are in the full report.¹²

These numbers measure today's deployed engine, graded by people and machines. They are the floor the interactive system starts from, not a measurement of interactivity itself. No benchmark for interactive informative video exists yet; more on that below.

Limitations and future work

Measurement itself is the biggest open problem. There is no agreed benchmark for interactive informative-video quality — the field has barely begun to define what to measure. Our vision-language-model judges are our working answer, and they are why our videos lead today, but a shared, external standard for this class of system does not yet exist. We think building one is worth the field's attention, we intend to contribute to it, and we invite the community to hold us to the numbers when it exists.

Our accuracy is bounded by the frontier. The system is a faithful channel from a grounded reasoner to video, so it is as truthful as the best reasoner we can call — and our measured fidelity is only as reliable as the evaluator that checks it, which is itself a model. Both ceilings rise as frontier reasoners and our own self-evolving judge improve; neither requires a different architecture.

Marketing content. Tap8's Factual Coverage on Marketing scenes is 53.8%, against 80–87% on the four information-dense use cases. The split is diagnostic: on Marketing, Tap8's Content Fidelity is 51.1 while its Visual Quality holds at 74.8 — the rendering is sound, and the content brief is what misses. Marketing is a persuasion task rather than an information-transfer one, which is why the information-dense frame exists and why we disclose it rather than assume it.

Aesthetic appeal, without trading the informative lead. Tap8 sits at 61.7 on Aesthetic Preference, sixth of seven. Preference is empirically near-orthogonal to the informative metrics, so this is an independent programme — motion, polish, appeal — not a trade against coverage or delivery. The information-dense frame already shows the ceiling: Tap8 reaches preference parity there while holding its content lead. The runtime refinement loop that DLM speed unlocks is aimed squarely at this axis.

Tell us what you think, join us

In the coming months we will open a limited research preview of Tap8 to collect feedback, with a wider release to follow. If you are building evaluations or benchmarks for interactive information delivery, working on recursive self-improvement, training diffusion language models, or simply believe the interface to AI deserves more than a scrolling transcript — we'd love to hear from you, and we're hiring. Reach out at derek@sigmatrx.ai.