Kimi K3.1 and the Distance Between an Impressive Demo and a Dependable Result

1. Host and Subject Background

Matthew Berman is the speaker and evaluator in this approximately 26-minute episode from Matthew Berman / Forward Future. Titled “They actually did it... (Kimi K3.1),” it presents his commentary and demonstrations, with no verified guest interview in the supplied evidence. The relevant background is therefore the relationship between the host, his testing approach, and the model under examination.

Berman says he purchased a Kimi subscription, installed the desktop application, and used it to build several projects. During the phone demonstration, he identifies the model as K3.1 Pro running at high reasoning effort. He explains that he is avoiding maximum effort to maintain consistency. His evidence combines these hands-on outputs with specifications, benchmarks, and demonstrations discussed in published material.

His overall judgment is enthusiastic but qualified. He regards Kimi as a very good open-weight model, while saying his initial experience does not put it alongside Opus 5.5 or GPT 6.1 Sol. That is the host’s assessment of the work he obtained, rather than an independently established ordering. The episode becomes useful when those broad impressions are unpacked into the particular successes and failures behind them.

2. What the Episode Covers

The Rubik’s cube is the clearest success. Berman demonstrates a three-dimensional object with reflections that change as it moves, face-turn controls, scrambling, keyboard input, a timer, undo, reset, and adjustable turn speed. He also says its solver computes a solution rather than simply reversing the scramble sequence. Watching the calculated moves play out, he awards the result an emphatic pass. The achievement is broader than an attractive still image: the object supports a set of visible, connected interactions.

The phone model immediately complicates that impression. Asked for an accurate, realistic, browser-based WebGL rendering of the product the episode calls iPhone Duo, Kimi produces metallic edges and camera reflections that Berman likes. Yet he identifies an inaccurate logo and an inappropriate division through the screen. He reports a generation time of 28 minutes and 45 seconds, with web searches already present in the tool history. He requests further research into appearance, specifications, and measurements, but the supplied transcript does not establish how that revision ended.

A Blender knot task raises the difficulty in a different way. Berman intends it to exercise computer control, search, and spatial understanding together. The output resembles a figure eight, but he says it is neither properly knotted nor attached to the harness. Turning the subject into a motion graphic does not resolve the central problem. The first attempt reportedly takes about five and a half minutes and produces a silent, 28-second, 1080p guide, yet its crossings do not clearly show how the rope passes over and under itself.

Berman then shows a 54-second comparison from GPT 6.1 Sol, made at medium effort with the same stated task, and praises its more legible rope relationships. His subsequent request to make Kimi’s version prettier produces shadows, stronger depth cues, and additional camera movement. It still does not, in his judgment, communicate the tying process.

The browser obstacle race provides a more conventional prototype example. Berman reports that Bean Blitz took 48 minutes and includes CPU competitors, several obstacle types, and movement controls. During play, however, he points to inaccurate collisions, flickering, and overly pastel colors. The project’s breadth is evident, but its feature list does not settle the question of how well those features behave.

3. Core Views: Reasoning, Examples, and Limits

The demonstrations argue for treating visual generation as several capabilities rather than one. The cube rewards coherent interaction and state changes. The phone requires fidelity to a particular reference object. The knot depends on relationships among continuous strands and their connection to a harness. Success in rendering reflective surfaces cannot establish accuracy on the second task or structural correctness on the third. These outputs offer evidence about particular abilities, not a transferable guarantee that the model understands every three-dimensional problem.

The knot revision makes the distinction unusually concrete. Added shadows and moving cameras improve presentation without supplying the information a learner needs. For an explanatory animation, the decisive question is whether the operation becomes intelligible. A rope crossing is part of the explanation, not a decorative detail. Berman’s favorable comparison with the GPT output identifies a specific difference in legibility, but it does not establish a general success rate. The episode offers a small set of illustrated results, not repeated trials under fully matched conditions. The cautions about qualified instruction appearing in the demonstrations should also remain part of their interpretation.

Even the task’s name requires care. Different knot terms occur in the transcript, while the motion graphic is explicitly introduced as figure-eight follow-through. The strongest analysis stays with that displayed task and its visible connection problems. Silently treating every transcribed label as an authoritative technical identification would add certainty that the source does not provide.

The game and website expose a related gap between meeting surface requirements and satisfying the intended use. A race can contain competitors and obstacles while delivering unreliable collisions. Berman says the website redesign follows the brand’s colors and font, yet criticizes spacing and its recognizable AI-generated appearance. His short prompts are deliberate: he wants to see the model’s assumptions and taste. He also advises adding specifications when users already know the desired colors, animations, level elements, or technology. These tests consequently reveal default behavior more directly than they reveal the best result achievable with detailed direction.

The benchmark graphic introduces a more serious distinction: plausible presentation versus factual support. Kimi initially supplies broad categories instead of named evaluations. Once prompted for names, it presents figures that it later acknowledges are invented placeholders. Berman supplies published benchmark data and then has to request the image again. Whatever the cause, the sequence demonstrates that a complete-looking chart can arrive before the factual work is sound. Specific labels and precise numbers are not substitutes for evidence.

That principle also applies to the episode’s specifications and promotional claims. Berman cites 2.8 trillion total parameters, 104 billion active parameters, a million-token context window, and mixture-of-experts operation. He confirms image input, says video remains untested by him, and only speculates about audio. The reported 65% reduction in financial-modeling hallucination is a promotional claim without a comparison baseline or method supplied in this discussion. It cannot responsibly become a promise about an arbitrary financial assignment.

The cited benchmark results are impressive: Terminal Bench puts Kimi at 65.7 against Opus at 66.4; GDPVal is quoted at 1846 versus 1866 Elo; Kimi’s OSWorld result is 79.6, and its Artificial Analysis intelligence index score is 55. These remain figures relayed in the episode. Berman himself says his preferred model for computer use differs from the OSWorld ordering, illustrating why benchmark evidence and task experience should be read together.

His suggestions about distillation or benchmark emphasis require an even firmer boundary. Similar typography, warnings, and capability priorities motivate his suspicions, but do not establish training provenance. Likewise, his enthusiasm for downloadable weights and deployment choice does not supply licensing terms, hardware requirements, or operating economics. Open weights create an option worth evaluating; this episode does not demonstrate that any particular local deployment is feasible or unrestricted.

4. Learning and Application

A practical response to the episode is to build separate acceptance checks for operation, correctness, reference fidelity, factual support, and appearance. This is an application derived from the cases, rather than a workflow validated by Berman. Its purpose is to make revision specific. An inaccurate phone needs reference checking; a race with poor collisions needs interaction work; an unreadable knot needs clearer structural relationships. Asking all three to look more polished risks improving the wrong property.

Choose tests that expose the relationships on which the intended use depends. For a cube, inspect whether colors and positions remain consistent through turns. For a product model, compare the screen structure, identifying marks, and measurement references with the target. For a game, trigger collisions and inspect the resulting feedback instead of relying on a list of implemented features. These checks add review effort, but that effort is what distinguishes an attractive prototype from an accepted deliverable. The phone example also shows why a search log is insufficient: tool use establishes that a search happened, not that the information was correctly incorporated. Its unreported revision outcome should remain unknown.

For explanatory animation, describe the missing information in the revision request. Ask for explicit over-and-under crossings, recognizable connection points, and clear before-and-after states. Then inspect whether those changes appeared. The failed beautification pass supplies a reason to prioritize this work before cinematic treatment. Where correct physical action matters, the episode’s comparison cannot certify an animation as adequate instruction. Such material can support review or discussion while retaining the boundary around qualified teaching.

For charts and reports, confirm the data before commissioning its visual presentation. A useful requirement is that each evaluation name and value map to an identified source. Missing figures should stay visibly unresolved rather than being completed with plausible numbers. This recommendation responds to the admitted placeholders in the episode; it does not assume that every other output is fabricated. The point is to make verification a separate step, especially when the result already looks publishable.

Model selection should also preserve the distinction between published showcases, hands-on results, and outside evaluations. Berman shows financial modeling, a FreeCAD turbofan, and Blender architecture, while acknowledging that he has not personally achieved comparable 3D results yet. A team needing those capabilities should attempt its own representative task before turning a showcase into a delivery commitment. A demonstrated success is valuable evidence, but the small sample here cannot supply a dependable success rate.

Finally, compare the cost of acceptable work rather than token prices alone. The episode quotes regular input and output prices of $2 and $10 per million tokens, with cached input at $0.20. Its high-speed option is quoted at $4 and $20, with caching unchanged and a claimed two-to-threefold speed increase. Those are episode-cited prices and performance claims. Berman also cites $2.11 per intelligence task for Kimi, against $7.63 for Fable and $4.54 for Muse, emphasizing that token consumption changes the economics.

For an actual purchasing decision, record total usage, waiting time, revision work, and acceptance outcomes on representative assignments. This adds upfront measurement but answers a more useful question: what does it cost to obtain something the team can use? The episode does not provide enough conditions to transplant its per-task figures into a particular business. Downloadable weights similarly require a separate assessment of licensing, hardware, and operating expense before anyone can claim a deployment saving.

Source

• Original episode: They actually did it... (Kimi K3.1)