Open sourceAI systems2026

Decision Lab

See what a decision model decides, and what changes its mind

Role: Sole developer: I did the design, implementation, testing and documentation
Status: Open source · experimental tool
Stack: Python · FastAPI · JavaScript · Docker
Links: Source on GitHub (opens in a new tab)

Fig. 06 · Decision Lab, schematic. Build → Compare → What-if → Inspect

00 · Summary

A local visual playground for System One decision APIs. Build yes/no, choice and score decisions, compare several models on identical input, and run what-if variations to see what moves the answer.

01 · The problem

Comparing model behaviour by reading raw JSON responses is slow and error-prone. A team evaluating a decision model needs to see where models disagree and how sensitive an answer is to the evidence.

02 · The idea

Make the experiment the unit of work. Keep a form and the JSON in sync, run the same experiment against several endpoints, capture a baseline and diff every change to the input and the output.

03 · How it works

01
Build

Visual yes/no, choice and score decisions with optional state as evidence.
02
Compare

One experiment against several models side by side.
03
What-if

Change the evidence and see the exact input and output differences.
04
Inspect

Raw request and exact upstream response text.

04 · The hard part

Refusing to invent numbers

Token counts come only from what an endpoint reports, never from a guessed tokenizer. Built-in demos are clearly labelled as simulated so they cannot be mistaken for model output.

05 · What I built

FastAPI gateway and browser UI with a two-way form and JSON editor
Multi-model compare across hosted Jev, OpenJEV, local Laya and custom endpoints
What-if baselines with exact input and output diffs and repeatability reruns
Import and export of experiments, with secrets stripped from exports

06 · Results

No · No · Yes On one release-readiness input, Jev and OpenJEV said No and local Laya said Yes, and each named a different blocker.

07 · Why it matters

Teams that choose a model on vibes ship surprises. Decision Lab is a small, concrete example of the evaluation tooling an AI feature needs before anyone relies on it.