There’s been a lot of excitement around TypeSafe AI’s new System One model, Jev. What better way to get familiar with Jev than to use it for my own needs? The following describes how I used Python, Jev, DuckDB, and Marimo to enrich and analyze the quotes I save each year.

Note: This post follows the Marimo notebook it came from. If you prefer Marimo’s aesthetics, you can read the notebook directly.

Background

Each year, I create a “Quotes YYYY” note on my phone. Each entry in this note has a date, a quote/lyric/definition, and a source. Here’s an example entry:

08/12/2022 - “I hate stories like that. Men only think about the past right before their death. As if they were searching frantically for proof that they were alive.” -Cowboy Bebop

Let’s see if the new Jev model can help me enrich these raw entries by accurately answering questions about them. (We’ll see further down that Jev returns typed responses to questions.)

Each question must be one of three types:

1. A Noul question gets a probability that the answer is yes.
2. A Choice question gets one option from a list.
3. A Score question gets a rating on an ordered scale.

Here are the questions I gave to Jev for each quote:

• Does this entry contain profanity? Noul
• Does this entry contain personally identifying information about the person who saved it (not the person being quoted)? Noul
• What kind of entry is this? Choice
  • quote, definition, lyric, or other
• If this entry is a definition, what kind of word is being defined? Choice
  • noun, verb, adjective, adverb, other, or not_a_definition
• What kind of source is this quote attributed to? Choice
  • film, tv, anime_manga, music, book, real_person, reddit, social_media, youtube, meme, or other
• Should the source of this quote be its own category, separate from all of these: film, tv, anime_manga, music, book, real_person, reddit, social_media, youtube, meme, other? Noul
• How uplifting is this entry? Score
  • bleak, somber, neutral, hopeful, or joyful
• How rare, uncommon, or niche is this quote or word? Score
  • known by almost everyone, widely known, known within a fandom or field, obscure, or almost nobody would recognize it

Note: I didn’t happen to write down any definitions in 2022.

Running the Experiment

I’m going to handwave the Python script I created to run the cleaned entries through the Jev API and write the responses to a local file. Here’s what the API calls look like, though.

with TypeSafeClient() as client:
    # cleaned file marks each day with "=== <date>" and each entry with "---"
    for day in (Path(__file__).parent / "words-2022-cleaned.txt").read_text().split("\n=== ")[1:][: args.limit]:
        # header is the date plus any note that is not an entry itself (e.g. "Both…")
        header, *entries = day.split("\n---\n")
        days.append(
            {
                "date": header[:10],
                "entries": [
                    {"text": text, **client.system_one(state=text, questions=QUESTIONS).model_dump(mode="json")} for text in map(str.strip, entries)
                ],
            }
        )
        print(days[-1]["date"], f"{len(days[-1]['entries'])} entries", flush=True)

I’m also going to handwave the manual effort I went through to answer all the questions for each entry myself. Ultimately, load_duckdb.py consolidated Jev’s answers and my answers into a single DuckDB file.

Exploring the Data

That DuckDB file holds normalized data for the entries, questions, and answers. It captures the answers I gave manually and the answers Jev gave from behind the TypeSafe AI API.

Here’s what the schema looks like for the DuckDB file:

We can open a DuckDB connection, query for the tables available to us, and view sample rows from each table.

import altair as alt
import duckdb
import marimo as mo

# read only so the committed database never changes from here, rebuild it with load_duckdb.py
con = duckdb.connect(str(mo.notebook_dir() / "answers-2026-10-18-combined.duckdb"), read_only=True)

First, the tables in the database, which match the schema above.

show tables

Each row in entries is one quote and the date I saved it. There are 38 entries in total.

Note: I’ve left the text of entries with profanity out of the tables in this post, so some entry ids are skipped. The Marimo notebook shows everything.

from entries limit 5

questions holds the eight questions from earlier. Each one has a type and the instructions that get sent to Jev.

from questions limit 5

question_options has one row for every answer a question allows. There are 37 rows across the eight questions.

from question_options limit 5

Five rows don’t say much on their own. Grouping the options by question shows everything each question allows.

-- each question with every answer it allows
from
    questions q
    join question_options o using (question)
select
    q.question,
    q.type,
    q.instructions,
    list(
        o.label || ' = ' || o.name
        order by
            o.label
    ) as options
group by all

Finally, answers has one row per source, entry, and question. Jev and I each answered all eight questions for all 38 entries, so each of us has 304 rows.

from answers limit 5

Exploring Different Types of Questions

We can view how Jev answers different question types.

Labels vs Names

Each answer in the database stores a label. Comparing answers means comparing labels. question_options lists every label a question allows.

Each label also has a name, which is the human-friendly description of the label. labels and names always match exactly, except for Score questions.

Noul (yes/no)

Jev only returns a probability that the answer is yes, e.g. 0.99.

load_duckdb.py stores yes at 0.5 or more and no otherwise, and the raw probability is not in the database. The label and name are both yes or no.

-- first noul answer from Jev, next to every option its question allows
from
    answers a
    join entries e on e.id = a.entry_id
    join questions q using (question)
    join question_options o using (question, label)
select
    e.text,
    q.question,
    q.instructions,
    a.label,
    o.name,
    (
        select
            list(
                all_o.label || ' = ' || all_o.name
                order by
                    all_o.label
            )
        from
            question_options all_o
        where
            all_o.question = q.question
    ) as options
where
    a.source = 'jev'
    and q.type = 'noul'
order by
    e.id,
    q.question
limit
    1

Choice (pick one option)

Jev picks one of the option keys from the question’s criteria in questions.py. That key is the label, and the name is the same key.

-- first choice answer from Jev, next to every option its question allows
from
    answers a
    join entries e on e.id = a.entry_id
    join questions q using (question)
    join question_options o using (question, label)
select
    e.text,
    q.question,
    q.instructions,
    a.label,
    o.name,
    (
        select
            list(
                all_o.label || ' = ' || all_o.name
                order by
                    all_o.label
            )
        from
            question_options all_o
        where
            all_o.question = q.question
    ) as options
where
    a.source = 'jev'
    and q.type = 'choice'
order by
    e.id,
    q.question
limit
    1

Score (ordered scale)

Jev returns a number on the scale, e.g. 2.22. load_duckdb.py rounds it to the nearest level, and the label is that level’s index as text ("0" to "4"). The name is the readable level from the legend in Jev’s response, which matches the question’s criteria list, e.g. 0 = bleak for mood. Compare by label, read by name.

-- first score answer from Jev, next to every option its question allows
from
    answers a
    join entries e on e.id = a.entry_id
    join questions q using (question)
    join question_options o using (question, label)
select
    e.text,
    q.question,
    q.instructions,
    a.label,
    o.name,
    (
        select
            list(
                all_o.label || ' = ' || all_o.name
                order by
                    all_o.label
            )
        from
            question_options all_o
        where
            all_o.question = q.question
    ) as options
where
    a.source = 'jev'
    and q.type = 'score'
order by
    e.id,
    q.question
limit
    1

My Answers vs Jev’s Answers

First, let’s count how often Jev’s answers exactly matched mine. Here are the aggregate values:

-- share of entries where Jev picked the same label as me, a score one level off counts as a miss
from
    answers a
    join answers j on j.entry_id = a.entry_id
    and j.question = a.question
    and j.source = 'jev'
    join questions q on q.question = a.question
select
    a.question,
    q.type,
    count(*) as entries,
    round(avg((a.label = j.label)::int), 2) as agreement
where
    a.source = 'alex'
group by all
order by
    agreement

If we care about particular answers we differed on, we can query for them:

-- every answer where Jev and I differ
from
    answers a
    join answers j on j.entry_id = a.entry_id
    and j.question = a.question
    and j.source = 'jev'
    join entries e on e.id = a.entry_id
    join question_options ao on ao.question = a.question
    and ao.label = a.label
    join question_options jo on jo.question = j.question
    and jo.label = j.label
select
    e.date,
    e.text,
    a.question,
    ao.name as alex,
    jo.name as jev
where
    a.source = 'alex'
    and a.label != j.label
order by
    a.question,
    e.date

Here are a few of the medium rows:

Marimo stores each SQL cell’s result as a dataframe. The charts below read the agreement, option count, and mood results as agreement, choice_counts, and mood.

We can use our earlier aggregates and get a visual feel for Jev’s performance on my particular use case. How often did Jev and I agree on each question?

# every chart lays its legend out in one row along the bottom
@alt.theme.register("bottom_legend", enable=True)
def _():
    return {"config": {"legend": {"orient": "bottom", "direction": "horizontal"}, "view": {"continuousWidth": 300, "continuousHeight": 300}}}

alt.Chart(agreement, title="How often Jev picked my answer").mark_bar().encode(
    x=alt.X("agreement", axis=alt.Axis(format="%"), scale=alt.Scale(domain=[0, 1])),
    y=alt.Y("question", sort="-x"),
    color="type",
    tooltip=["question", "type", "entries", "agreement"],
)

OK, cool. We agree there’s no PII in these entries. It seems like we don’t agree on how obscure these entries are, or how joyful they are. It’s interesting that the Score questions performed poorly. More than likely, rounding to the nearest label when building the DuckDB file penalizes scores that were pretty close to my manual choice.

Are Jev and I using the full range of answer options available to us? Let’s count how many times we answered with each available option.

-- how often each of us picked each medium and kind, labels picked fewer than 3 times between us fold together so the donuts stay readable
with
    counts as (
        from
            answers
        select
            question,
            source,
            label,
            count(*) as entries
        where
            question in ('medium', 'kind')
        group by all
    )
from
    (
        from
            counts
        select
            *,
            sum(entries) over (
                partition by
                    question,
                    label
            ) < 3 as rare
    )
select
    question,
    source,
    if(rare, 'rare (under 3 picks)', label) as label,
    sum(entries)::int as entries
group by all

Again, we can get a more intuitive feel for these results if we use a visualization. We’ll collapse all of the rare answer choices into a single slice so there’s less noise.

alt.vconcat(
    *[
        alt.Chart(choice_counts)
        .transform_filter(alt.datum.question == question)
        .mark_arc(innerRadius=50, stroke="white", strokeWidth=2)
        .encode(
            theta="entries",
            # sorting the colors by total picks also orders the slices, so both donuts read in the same order
            color=alt.Color("label", sort=alt.EncodingSortField("entries", op="sum", order="descending")),
            tooltip=["source", "label", "entries"],
        )
        .properties(width=200, height=200)
        .facet(column=alt.Column("source", title=None), title=f"{question} picks")
        for question in ["medium", "kind"]
    ]
).resolve_scale(color="independent")

Nice, this quickly shows Jev and I were largely in agreement on these questions. In fairness to Jev, some of these questions are poorly designed and have ambiguous answers. For example, reddit is a form of social_media, yet both are available options for the medium question. Who’s to say which is right?

As for the full range of options, neither of us ever picked definition for kind. For medium, meme, other, reddit, social_media, and youtube each got fewer than 3 picks between us.

Note: If this were more than a brief exploration of Jev, more effort would go into refining the questions and grading rubrics.

One learning from this experiment is that Jev is only as good as the data AND the questions you feed into it.

For the Score questions, Jev’s answers and my answers exist on the same relative scales. Let’s see how far apart our answers are for the mood of each entry.

-- my mood next to Jev's for every entry
from
    answers a
    join answers j on j.entry_id = a.entry_id
    and j.question = a.question
    and j.source = 'jev'
    join entries e on e.id = a.entry_id
select
    e.id as entry_id,
    e.date,
    e.text,
    a.label::int as alex,
    j.label::int as jev,
    case
        when jev > alex then 'jev higher'
        when jev < alex then 'jev lower'
        else 'match'
    end as jev_vs_alex
where
    a.source = 'alex'
    and a.question = 'mood'
order by
    e.id

As always, let’s build a visualization to get an intuitive understanding:

_base = alt.Chart(mood, title="How far Jev's mood is from mine").encode(x=alt.X("entry_id:O", title="entry", axis=alt.Axis(labels=False)))
# dots and lines share one scale, so a line takes the color of Jev's dot
_color = alt.Scale(domain=["alex", "jev higher", "jev lower", "match"], range=["#4c78a8", "#54a24b", "#e45756", "#9e9e9e"])
alt.layer(
    _base.mark_rule(strokeWidth=2).encode(y=alt.Y("alex", title="mood"), y2="jev", color=alt.Color("jev_vs_alex:N", scale=_color, title=None)),
    # a match is one gray dot, otherwise mine stays blue and Jev's shows which way it missed
    _base.transform_fold(["alex", "jev"], as_=["source", "mood"])
    .transform_calculate(dot="datum.jev_vs_alex == 'match' ? 'match' : datum.source == 'alex' ? 'alex' : datum.jev_vs_alex")
    .mark_circle(size=80, opacity=1)
    .encode(
        y=alt.Y("mood:Q", title="mood", scale=alt.Scale(domain=[0, 4], padding=10), axis=alt.Axis(values=[0, 1, 2, 3, 4], format="d")),
        color=alt.Color("dot:N", scale=_color, title=None),
        tooltip=["date:T", "text", "alex", "jev"],
    ),
).properties(width="container")

Interesting. We have a lot of off-by-one misses, which fits the rounding idea, but we also have a couple of off-by-two cases.

Note: You can hover over the dots in the Marimo notebook to see the relevant quote.

OK, that’s enough analysis of Jev’s accuracy. The quotes and questions aren’t that important, and there likely isn’t much more for us to glean from them.

Looking Forward

In a future post, we’ll explore how the cost of these requests scales. We’ll experiment with how different combinations and lengths of questions and quotes impact the input tokens we’re billed for.

Spoiler: Every request in the usage screenshot below, including experiments for that future post, cost less than five cents in total.