Cleaning up 6,300 notes, from decision models to a working review queue

My Visual Notes collection had 6,349 notes when this shipped. Some were saved chatbot conversations, some were import leftovers, and some were plans for events that already happened. Useful information was often buried inside material I no longer needed.

The cleanup feature now ranks those notes for review. Jev assesses them, code suggests an action, and an LLM drafts a shorter version when I open a distill card. I decide what to keep. Getting there required an experiment that changed the product's direction, followed by production checks that found several things the tests missed.

The reference had the wrong taste

On October 24, I tested 144 production notes with Jev jev-1.14.0 and OpenAI Decisions gpt-6-luna. The experiment repeated calls, reversed choice order, asked section and pair questions, and included a normal LLM baseline. It cost $0.061 on Jev and $0.232 on OpenAI, including that baseline.

A Claude reference read every note and supplied labels. It favored deletion: 58 of the 144 notes. The models could agree with that reference and still produce a cleanup queue I did not want.

I then labeled 60 notes blind: all 41 holdouts plus 19 development notes selected for disagreement.

Five of the keeps also needed regrouping. These are labels for a selected sample, not an estimate of the whole collection.

Claude matched only 33% of my exact verdicts. A saved answer can be generic and still contain a fix worth keeping. An expired event plan can lose its schedule while keeping a reusable checklist. Several short notes can belong together without any of them being junk.

The useful action was often to distill the durable part or put it with related information. A reference model's labels could not settle that choice for me.

Ask the small question

The strongest trim signal was one yes/no question: was most of the note copied or generated?

That signal ranked my trim labels at AUC 0.93 on Jev and 0.92 on OpenAI. The holistic “keep but trim” verdict scored 0.51 and 0.40, near chance or worse. On both vendors, the top ten notes ranked by the narrow signal contained nine trims and one archive.

This gave the implementation a useful division of work. Code computes source, word count, and dates. The decision model answers narrow questions about content. Code combines those answers into a ranked queue. A writing model drafts when there is something to review.

Reversing the note-kind choices changed OpenAI's answer on 14% of notes and Jev's on 5%. A probability is useful for sorting, but it does not make a suggestion true. The experiment results include the protocol, costs, and both sets of labels.

What the whole collection looks like

The first production sweep saved 6,335 Jev assessments with zero failures. Collection rows, events, and places are excluded from assessment. Their structured fields can hold the value even when the note body is empty.

A read-only snapshot at 10:20 p.m. EDT on October 25, 2026 showed this distribution. Percentages use the 6,335 assessed notes as the denominator.

All assessments used jev-1.14.0, question set v1. All 6,335 were still unreviewed. These are model signals combined with code rules, not my decisions. The delete bucket's size does not establish that a quarter of the collection should disappear.

Regroup is separate from this per-note partition. The snapshot held 4,096 pair assessments, with 460 pairs meeting the combine threshold. Pairs can share notes, so those counts cannot be added to the table or treated as 460 independent merges.

Distillation is the largest suggested job. The next evidence comes from reviewing the suggestions and seeing which ones I accept.

At an earlier billing checkpoint, the production key showed $1.14 estimated spend, 15,888 requests, and 27,142,782 input tokens, including regroup work. That is separate from the small experiment above. Dashboard totals can lag; this was not a final bill.

A queue I can act on

The shipped interface has ranked buckets for a deep pass and a daily stack with eight cards. Each note explains why it appeared. Distill and combine drafts use OpenAI on demand and can go into a new or existing note. A header chip brings the suggestion into the normal note view.

Keep hides a note from cleanup for six months. Snooze hides it for 30 days. Regroup can combine related notes or file a note into a path or collection without rewriting it.

Approving a distill or combine archives the source. Links follow the retained content, and the archive provides Restore. After 30 days, archived notes are automatically deleted, leaving a small record so old links can still resolve. That retention rule starts after an approved action. Assessment alone does not archive or delete anything.

Production verification loaded the buckets, daily cards, archive page, search, and header chip. It generated a successful draft with an enabled Distill action. It did not execute an archive or delete action, so those paths remain outside the live verification claim. The implementation notes describe the shipped flows.

What production caught

PostgreSQL lived inside the application's Compose project, rather than in a separate Coolify database resource. A roughly 1.2 GB dump passed a full read validation, with another copy on the host. Both copies remained on the same server. Four additive migrations applied; the handoff had counted three.

Compose omitted TYPESAFE_API_KEY and CLEANUP_DECIDER; the OpenAI key already existed. PR #233 fixed the mappings. A real draft then failed because the shared helper sent temperature to gpt-5-mini. PR #234 removed unsupported sampling settings and restored successful drafts.

A background embedding timer then failed at singleFlight. It called a Solid router action without an HTTP router context. Tests had hidden the problem by replacing router actions with plain functions. PR #235 moved the business logic into plain server workers, leaving router actions for browser requests.

Once that worked, a larger vector batch took over seven seconds and exceeded Prisma's five-second interactive transaction limit. PR #236 gave that vector-write transaction 60 seconds while keeping deletes and replacement writes atomic.

The final rollout also filled the 71 GB server disk with repeated image builds. Coolify returned 500 and the app stopped. Removing unused Docker build cache recovered space without removing volumes or backups. The rollout finished from the already built final image. The stale job was cancelled after manual recovery, so its cancelled status did not describe the running application.

Working, with catch-up still running

The verified image was 137cbdb9f817792b97041244fa5d6c45b87f6935. The login route returned HTTP 200, no migrations remained, and the server had about 16 GB free. Both embedding fixes passed CI, and 19 focused tests passed.

The embedding count rose from 6,263 to 6,347, covering 84 previously missing notes. A later checkpoint recorded 75 replacements after the final startup. Multiple changed-note batches projected 25 points each with zero failures.

That did not finish the backlog. One checkpoint still had 6,138 changed notes waiting, and two notes produced no embedding chunks. Catch-up continued in the background.

The useful result is a review queue backed by a measured signal and a working draft flow. Whether it makes the collection better depends on the next step: keeping the durable information, approving the changes, and using those decisions to judge the queue.