Diktafone

Diktafone is a private Android app for remembering people. You press one button, talk for two minutes about a conversation you just had, and the app turns that into a structured profile for that person - bio, worldview, quick facts, contact details, what you talked about and when.

The part I care about most: the transcription and the extraction both run on the phone. No audio and no transcript ever leaves the device for inference. Whisper large-v3-turbo does speech-to-text, Qwen3-1.7B turns the transcript into structured fields, both through llama.cpp bindings, both on CPU. Convex stores the result and syncs it, but it never sees the model.

The second part I care about: the AI is never allowed to write to the database. Every inference lands as a draft with per-field confidence and the source snippet it came from. Nothing becomes canonical until I approve it. That constraint shaped most of the architecture. Without it, a model that gets a detail slightly wrong writes a plausible-sounding fact into my contact book and I have no way of catching it later.

The repository is private. This document is the public record of how it was built.

I have a specific, boring problem. I meet a lot of people through college, freelance work, and student organisations. I remember faces and I remember that something mattered to someone - but I lose the detail. What they were building last time we spoke. Which of my friends introduced us. What they said they were dreading. The things that make the next conversation feel like a continuation instead of a restart.

Existing options each failed on one axis:

• Clay.earth is the closest commercial product and genuinely good, but it works by ingesting your email, calendar, and social graph into someone else's cloud. The whole value of these notes is that they are candid. Candid notes about my friends are the last thing I want sitting in a third-party index.
• A notes app (I already run an Obsidian vault) has no structure, no cadence signal, and no capture flow. I tried keeping person pages by hand after every coffee and stopped doing it inside a week.
• A traditional CRM models pipelines and deals. I don't want a pipeline. I want memory.

So the requirements fell out fairly cleanly:

1. Capture has to cost nothing. One button, talk for two minutes, walk away. If capture requires typing, I won't do it.
2. The data stays mine. Local inference, single private account, no analytics.
3. The AI proposes, I dispose. Structured extraction is the point, but I have to approve everything before it becomes truth.
4. It has to model closeness honestly. Not just a contact list - tiers, cadence, and a nudge when someone I claim is in my inner circle hasn't been spoken to in two months.

The rest of the app follows from those four.

Home: recent people, who's gone stale, and the record button.

The core loop I built around:

tap record → speak → on-device transcription → on-device extraction
→ person draft (per-field suggestions) → review → canonical person page

Everything else in the app exists to support that loop.

The record screen writes audio to disk before it writes anything to the network. Every recording produces two files: a 16 kHz mono WAV (what Whisper actually wants) and a compressed M4A (what I keep, so I can listen back). The WAV is deleted the moment transcription finishes; the M4A stays.

Memos are queued locally with a client-generated id, so capture works with no signal and the queue flushes idempotently when the device reconnects. I record in places with no usable connection often enough that capture can't depend on one.

Capture writes to disk first and syncs whenever a connection turns up.

A finished memo becomes a draft, not a person. The draft screen shows each extracted field alongside the confidence the model attached to it and the snippet of transcript it came from, and lets me approve all, edit a single field, reject a single field, or re-target the whole draft at a different person.

Every AI-derived field arrives with a confidence and the sentence it came from.

One canonical profile per real human. Structured fields (name, DOB, address, contact details), AI-derived fields (bio, worldview, quickbits), a closeness tier, tags, linked identities including a tappable Instagram handle, and a timeline of memos and check-ins.

The canonical page. (The tier chip in the header is tappable.)

One place for everything the app wants from me: pending drafts, tier-change nudges, stale-relationship prompts, possible duplicates, and unresolved field conflicts. Dismissing a nudge snoozes it - it never deletes the underlying item.

Nudges, approvals, and duplicate suggestions in one queue.

Owner-scoped full-text search across names and transcripts, with people ranked above raw memo hits. A duplicate detector plus a manual merge tool. And a device-contacts import that seeds profiles from the phone's address book - including contact notes, which get treated exactly like a spoken memo and run through the same extraction and review pipeline.

Three boundaries matter here:

Device vs cloud. All inference happens left of the Convex box. Convex receives transcripts and structured data - it never receives audio to process, and never receives a prompt. I did this for privacy rather than for speed; a hosted API would be considerably faster.

Draft vs canonical. The person_drafts → draft_field_suggestions → approval → people path is the only way an AI-derived field reaches a person. There is no code path where extraction writes to people directly. The schema enforces this, so there's no shortcut for me to take when I'm in a hurry.

Manual vs inferred, per field. Name, DOB, contact details, handles, tags, and closeness tier are user-controlled and directly editable. Bio, worldview, quickbits, and search chunks are AI-derived and only change through review. The person page gives the two classes different editing controls, so it's always clear which kind of field I'm touching.

Most of the work on this project went into these choices, so each one below records what I picked, what I turned down, and why.

Chose: Whisper + Qwen3 running locally through whisper.rn and llama.rn. Rejected: OpenAI/Anthropic/Gemini APIs, or self-hosting a model behind my own endpoint.

A hosted API would have been faster to build, faster at runtime, and far more accurate at extraction. I gave all of that up on purpose. The content here is candid commentary about people who did not consent to being processed by a third party. If the app's premise is "your relationship memory is private", then shipping the transcripts to an API contradicts the premise no matter what the vendor's retention policy says.

The cost is real: a 1.7B model extracts noticeably worse than a frontier model, and a cold extraction run takes minutes rather than seconds. The approval queue is what makes that acceptable. I'm reviewing every field before it lands anyway, so a weak first draft just means more editing during review.

Chose: Convex as the canonical store. Rejected: Supabase (Postgres), Firebase.

The reactive query model means the person page, the inbox, and the home dashboard all update the moment a mutation lands, with no cache invalidation code. Most of this app's UI is several views of the same evolving graph, so that saved me writing a lot of state-syncing code. Schema-as-TypeScript also means the client and server share types with no codegen step of my own.

The tradeoff: no SQL. The cadence engine would be one GROUP BY in Postgres; in Convex it's a bounded read-time computation with explicit caps (MAX_PEOPLE = 200, CADENCE_CAP = 50). At personal scale - a few hundred people, not a few million - that's fine, and I'd rather pay it than hand-roll subscriptions.

Chose: Clerk via ConvexProviderWithClerk, Google sign-in. Rejected: rolling auth into Convex directly.

Purely pragmatic: Google sign-in on Android, session handling, and token refresh all worked out of the box, and users keys off the Clerk JWT's tokenIdentifier, which is stable. It cost me one build-time problem (§5.4) and nothing else.

Chose: @siteed/audio-studio. Rejected: expo-audio (the obvious default).

whisper.rn decodes WAV PCM16 from a file path and nothing else. expo-audio on Android emits AAC/M4A. I found this out after wiring the entire capture flow to expo-audio and watching transcription fail on every file. audio-studio emits both formats from a single recording, which is the dual output I needed anyway - WAV for the model, M4A to keep.

It did not compile on SDK 56 (§5.4), which I patched rather than forked.

Chose: ggml-large-v3-turbo-q5_0 (~574 MB). Rejected: small-q5_1 (~181 MB), which I shipped first.

The small quant hallucinated and looped on real recordings - not "slightly wrong", but inventing sentences that were never said. I'm going to approve this output and then treat it as memory, so invented detail is the failure I can least afford. large-v3-turbo fixed it, and the turbo decoder stays fast enough to be usable. Tripling the download size was worth it.

There's a related decision hiding here: I set language: 'en' rather than 'auto'. Most of my conversations are Hinglish, and large-v3 with 'en' translates spoken Hindi into English rather than transcribing Devanagari. For a searchable relationship log, English output is more useful than verbatim output. One constant flips it back.

Chose: Qwen3-1.7B at Q8_0 (~1.83 GB), n_gpu_layers: 0, contexts loaded strictly sequentially. Rejected: Q4_K_M for speed; GPU offload; keeping both models warm.

Three sub-decisions, each with a measurement behind it:

• Q8_0 over Q4_K_M - Qwen's official GGUF repo ships only Q8_0. A 1.7B model is already at the edge of being able to hold a schema in its head; I didn't want to find out what a 4-bit quant of it does to structured output. Q4_K_M stays commented in modelManager.ts as the escape hatch if speed becomes the binding constraint.
• CPU over GPU - I tried both and the GPU delegate came out slower on this device class, so n_gpu_layers stays at 0.
• Sequential contexts - the two models together are ~2.4 GB of weights. Holding a Whisper context and a llama context resident at the same time on an 8 GB phone gets the app OOM-killed mid-pipeline. So the pipeline runs on a serialized promise chain: one native context at a time, transcription frees Whisper before extraction loads Qwen, and everything else that needs a model (like contacts-note extraction) enqueues on the same chain instead of racing it.

Chose: fetch from Hugging Face on first use into documentDirectory/models/, cache permanently. Rejected: shipping the weights in the APK.

A 2.4 GB APK is painful to sideload and worse to re-download on every release. Downloading at runtime also means I can swap a model by editing a URL instead of rebuilding - the registry in modelManager.ts is plain config, with alternative quants sitting in comments next to the active URL.

The non-obvious part is the failure mode. A partially-downloaded GGUF handed to llama.cpp surfaces as an uninformative native std::exception at load time, which is miserable to debug. So downloads are size-checked against the expected byte count (within 5%), and a truncated file is deleted and reported as a readable error instead of being cached and re-crashing forever. Concurrent ensureModel calls dedupe onto a single download task, because "manually install from Settings while the pipeline lazily downloads the same file" deletes the partial file out from under the other task.

Settings shows model state and download progress. 2.4 GB of weights is worth being explicit about.

Chose: every AI-derived field becomes a draft_field_suggestion with confidence and a source snippet, gated on review. Rejected: writing high-confidence extractions straight to the person.

This is the decision the rest of the architecture is built around. The tempting version is "auto-apply anything above 0.8 confidence", and I don't do it for one reason: a small model's confidence isn't calibrated, and this failure is invisible when it happens. A wrong bio doesn't throw an error. It sits on the page looking reasonable until I repeat it back to the person it's about.

Making review mandatory forced good structure everywhere else - suggestions had to be per-field rather than per-record so I could reject one line without losing the rest, and every suggestion had to carry its source snippet so I could check it in one glance instead of replaying audio.

The honest limitation: source snippets are derived by substring-matching the extracted value against the transcript. Verbatim fields get an exact snippet; paraphrased fields (a summarised bio) get nothing. That's a known gap I haven't closed yet.

Chose: tier is a manual field. The cadence engine can suggest a change; it can never make one. Rejected: deriving closeness from interaction frequency.

How often I talk to someone is not how close I am to them. I speak to a project collaborator four times a week and my oldest friend once a quarter, and a system that ranks the collaborator higher has misread both relationships. So frequency stays a second axis that only generates prompts, and the hierarchy stays mine.

The mechanics: cadence is counted over a trailing 90-day window against the median of the person's tier peers, and a tier needs at least 3 members before its median means anything. Above 2× the tier median suggests a bump up; below ⅓ suggests a move down. When I manually set a tier, that person's tier_change nudge is snoozed for 7 days - if I've just made a decision, the app doesn't get to immediately argue with me about it.

Memo-less check-ins exist for the same reason. If I see someone and don't record a memo, the cadence signal quietly goes stale. A check-in is a one-tap "this happened" that feeds the same window, so the nudges track what actually happened rather than how diligent I've been about recording.

Chose: read the device address book with expo-contacts. Rejected: Google People API via OAuth.

On a Google-synced Android phone these return the same data. The People API costs a Cloud Console project, an OAuth consent screen, and a token refresh path; expo-contacts costs a runtime permission prompt, the same as the microphone. The only price was one native rebuild. I'd revisit this only if I needed cloud-direct access independent of device sync.

The import branches three ways depending on what it finds:

• Already imported (the contact's source id is stored as a google_contact identity) → additive sync: add newly-present phones and emails only. A number added on the phone later gets pulled in; nothing existing is ever clobbered.
• Matches an existing person by name/phone/email → a pending review draft targeted at the best candidate. It goes through the same approval flow as a voice memo.
• No match → imported directly as a canonical person. An imported contact is trusted data rather than an inference, and making me approve 300 of them one at a time would make the feature useless.

Contact notes become a no-audio memo whose transcript is the note text, which then runs the normal extraction → draft → review pipeline. Reshaping the input to fit the pipeline I already had was less work than building a second ingestion path.

Chose: survivor-wins merge; the loser is tombstoned via mergedInto and its page redirects. Rejected: deleting the duplicate row.

Duplicates are inevitable - the capture pipeline matches people by exact name, so two unreviewed drafts about the same person with slightly different names produce two people. (I hit this with two "Wilson"s, which is how the feature got prioritised.)

A merge reassigns nine child tables from loser to survivor - memos, memo_people, identities, tags, relationships, drafts, interactions, search chunks, nudge dismissals

• with dedupe and self-loop guards, then folds the scalars: survivor wins, loser fills gaps, quickbits union, lastInteractionAt takes the max. The loser row is kept as a tombstone and filtered out of every list, so any stale id anywhere still resolves instead of dangling.

The detector pairs on normalized name, last-10-digits phone (so +91… matches 0…), email, and handles. Address is deliberately not a signal - people who live together are not the same person. And "not a duplicate" is recorded permanently in merge_dismissals, keyed by a sorted pair key so (a,b) and (b,a) collapse to one row. If the detector kept re-suggesting a pair I'd already rejected, I'd stop reading the inbox.

Chose: ship Convex full-text search over names and transcripts. Rejected: waiting for embeddings to ship search at all.

The spec always called for hybrid keyword + semantic search. Semantic search needs an embedding model, which needs a third model on the device, which needs a gated Hugging Face token proxied through a Convex action. That's a chunk of work standing between me and any search.

So I shipped the keyword half, with people ranked above memo hits (a name query should return the person, not a memo that mentions them). Semantic re-ranking layers on top later without rework - search_chunks is already a layered table (person summaries, memo chunks, event snippets, source text) waiting for a vector index and a dimension that I'll measure at runtime rather than hardcode.

Chose: two Expo config plugins. Rejected: hand-editing the generated android/ project.

android/ is generated by prebuild and gitignored. Every edit I made there worked perfectly on my machine and vanished on a clean checkout, which meant it would fail in CI on the first release build. Both fixes moved into plugins that re-apply on every prebuild:

• withReleaseSigning - wires a release signing config reading a gitignored keystore.properties, and falls back to debug signing when it's absent so expo run:android keeps working untouched on a dev machine.
• withAndroidPackaging - the Clerk Android SDK ships a duplicate META-INF entry that fails the merge task. This declares the exclusion via gradle.properties, the route Expo's own template supports, instead of injecting a Gradle block.

Both have unit tests. That seems like a lot for two build files, but they only ever execute on a CI runner I can't inspect while it's running.

captured → transcribing → transcribed → extracting → extracted
                                                   ↘ failed

Status lives on the memo row and is written at every transition, so a memo that dies mid-run is visibly stuck at a known stage rather than silently missing. Status writes are best-effort: if the device is offline the on-device work still completes and the status simply lags.

Failure semantics are per-stage. A transcription failure is a full stop - there's nothing downstream to do. An extraction failure keeps the transcript and skips the draft, so the memo is still searchable and still listenable. A draft-creation failure keeps both the transcript and the extraction and leaves the memo in extracting, so a retry has everything it needs.

Most of the extraction work went into the prompt rather than the model choice. A 1.7B model tolerates far less than I expected:

• Nested schemas were ignored. Asking for {person: {contact: {phone}}} produced confidently malformed output. A flat schema plus one worked example produced usable JSON. Each level of nesting I removed took a class of malformed output with it.
• The chat template had to be built by hand. Using llama.rn's jinja: true with messages, or a response_format: json_object grammar, threw a native std::exception at completion start with this GGUF. Constructing the ChatML prompt as a string - <|im_start|>system … <|im_end|> - works reliably.
• Reasoning mode had to be switched off. Qwen3 emits <think> blocks by default, which burn the token budget before any JSON appears. /no_think in the system prompt, and the parser strips the blocks anyway as a belt-and-braces measure.
• The parser is deliberately tolerant. It expects JSON-ish text, not JSON, and recovers from trailing prose, code fences, and minor malformation. It has the largest share of the test suite, since it's the component most exposed to bad model output.
• Generation is capped at n_predict: 512 with n_ctx: 2048. Extraction JSON is small; the cap exists so a non-stopping model can't grind for minutes. That single change took a cold Q8 run from ~3.5 minutes down to something usable, and halving the context halves the KV cache.

Settings: temperature: 0.2, top_p: 0.9, explicit stop tokens. Temperature stays low because the job is extraction - I want the same fields out of the same transcript every time.

llama.cpp failures reach JavaScript as the string std::exception, and that's the whole message. So the extraction module attaches a native log listener, keeps a rolling 40-line buffer of llama.cpp output, and re-throws any native error with the last 15 lines appended. Two separate root causes - a truncated model download and bad completion params - were both diagnosed from that buffer, and I don't think I would have found either without it. It costs almost nothing to add, and I'd wire it up on day one next time.

• @siteed/audio-studio doesn't compile on Expo SDK 56. Its Kotlin calls Promise.reject(code: String, …) while expo-modules-core declares code: String?. Fixed with bun patch and committed to patches/, pinned through package.json's patchedDependencies. A fresh bun install without it re-breaks the Android build.
• New Architecture is mandatory on RN 0.85 / SDK 56 - it can't be disabled. This was the M0 gate for the whole project, since it decided whether whisper.rn and llama.rn were viable at all. Both compiled fine. I checked this before building anything on top of it, since a failure there would have ended the project.
• Windows + llama.rn's postinstall. The artifact extraction fails because Git's GNU tar misreads C:\ paths. Worked around with a root postinstall script that forces bsdtar and fetches the artifacts bun blocks.

17 Convex tables. Every domain table carries ownerId and is read through owner-scoped indexes - including the search indexes, where the query filters on ownerId inside withSearchIndex rather than after it.

A few shapes worth calling out:

memos.clientId is generated on the device before anything touches the network. It makes the offline queue idempotent - re-flushing a queue after a crash can't create duplicate memos, because the insert is keyed on an id the device already chose.

quickbits is its own table, not an array on people (which is how it started). Each bit carries provenance, an event id grouping bits from one interaction, an occurredAt for recency windowing, and a pinned flag for facts that should survive past the window. An array column couldn't express "show me the last five interactions' worth, plus anything I pinned".

field_conflicts exists for the multi-device case. When two devices edit the same factual field from the same starting value, the second write to land finds the server value no longer matches what it was edited from. Rather than silently clobbering, it records both values and surfaces a review item. It's the approval queue idea applied to sync - when the app can't tell which value is right, it asks me.

person_relationships is populated by nothing yet. The typed-relationship data model is defined and the graph view is deferred. I'd rather have the table shaped correctly now than migrate people's data later.

One version, one git tag, one signed APK, built on GitHub Actions. No Play Store listing and no iOS build (no Apple account wired) - it installs as a sideloadable APK from the Releases page.

bun run bump <ver> → commit → push tag v<ver> → release.yml

• preflight asserts the git tag matches app.json's expo.version, then runs typecheck, lint, and tests. It fails in ~2 minutes instead of after a 15-minute native build.
• build-android runs prebuild → assembleRelease → signed APK, arm64-v8a only, and verifies the APK is not debug-signed before it's allowed to publish.
• publish cuts a GitHub Release with the APK attached.

Pushing to master deliberately does not build an APK. ci.yml runs typecheck/lint/tests on every push and PR and skips the native compile to stay fast; a separate manual workflow builds a debug APK on demand for when I've touched a native module or a config plugin. Three tiers, because a 15-minute native build on every push is slow enough that I'd start ignoring the results.

The version-match preflight and the debug-signature check exist because those are the two mistakes I'm most likely to make while cutting a release late at night.

What's still missing as of v0.1.0:

• Semantic search. Keyword search is live. The embedding model (EmbeddingGemma) is gated behind a Hugging Face token and needs a Convex action proxy; search_chunks has no vector index yet. Deferred, but the schema already accounts for it.
• The graph view. The original spec's Obsidian-style cluster view. The table exists; the UI doesn't.
• Remote push delivery. The device-side path is verified and the Convex-side scheduling, quiet hours, and digest batching are code-complete, but the actual FCM delivery leg was never wired live. The plan is Convex → FCM directly.
• Fuzzy matching at capture time. Auto-matching a memo to an existing person is still exact-name only. The merge tool cleans up afterwards; the pipeline doesn't prevent it up front.
• No auto-retry. A pipeline interrupted by an app kill leaves the memo at an intermediate status and waits for me. Deliberate for now - I'd rather it stall visibly than retry a 3-minute CPU-bound job in the background without being asked.
• iOS. Android-first was the plan and remains the state.

1. Test the riskiest native dependency first. Whether whisper.rn and llama.rn compiled under the mandatory New Architecture decided whether the project was possible at all. Everything else could have been worked around.
2. Buffer native logs from the start. std::exception on its own tells you nothing. The rolling log tail is what identified both of my native failures.
3. Size-check every model download. A truncated GGUF fails at load with an unhelpful error, and once it's cached it fails the same way every time.
4. Keep the output schema flat for small models. Flatten the shape and give one worked example. Nested schemas produced confident garbage.
5. Decide what the AI is allowed to write before building the AI. Making review mandatory improved the schema and the UI, and it's the reason I trust the data. Adding it afterwards would have meant a rewrite.
6. Anything hand-edited in a generated directory disappears on a clean checkout. Move it into a config plugin the same day.

Stack: Expo SDK 56 · React Native 0.85 (New Architecture) · TypeScript · NativeWind · Convex · Clerk · whisper.rn · llama.rn · @siteed/audio-studio · expo-contacts · expo-notifications · Jest · GitHub Actions · bun