The Premise
I'm working with Hamilton County ARES to do a postmortem on an event during a race where someone needed medical attention. Our objectives were fairly straight forward: What could we have done to improve outcomes.That requires a careful review of our communications during the event. Luckily everything was recorded, and it only totals about 17 minutes of audio. I first considered transcribing it by hand, then figured "nah, let's use an LLM." After reviewing the first file I ended up feeding in manual corrections, a lot of them, before deciding to try a few different models and approaches.
The Setup
I'm a complete amateur here. I don't build speech models; I'm a consumer hitting the tool over the head until I get what I need.All of the work was done in Fable 5 on high effort, trading more expensive tokens for a better outcome. I usually ran two agents: one focused on review and correction of the transcription, the other on the postmortem itself.
There's more I could do for efficiency. We'd talked in #AI about tooling so I'm not hand-feeding corrections to Claude one line at a time, and can mark and handle corrections more easily. If I keep doing this, that's the goal.
Hardware for everything below: one Linux box, 8 cores (Zen 2 server on RoushTech's infrastructure), 32 GB RAM, no GPU. Everything ran in a Docker container.
The Process
Pass 1: Whisper small.en
Like most transcription projects I started with Whisper, small.en via sherpa-onnx with fine-grained VAD (voice activity detection) segmentation.
What we got:
What was said:
Almost every callsign was busted. "K4 CLK" for K4ZLK was common, KN4EOV came out as "K and 4UV", random words were wrong. Hints we were on the right track, but unusable.
Pass 2: CrisperWhisper
Next we moved to CrisperWhisper 2.0 Large (a Whisper variant trained to transcribe verbatim: fillers, stutters, false starts), running on CTranslate2 with int8 weights so it fits on a CPU. The audio was fed in 240-second pieces cut at the quietest point near each boundary, and words were regrouped into transmissions wherever the gap between two word timestamps exceeded 1.2 seconds.We built a roster.txt with 5 callsigns for a callsign cleaner (joins spelled letters like "K N four E O V" into KN4EOV and snaps near misses to the roster) and a phrase repairer for things like "hot power count" to "half hour count".
The text was much better. But the speaker column on every line was ????, because there was no speaker attribution at all, and you can't get it from the text alone: callsigns are only spoken on some transmissions. So we moved on to diarization (figuring out who is speaking).
Pass 3: Diarization, two ways
Nets have procedure, and we can use this procedure to know the speaker. So rule one is just reading the words:- "this is X", "X for net (control)", "X out / clear / monitoring / secure": sender is X
- a trailing callsign after the content ("Thank you sir, KQ4ECU"): sender
- an opener of the form addressee, sender ("Net control, N4DWY"; "Go ahead, DWY, KN4EOV")
- "X, net control" where X isn't net control: X is calling in
- a leading bare callsign ("K4ZLK, they sent us a gator"): sender, unless wrapped in "go ahead"
- a transmission that opens or closes with "go ahead": net control, since only NC hands out the floor
CALLSIGN = tactical, tactical | misrecognition, misrecognition
W3UWU = run 3, run aid 3 | three three u w, three u w, 3uw
Rule two is the voice. Each transmission is run through a speaker embedding model, which turns a clip of speech into a vector (a fingerprint of the voice). Vectors from the same person land close together, measured by cosine distance (0 = identical, 1 = unrelated). Then agglomerative clustering groups them: start with every transmission alone, repeatedly merge the two closest groups, stop when the closest remaining pair is further apart than a threshold. Each cluster gets named by majority vote of the callsigns spoken inside it, so a transmission with no callsign inherits the callsign of its voice.
Labels ended up as:
- W3UWU: callsign spoken in that transmission
- ~W3UWU: voice-only inference
- W3UWU?: text and voice disagree
- ????: nothing attributable
Pass 4: Tuning the clustering
Lots of disagreements, lots of unattributed lines, so we started sweeping the threshold.| threshold | clusters | notes |
|---|---|---|
| 0.4 | 82 | fragments |
| 0.5 | 52 | KN4EOV split three ways |
| 0.6 | 30 | clean K4ZLK, N4DWY, AE0GG, KQ4ECU, KS4VY, KG4VQ; KN4EOV split in two |
| 0.7 | 13 | KN4EOV merged with K4ZLK |
| 0.8 | 5 | one blob |
Pass 5: Widening without collapsing
Net control's voice was still split across clusters, so lines that were clearly NC got labels like ~S23 (how we'd label unattributed but distinctive voices). Raising the threshold to 0.7 merged NC with K4ZLK, so instead we added a second step: build an average voice (a centroid) per station from transmissions where the spoken callsign and the cluster agree, then hand any leftover anonymous transmission to the nearest station if its similarity is above a cutoff.The first cut required similarity 0.4 and still left a few NC lines unlabeled. We only had six or so people on the net, so I asked to widen it and it settled at 0.3 (--assign-threshold 0.7, expressed as a distance).
One bug bit us here. "K four U V" is ambiguous by spelling (KN4EOV, KQ4ECU and KS4VY all score about the same), and it was snapping to KS4VY while being net control's voice. That dragged the KS4VY centroid toward NC. The fix was only letting a transmission contribute to a centroid when its text and its cluster agree.
Pass 6: The squelch crash as the boundary
Next we had boundary problems. An operator hesitating mid-sentence could split one keyup into two lines, and fewer words per line makes the voice model struggle. Worse, the word timestamps from the model are not exact. In "You got a bib number over? Two six five nine" the "Two" was stamped on net control's side of the boundary even though it was the start of K4ZLK's reply.But we're on a repeater! When a station unkeys you hear the squelch tail and static crash. We can ride that to find boundaries (when nobody is quick-keying, at least).
We measured it. The unkey shows up as an 80 ms broadband burst, and right after it the squelch-gated recorder drops to a dead-air floor around -60 dBFS. A pause inside a keyup, with the carrier still up, only drops to about -48 dBFS. That 12 dB of separation is the whole trick: gate at -54 dBFS, require 150 ms of it, and everything between gates is one keyup.
Then we rebuilt transcription around it. Each keyup is transcribed as its own unit, so a word can't land in the neighbouring transmission, and the start and end times come from the audio instead of the model. Transcribing 150 keyups one at a time was slow (about 3 per minute; the model has per-call overhead due to how Whisper's encoder works -- padding audio files to effectively 30s), so consecutive keyups are batched into 90-second chunks with one second of inserted silence between them and each word is mapped back to the keyup its midpoint falls in, now we're not paying the padding tax (and on my poor VM's CPUs, it matters).
Things cleaned up a lot, we stopped having breaks and run-ins all over the place.
What it can't split is a double: a station keying over the tail of another, luckily we had few of those and they didn't run long into each other.
Pass 7: A better voice model
Attribution was much better but still annoying. The speaker model was ECAPA-TDNN from speechbrain. We swapped in WeSpeaker ResNet34 (through pyannote.audio, ungated, no token needed) and re-ran the same purity check:| model | threshold | clusters | purity | net control |
|---|---|---|---|---|
| ECAPA | 0.6 | 28 | 0.88 | split 9 + 13 |
| WeSpeaker | 0.6 | 20 | 0.88 | one cluster of 25 |
| ECAPA | 0.7 | 13 | 0.77 | merged with K4ZLK |
| WeSpeaker | 0.7 | 12 | 0.74 | merged with K4ZLK |
Pass 8: Trying other transcription models
I asked whether a bigger or ATC-style model would do better, so we ran four alternatives through the exact same keyup pipeline:- faster-whisper large-v3 with the roster fed in as hotwords: the best native callsign recognition of anything we tried (N4DWY, KQ4ECU, even "station calling net control" came out right with no cleanup). But it hallucinates on noisy keyups ("no sound no sound no sound", a line of KKWCWZW YQWZW where CrisperWhisper gave "Roger, Roger. K"), writes numbers as digits, and drops the fillers and stutters. About 90 minutes of CPU, mostly the model retrying clips it suspects it hallucinated on.
- A Whisper large-v3 fine-tuned on air traffic control audio (jacktol's): all caps, spells out digits, gets some callsigns, and forces aviation phraseology onto ham traffic. "Go ahead, DWY, KN4EOV" came out as "GOOD EIGHT TWO ONE CLEARED FOR UOV". Completely useless. A minute per keyup, too.
- Two more ATC fine-tunes were queued behind it and cancelled.
Pass 9: The packet as the roster
The ICS packet for the event lists every station and assignment, and it was important as part of the postmortem, so we dumped all 15 callsigns into the roster with their tactical names. That fixed a bunch of guesses in one go: "Papa Lima Papa" is W4PLP at Bike Aid 3, not a police unit (Whisper would make it sound like they were Chattanooga PD, not relaying a message, the postmortem would then dictate Chattanooga PD was on our frequencies... they were not) ; "K4IDH" was actually KE4IDH at the EOC; "KG4VQ" was KG4OVQ, the liaison; "three three U W" was me, W3UWU, on Run 3.And we add another rule: labels and text always show the full callsign, never a suffix. If the roster knows who DWY is, the transcript says N4DWY.
The callsign cleaner picked up a lot of small cases along the way: two callsigns spoken back to back ("four I D H K N four E O V" is KE4IDH then KN4EOV), a fragment like "4IDH" resolving by unique suffix owner, glued tokens ("Nfour Dwy", "KfourZLK"), hyphenated spellings ("KN-four-O-Z"), and context (a spelled run like "E W I" that matches nothing snaps to a station heard in the previous two transmissions if it shares a letter position, so it becomes N4DWY). An acronym guard keeps EOC, EMS, PD and QRT from being snapped to callsigns; "at the EOC" briefly became "at the KN4EOV".
Pass 10: Repair rules that bit us
The phrase repairer did fuzzy matching (string similarity over token windows). It was fine with five phrases and dangerous with thirty:- a single-word phrase niner swallowed "in", "nine" and "enter"
- station calling rewrote "started cooling" and "aid station captain"
- Thanks. Out. rewrote "Thank you" (the similarity of "you" and "out" is 0.67 because of the shared "ou")
Pass 11: When ears beat models
Some things only I could settle from the audio: a KS4VY transmission that faded into static and got cut into three keyups (the voice fingerprints of the pieces were as far apart as two different people, so no automatic merge could rejoin them), a half-second sign-off of mine that the model rendered as "This", a keyup that carried nothing but produced a hallucinated "Roger,".Rather than turn each of those into a general rule, we added overrides.txt:
join 202605171317-REPEATER 01:03 01:09
speaker 202605171317-REPEATER 01:03 KS4VY
text 202605171338-REPEATER 01:13 [unintelligible]
Applied at render time, so the transcript files are never hand-edited; they're always regenerated from the model's output plus the roster plus the overrides. Keyups where the model produced no words now render as [unintelligible] instead of disappearing.
So Where Did We End Up?
It took less time at the keyboard than typing the transcript would have. Transcribing at spoken speed is hard; transcribing verbatim, with the [UH]s and stutters and false starts, is harder. Most of my time went into listening to flagged lines and telling the agent what was actually said.Can I reuse what I have across other nets? Partially, the squelch gate, teh procedure rules, the voice clustering, the roster system... all are flexible but may need per-net tuning, but artifacts still need clean-up.
There was some suggestion to fine tune using LoRa, but we need 5-10 hours of content -- maybe we'll get there some day.