What 'on-device AI' really means (and what 'private cloud' doesn't)
When a software vendor tells you their AI is “private,” that word is doing a lot of work, and not always honest work. The phrase you actually want to interrogate is on-device AI. The plain on-device AI meaning is this: the computation happens on the hardware in front of you, and the raw material of the work, in our case the audio of a therapy session, never leaves that machine. Everything else, including the increasingly common marketing term “private cloud,” is a different arrangement with a different threat model. The distinction is not pedantic. For a clinician holding protected health information, it changes who could ever be in a position to see your client’s words.
The on-device AI meaning: three arrangements, one dividing line
Strip away the brochure language and there are really three ways an AI feature can run.
- On-device. Recording, transcription, and draft generation all happen on your Mac’s own processor. No audio, no transcript, and no metadata travels over the network for the work to get done. If you pulled the Ethernet cable and turned off Wi-Fi, it would still work.
- Cloud. Your audio (or its transcript) is uploaded to a vendor’s servers, processed there, and the result is sent back. This is how most well-known transcription and note tools operate, even ones that feel instant.
- Private cloud. A middle category that sounds like the first but behaves like the second. The audio still leaves your machine and is processed on remote servers. The “private” part usually refers to encryption in transit, a dedicated tenant, a promise of short retention, or a signed business associate agreement. Those are real protections. But they are protections applied after the data has crossed the network boundary, not instead of crossing it.
That boundary is the whole question. Not “is it encrypted,” not “do they have a good privacy policy,” but a simpler and more physical one: does the audio leave the machine at all? On-device is the only answer where the honest reply is no.
Why “private cloud” is a weaker claim than it sounds
A private cloud reduces who is likely to see your data. On-device changes who is able to. That gap matters in clinical work because your obligations do not end at the vendor’s good intentions.
Once a session recording reaches someone else’s server, you have introduced a third party into the room. That third party can be subpoenaed, breached, acquired, or simply audited by a subcontractor you have never heard of. Encryption in transit protects the data while it travels; it says nothing about what happens once it arrives and is decrypted for processing. A retention promise is only as durable as the company making it, and “we delete after a set window” is a policy, not a property of the system.
A privacy policy describes what a vendor intends to do with your data. On-device processing describes what they cannot do, because the data was never theirs to hold.
This is also why “we’re HIPAA compliant” should not end the conversation. A business associate agreement is necessary if you are sending PHI to a vendor, but it is a contract about liability, not a guarantee that disclosure won’t happen. We unpack that distinction at length in why “HIPAA compliant” is not enough. The shorter version: on-device processing lets you sidestep part of the question entirely, because there is no transmission to a business associate to govern in the first place.
How to actually verify a vendor’s claim
Marketing copy is not evidence. Here is how to test whether “on-device” or “private” means what you hope it means, in roughly increasing order of effort.
- Ask the airplane-mode question. “If I disconnect from the internet, does transcription still work?” A genuinely on-device tool will say yes without hedging. If the honest answer is “you need a connection,” the work is happening somewhere else.
- Read for the verbs. Watch for “uploaded,” “sent to our servers,” “processed in the cloud,” or “synced.” Watch equally for soft language: “your data is secure” and “enterprise-grade” describe handling, not location. The location is what you are checking.
- Ask where the model runs, not where the data is stored. A vendor can store transcripts locally while still sending the audio out for processing. Storage and processing are separate questions; ask both.
- Check for a business associate agreement requirement. If a tool needs you to sign a BAA, that is a strong tell that your PHI is leaving your machine. Tools that process on-device generally don’t need one, because there is no associate handling the data.
- Look at the network yourself. You don’t need to be an engineer. macOS includes Activity Monitor, which shows a Network tab listing what each app is sending and receiving. Record a short test with no real client present, watch whether the app transmits while it transcribes, and you have your answer in plain bytes.
| On-device | Private cloud | Cloud | |
|---|---|---|---|
| Audio leaves your Mac | No | Yes | Yes |
| Works offline | Yes | No | No |
| Third party can be compelled to disclose | No | Yes | Yes |
| BAA typically required | No | Yes | Yes |
If you want to see what fully local processing actually involves under the hood, including why modern Macs can transcribe a full session without phoning home, we walk through the mechanics in how on-device AI transcription works on a Mac.
Where this leaves you
On-device is the design choice behind CouchNotes: a session is recorded, transcribed, and turned into a SOAP, DAP, or BIRP draft entirely on your Mac, with audio that auto-deletes per your setting and never crosses the network boundary. The draft is a starting point, not a finished record. You remain the author of record, reviewing, editing, and signing every note before it goes in the chart. That is the honest scope of what local processing buys you, no more and no less.
None of this is legal advice, and the specifics of consent, retention, and documentation vary by state, board, and payer, so confirm the details with yours. But the underlying question is one you can answer for any tool, today, without a lawyer: trace the audio. If it stays on the machine in front of you, “private” is a description of fact. If it leaves, “private” is a promise, and you are the one who has to decide whether the promise is enough.