Generating Voice from Text Using Local Models

I've been building a personal AI engine that runs entirely on my Mac Studio M4 Max. Text generation was the first step. The natural next question was: can I generate voice locally too? Turns out, yes — and the results are surprisingly good.

The Same Bedtime Page, Four Voices

The newest example is the goodnight page from Susu the Cat, the second of my son’s bedtime books. Same script, four voices — one in the cloud, three on the Mac Studio. The first two are both my voice:

When I first made this comparison, only the ElevenLabs version sounded like me; the local engines used preset voices. That has changed. The second version is my own voice, cloned on the Mac Studio. It follows the [softly] and [whispers] tags written into the script, just as ElevenLabs does. The two preset voices show the other problem I found: they read the same page more than 50% slower. That matters for a picture book, where the next page shouldn’t turn before the sentence ends. My local clone lands in between: 10.4 seconds, against 8.6 for ElevenLabs.

What Changed at ElevenLabs: Eleven v4

The bedtime books are narrated with Eleven v4, which is now my default. The big change from the older Multilingual v2 is how you steer it. v2 had dials — stability, style — and you tuned delivery by turning them. v3 and v4 ignore the style dial entirely and take audio tags written into the script instead: [warmly], [softly], [whispers], plus <break time="0.5s" /> for pauses.

One pleasant surprise: on my account, v4 used roughly a tenth of the quota the script length suggested — about 740 credits for an 80-clip, 4,700-character book — and the tags and breaks didn’t appear to count at all. Your plan may differ, so measure before you budget.

Local Voices Inside H_AI: Voicebox

In February I said the next step was building voice into my AI engine. It’s done, through Voicebox, an open-source local voice studio that runs on Apple Silicon through MLX. H_AI talks to it over a local API for both directions: speech-to-text (Whisper) when I record a question on my phone, and text-to-speech when I ask for an answer read aloud.

Engine	Type	Languages	Best For
Kokoro	54 preset voices	8, incl. English	Fast drafts and auditions — a 327 MB model
Qwen Custom Voice	9 preset voices	10	Delivery directions written in plain language
Qwen, LuxTTS	Voice cloning	10 / English	A voice from a short reference recording
Chatterbox	Voice cloning	23, incl. Arabic	Arabic — with a reference voice
TADA 3B	Voice cloning	10, incl. Arabic	Multilingual cloning, the heaviest at 8 GB
Fish S2-Pro (new)	Voice cloning + inline tags	Multilingual	My own voice — now H_AI’s default

The catch for my projects is Arabic. None of the local engines ship an Arabic preset voice; only the cloning engines speak it, so Arabic narration needs a clean 15–30 second recording to clone from — ideally of someone speaking Arabic. In the cloud, I just pick an Arabic voice.

My Own Voice, Locally: Fish S2-Pro

Cloning my own voice was at the top of my “what’s next” list, so I held a bake-off. I took the newest and most-liked open-source voice models on Hugging Face and gave each one the same short recording of my voice. Each then read the same bedtime book, page by page, and I ran every page back through speech-to-text to count the words it got wrong.

Model	Word errors	Follows [tags]	License
Fish S2-Pro	0.0%	Yes	Non-commercial
VoxCPM2	2.9%	No	Apache 2.0
Fish Drama-3	3.4%	Yes	Non-commercial
OmniVoice	3.4%	No	—
ElevenLabs Eleven v4 (cloud)	4.1%	Yes	Paid
Qwen3-TTS	4.3%	No	Apache 2.0
Chatterbox	4.9%	No	MIT

The surprise is the middle of the table: every local model read the words as accurately as ElevenLabs. Accuracy is no longer what separates them. What does is whether the voice sounds like me, and whether it can follow delivery. Fish S2-Pro won on both. It is also one of only two models that understand the tags already in my scripts. A line written as [whispers] Goodnight, Yusuf. for ElevenLabs is whispered locally too, so one script now works on both sides without stripping anything.

Two honest caveats. The license is non-commercial, which is fine for my site and my son’s books, but would rule it out for a product. And for this first version, the clone learned from about 19 seconds of my ElevenLabs narration rather than a fresh recording — a copy of a copy. A clean recording of my own is the next improvement.

To make it everyday, I added Fish S2-Pro to Voicebox as a new engine and made my cloned voice the default in H_AI. When I ask for an answer read aloud, it now comes back in my voice, rendered on the Mac Studio.

Egyptian Arabic, Locally: Not Yet

Every bedtime book also has an Egyptian Arabic edition, so I ran the same test in Egyptian. Two texts: the Egyptian edition of Bismillah, which ElevenLabs already narrates, and a new six-page story I wrote for the test. The general models list Arabic but not Egyptian, so I added the Egyptian-tuned open models I could find: VoiceTut (OmniVoice trained on about 380 hours of Egyptian speech), NileTTS and EGTTS (both built on XTTS-v2), and Habibi. The cloning models all copied the same short Egyptian reference voice; VoiceTut also has built-in Egyptian voices.

Model	Bismillah (Egyptian)	New story	License
ElevenLabs Eleven v4 (cloud)	16.2%	—	Paid
XTTS-v2	15.0%	27.5%	Non-commercial
OmniVoice	16.1%	28.1%	—
VoiceTut, voice “Hossam”	17.1%	29.8%	Apache 2.0
Fish S2-Pro	18.6%	26.5%	Non-commercial
NileTTS	19.1%	56.5%	Apache 2.0
VoiceTut, voice “Omar”	19.9%	24.7%	Apache 2.0
Qwen3-TTS	21.4%	27.0%	Apache 2.0
Habibi (Egyptian)	22.8%	26.9%	Apache 2.0
EGTTS	38.4%	34.4%	Non-commercial

Read this table differently from the English one. Here the speech-to-text is part of the problem: Whisper mishears Egyptian on its own, so even the ElevenLabs narration that ships in the book scores 16%. Most local models land in the same 15–30% band, inside that noise, so the numbers can’t rank them. The clearest sign is Qwen3-TTS: it doesn’t support Arabic at all, yet it scores in the same band.

What the numbers can show is a voice that breaks. NileTTS swung from 19% on one text to 56% on the other, and Habibi fell apart completely when it cloned my English recording instead of an Egyptian one (74% and 82%). So the scores tell me which clips to throw away, not which voice to keep. I haven’t judged the Egyptian voices by ear yet; until I do, my Arabic narration stays with ElevenLabs.

One Script, Two Backends

The newest piece is a batch voice-over tool in H_AI. It takes the same script file I use for ElevenLabs — one entry per page — and renders every page locally through Voicebox, with the same file names. A story player that plays audio/en/p1.mp3 doesn’t care which one made it.

Most local engines have never heard of [warmly] and would read it out loud, so for them the tool strips the tags, splits the script at every <break>, renders each piece, and splices them back together with real silence of the requested length. As the tool’s notes put it: pacing survives the move; emotion does not. With Fish S2-Pro the tags can stay in the script: it reads them as directions, so only the breaks become silence and the emotion survives the move too.

After each batch it checks the results the same way my ElevenLabs pipeline does — flagging clips that are silent, too short, or stretched. Stretched is the clever one: garbled speech still decodes as a perfectly valid file, but it takes far longer per character than its neighbours, so it shows up as an outlier.

Lessons from the Bedtime Stories

Text-to-speech falls apart on bare sound words. My first pass at the animal sounds fed the model strings like Mooooooo with expressiveness turned up, and the output came back warbling. Nonsense gives the model no context. Wrapping each one in a short sentence — “the cow says moo” — and turning expressiveness down fixed it completely.

Write for the ear. Numbers spelled out, no abbreviations, sentences under about 200 characters, one idea per page. Every script goes through a small linter before any audio is generated.

Check the words, not just the file. When I made the local versions of the Susu pages for this article, I ran every clip back through speech-to-text to confirm it said what the script said. The goodnight page came back word-for-word from all three voices — which is why it’s the example above. The new pipeline does this check automatically, one sentence at a time: if speech-to-text hears the wrong words, or the pace is off, that sentence is generated again, up to four times, and the best take is kept.

The Podcast, Then and Now

This is where the project started. In February I wrote a five-minute podcast script about my Personal AI Engine project and generated it twice — once locally with VibeVoice, once with ElevenLabs using a clone of my voice. In October I ran the exact same script, word for word, through the two new local engines, so all four can be compared directly.

February 2026

October 2026 — the new local engines

Version	Runs	Length	Time to Generate
VibeVoice 7B	Local	4:31	A few minutes
Kokoro	Local	5:10	26 seconds
ElevenLabs (my voice)	Cloud	5:24	—
Qwen Custom Voice	Local	5:34	About 9 minutes

The new versions were made with the batch voice-over tool described above: the script went in as ten paragraphs, each was rendered separately, and the pieces were joined with a short pause between them. Kokoro is the surprise — a five-minute episode in less time than it takes to listen to the first paragraph. Running every paragraph back through speech-to-text, both new versions matched the script about 98% word for word; nearly all the differences were spelling (“Fast API” for FastAPI), not misreadings.

One thing to keep in mind while listening: the script is from February, so it describes the project as it was then — DeepSeek, Llama and 22 skills. I kept it unchanged on purpose, so every version reads the same words.

What is VibeVoice?

VibeVoice is a frontier long-form conversational TTS model originally developed by Microsoft and now maintained by the open-source community. It generates expressive, multi-speaker audio from plain text — think podcast conversations, narrations, even content with background music that it generates spontaneously based on context. It’s still what I reach for when a script has more than one speaker.

The key innovation is its architecture: continuous speech tokenizers running at 7.5 Hz combined with a next-token diffusion framework. An LLM handles textual context while a diffusion head generates the acoustic details. The result is speech that sounds natural, with proper pacing, intonation, and speaker consistency.

Model	Max Duration	Speakers	Best For
VibeVoice 0.5B	Real-time	1	Low-latency live TTS
VibeVoice 1.5B	~90 min	Up to 4	Long-form podcasts
VibeVoice 7B	~45 min	Up to 4	Most realistic output

Running It Locally

With 64 GB of unified memory on the M4 Max, I can comfortably run the 7B model — the largest and most realistic option. Clone the VibeVoice repo, install dependencies with uv pip install -e ., download the model weights, and run inference from a text file with speaker labels like Speaker 1: Hello, welcome to the show.

Generation takes a few minutes for a full podcast episode. Not real-time, but completely acceptable for producing content. And every bit of it happens on my machine — no data leaves my network, no API calls, no per-minute charges.

Local vs. Cloud, Eight Months Later

I use ElevenLabs for anything that ships — the bilingual narration in my Islamic Kids Website, the Wireless 101 podcasts, and the bedtime books. Local covers everything before that. Here’s the honest comparison now:

What’s Next

In February, the next step was voice input. That’s done — I can record a question on my phone and H_AI transcribes it locally. What’s next now:

Cloning my own voice locally is done. What’s left: a proper recording to clone from, instead of my ElevenLabs narration. Arabic narration locally — the Egyptian test above narrowed it to a few candidates, VoiceTut among them, but speech-to-text can’t pick the winner, and I haven’t judged them by ear yet, so for now Arabic stays in the cloud. And more bedtime books — the series grows with my son, and each new one is another chance to see how far the local voices have come.

The tools are here. The hardware is capable. The open-source community keeps pushing the models forward. Running your own AI voice pipeline is no longer a research project — it’s a weekend project.