Immersion

‹

›

Immersion turns CEFR-graded classic novels into illustrated audiobook videos for language learners — full stories you can follow by ear and eye, with read-along text and vocabulary that appear as the narration unfolds.

It grew out of the same problem as Treasury: Albanian has almost no long-form learner content that is both interesting and level-appropriate. Treasury solved reading. Immersion is the video side — something you can put on YouTube and watch like a movie while the language stays mostly comprehensible.

Watch:

• Njëzet Mijë Lega Nën Dete (Twenty Thousand Leagues Under the Sea)
• Udhetim në Qendër të Tokës (Journey to the Center of the Earth)

Repo: github.com/mark-cd-hudson/immersion

What you see in a video

Each chapter is a timed composition:

• Scene illustrations that fade through key moments as the story progresses
• Multi-speaker narration — a narrator plus distinct voices for the main characters
• Read-along text with word-level highlighting synced to the audio
• Vocabulary panel that introduces new words (with learner definitions and thumbnails for visual terms) as they first appear
• Scored underscore generated per chapter and timed to the narration
• Page-turn transitions between chapters when the full book is concatenated

How it works

Two halves: a Python generation pipeline that produces every asset, and a Remotion renderer that turns those assets into video.

Generation starts from a graded-reader markdown of a public-domain novel (the same kind of CEFR rewrite Treasury uses). For each book it:

1. Splits chapters and designs recurring characters and settings (with reference sheets so illustrations stay consistent)
2. Segments the text for TTS, narrates with Gemini multi-speaker voices, and measures timing
3. Aligns word-level timestamps onto the original text via speech-to-text
4. Designs and generates timed chapter illustrations (OpenAI image models + a style reference)
5. Composes ambient music per chapter (Lyria)
6. Builds a book dictionary with translations, learner definitions, and illustrated entries
7. Creates a movie-poster thumbnail, YouTube description, and SRT subtitles
8. Exports per-chapter manifests for the renderer

Rendering is a Remotion (React) project: each chapter is a composition of artwork, audio, read-along, and vocab UI; chapters are then stitched into a full-book video with page-turn transitions.

Books produced so far (Albanian, Jules Verne):

• Njëzet Mijë Lega Nën Dete (Twenty Thousand Leagues Under the Sea)
• Udhetim në Qendër të Tokës (Journey to the Center of the Earth)
• Rreth Botës në Tetëdhjetë Ditë (Around the World in Eighty Days)

Status

Working end-to-end pipeline; first full book published on YouTube. Related: Treasury, Albanian Videos.