tscaps

@tscaps/engine

A client-side caption engine for video

@tscaps/engine is an open-source TypeScript library that burns subtitle captions into video, end-to-end in the browser. It transcribes audio with in-browser Whisper (or reads an SRT, or accepts a hand-built document), splits and tags the transcript through a configurable pipeline, renders each output frame from a CSS-styled DOM tree, and encodes the result with WebCodecs through MediaBunny. No backend is involved at any stage.

How it works

The pipeline runs five stages over an input video. Every stage exposes an abstraction the consumer can replace; the defaults are sized for the common case.

  1. 1. Transcription

    The default WhisperTranscriber decodes the input audio with Web Audio and runs Whisper on WASM (or WebGPU when available) through @huggingface/transformers. Model weights are cached by the browser after the first download. Alternative built-in transcribers: SrtTranscriber for SubRip files and PassthroughTranscriber for documents the consumer builds directly. Any class implementing the Transcriber interface plugs in.

  2. 2. Splitting

    Words emitted by the transcriber are partitioned into screen-sized cues by a SegmentSplitter, then re-balanced into visible lines by a LineSplitter. Defaults: a CompositeSegmentSplitter chaining a sentence-boundary cut with a scaled-character budget, and a BalancedPixelWidthLineSplitter backed by a DOM-probed text measurer. Individual strategies — LimitByWords, LimitByScaledChars, PauseBased, SpeakerChange — are exposed for custom chains.

  3. 3. Tagging

    A StructureTagger runs once after splitting and assigns positional tags to every element — first-word-in-line, last-segment-in-section, and so on — that surface as CSS classes. Semantic taggers (RegexTagger, WordlistTagger, SpanTagger) can be chained to attach domain-specific tags based on word content.

  4. 4. Effects

    Pure document-transforming stages run last over the structured document. Built-ins include SmartPunctuationEffect, SmartLowercaseEffect, RemovePunctuationEffect, CarryQuotesEffect, and GapFreeEffect. Any class implementing the Effect interface can be added to the chain.

  5. 5. Rendering and export

    For each output frame, the active document is materialised as a DOM tree carrying the engine's classes and CSS custom properties for the current time. That tree is wrapped in an SVG <foreignObject>, decoded as an image, and rasterised onto a 2D canvas. The MediaBunny pipeline composites that raster with the decoded source frame and encodes the result through WebCodecs into an mp4 or webm container. Audio is passed through, transcoded, or discarded based on container codec compatibility.

What CSS sees

Caption styling is plain CSS. The engine attaches a fixed set of classes to every rendered element and exposes timing values as CSS custom properties so animations can thread through the narration timeline without any JavaScript runtime.

Element classes

  • .section — the active Section being rendered.
  • .segment — a caption block (typically one cue).
  • .line — a visible line within a segment.
  • .word — a single word within a line.
  • .letter — a single letter; only emitted when splitWordsIntoLetters is enabled.

State classes

Computed per frame from the current playback time and attached to the matching .word / .line:

  • .word-not-narrated-yet, .word-being-narrated, .word-already-narrated
  • .line-not-narrated-yet, .line-being-narrated, .line-already-narrated

Positional tag classes

Assigned once by the StructureTagger after splitting:

  • .first-word-in-line, .last-word-in-line
  • .first-word-in-segment, .last-word-in-segment
  • .first-word-in-section, .last-word-in-section
  • .first-line-in-segment, .last-line-in-segment
  • .first-line-in-section, .last-line-in-section
  • .first-segment-in-section, .last-segment-in-section
  • .first-section-in-document, .last-section-in-document

CSS custom properties

Each rendered element exposes timing values relative to the current frame. --on-…-starts and --on-…-ends are seconds until the event and go negative once it is in the past. --…-duration is a span. Drive animation-delay and animation-duration directly from them.

  • Section: --on-section-starts, --on-section-ends, --section-duration
  • Segment: --on-segment-starts, --on-segment-ends, --segment-duration
  • Per line state: --on-line-being-narrated-starts, --line-being-narrated-duration, and the matching not-narrated-yet / already-narrated trios.
  • Per word state: same trio applied to --on-word-being-narrated-starts and friends.
  • Letter-level: --letter-index, --letter-count.
  • Layout and frame: --subtitle-region-width, --subtitle-region-height, --subtitle-region-x, --subtitle-region-y, and --video-frame (a data-URL of the underlying video frame, only set when rendering.videoFrame.required is true).

Install

npm install @tscaps/engine

Requires WebCodecs (Chrome 94+, Edge 94+, Safari 16.4+, Firefox 130+). Node ≥20 is only needed for tooling.

Minimal example

Build a pipeline with an input video and a transcriber, await run(), write the returned Blob.

import { RenderPipelineBuilder, SrtTranscriber } from '@tscaps/engine';

const pipeline = new RenderPipelineBuilder()
  .withInputVideo(videoBlob)
  .withTranscriber(new SrtTranscriber(srt))
  .build();

const { blob } = await pipeline.run();
// `blob` is a Blob containing the captioned mp4

Read more

  • README →

    Six worked examples (SRT, custom CSS, alignment, splitters, karaoke, animations), the full document model, and the catalog of what each builder method does.

  • Renderer internals →

    Per-frame rendering pipeline, MediaBunny integration, browser caveats around <foreignObject>, video-frame embedding, and SVG filter authoring.

  • Source on GitHub →

    TypeScript, MIT licensed. Issues and pull requests welcome.

  • npm package →

    @tscaps/engine