Canvas Timeline

Documentation

Media adapters

Connect timeline clips to native media or decoded Mediabunny previews without putting media-library policy in the timeline.

Browse documentation

Choose the preview boundary

Canvas Timeline keeps clips lightweight. A clip stores a stable sourceId, while your application owns files, URLs, proxies, offline state, relinks, and export media. A media adapter receives the app-resolved preview source for each ID and synchronizes it with timeline playback.

Choose the HTML media adapter when one native <video> or <audio> element can represent the preview. Choose the Mediabunny adapter when you need decoded canvas frames, separate visual/audio scheduling, local files, custom inputs, or track selection.

Both packages keep their imperative roots framework-free. React hooks live in the explicit ./react entry points:

import { useHTMLTimelineMedia } from '@techsquidtv/canvas-timeline-html-media-adapter/react';
import { useMediabunnyTimelineMedia } from '@techsquidtv/canvas-timeline-mediabunny-adapter/react';

The shared TimelineMediaSyncAdapter, source, timing, lifecycle, operation, and error contracts live in @techsquidtv/canvas-timeline-core. Use useTimelineMediaSync from the React package when implementing a different decoder, renderer, or external clock.

Define one resolved source

The common source has two required fields:

const sources = [
  {
    sourceId: 'camera-a',
    input: '/media/camera-a.mp4',
  },
];

sourceId joins the descriptor to timeline clips. input is the media your application selected for this preview session. Ordinary inline URL descriptors do not need useMemo; hooks reconcile definitions and preserve adapters and loaded media when values have not changed. Equivalent React source props also preserve pending and completed imperative replaceSource() calls. A semantic change to the supplied registry still reconciles the complete registry and supersedes pending replacements.

Use fallbacks only for ordered, content-equivalent ways to load the same choice—for example, an HLS endpoint followed by a direct file, or a CDN URL followed by an offline cache entry:

import { HLS_FORMATS } from 'mediabunny';


const sources = [
  {
    sourceId: 'camera-a',
    input: {
      kind: 'url',
      url: manifestUrl,
      formats: HLS_FORMATS,
      urlSourceOptions: { maxCacheSize: 32 * 1024 * 1024 },
    },
    fallbacks: [directFile],
  },
];

Mediabunny accepts URL strings, URL, Request, Blob, and File directly. Use its object descriptors only for explicit formats and UrlSource options, an existing Mediabunny input, or an input factory. Keep Request, Blob, File, factory, selector, and custom option identities stable when their identity represents a distinct resource or executable policy.

Keep proxy policy in the media library

A proxy is not a decoder fallback. It is a separate media-library choice with its own purpose, lifecycle, quality, and timestamp mapping. Resolve the choice in app state and replace the adapter source without changing timeline clips:

const asset = {
  sourceId: 'camera-a',
  original: { input: originalFile },
  proxies: {
    editing: {
      input: proxyUrl,
      timing: {
        sourceTimeSeconds: cameraTimestamp,
        mediaTimeSeconds: 0,
      },
    },
  },
};


const selected = useProxy ? asset.proxies.editing : asset.original;


await media.adapter.replaceSource({
  sourceId: asset.sourceId,
  ...selected,
});

The optional timing anchor says that sourceTimeSeconds and mediaTimeSeconds identify the same frame or sample. Omit it when logical source timestamps already equal media timestamps. This keeps epoch/live source time, proxies, transcodes, and trimmed cache files aligned.

Mediabunny preserves negative decode timestamps in firstTimestampSeconds and maps them into sourceFirstTimestampSeconds. Presentation starts at the later of zero and the first timestamp; durationSeconds excludes negative preroll. endTimestampSeconds is an end timestamp, including for Matroska files whose container duration is measured relative to their first packet.

Use setSources(nextSources) when replacing the complete registry. Use replaceSource(source) for one app-level media choice, and retrySource(id) to retry the preferred input and its fallbacks. Replacement and retry return a discriminated result with ok, sourceId, and either a configured/ready state or a machine-readable failure reason.

Understand readiness and lazy loading

HTML and Mediabunny have intentionally different readiness boundaries:

  • useHTMLTimelineMedia().ready means its returned callback ref connected a native media element. Read sourceStateById to determine whether a source is idle, loading, recovering, ready, or failed.
  • useMediabunnyTimelineMedia().ready means at least one source is registered and can be resolved. Sources start idle and load only when their clips are active or explicitly preloaded.

Mediabunny exposes lifecycle control for larger projects:

const result = await media.adapter.preloadSource(nextSourceId);


if (!result.ok) {
  console.error(result.reason, result.error);
}


media.adapter.unloadSource(previousSourceId);

preloadSource warms a registered source without changing proxy/original policy. unloadSource releases its decoder and audio resources but keeps the definition registered; the source returns to idle and reloads when needed. Module-loader failures use the same discriminated operation result and failed source state as input failures. A later retrySource() call invokes the lazy module loader again.

Adapter disposal is terminal. Work already in flight releases staged resources without publishing late state. New loading, decoding, rendering, or mutating calls after dispose() throw or reject, while read-only state and idempotent teardown calls remain safe. React hooks enforce this ownership boundary when Strict Mode or option changes recreate an adapter.

sourceStateById snapshots are referentially new when observable state changes. Each state includes the selected input index, ordered attempts, and terminal error. Mediabunny ready states also expose source-domain and media-domain timestamps, duration, dimensions, rotation, detected frame rate, and audio sample rate.

Select tracks and configure audio

Mediabunny uses the primary video and audio tracks by default. Supply selectTracks(context) when the editor needs an alternate camera angle, language, commentary mix, or another explicit track. The context contains all available tracks plus the logical source and selected preferred/fallback input.

The adapter creates an audio graph only after loading a source with a decodable audio track. A supplied AudioContext and destination remain caller-owned; disposal closes only a context created by the adapter.

Use setVolume() and setMuted() for imperative runtime changes. In React, updating audio.volume or audio.muted changes the master gain without disposing the adapter or reloading sources.

Browser audio activation is separate from visual transport. The high-level sync hook starts activation without awaiting it, so suspended audio cannot prevent video playback. Observe adapter.audioStatus for suspended, running, or degraded state. Imperative integrations can call requestClockActivation() directly. An activation requested before lazy media loading is retained until a selected decodable audio track creates the graph; video-only media never resumes a caller’s suspended context.

The Mediabunny adapter itself implements TimelineMediaSyncAdapter, so pass the adapter directly to useTimelineMediaSync(). Its transport clock is independent of individual loaded source controllers: lazy source transitions, unloads, and runtime input recovery do not reset timeline time. When synchronization reaches a source whose decoder is recovering, it waits for the fallback controller to commit. If recovery exhausts the source’s fallback chain, synchronization reports sync-failed and the high-level transport pauses; superseded recovery work cannot publish an error. With Mediabunny 1.56.0 or later, background URL and Blob reads use the same recovery path even when no read promise is pending. A supplied urlSourceOptions.handleUnhandledError also receives the original error as an observer. For supplied inputs and input factories, configure error callbacks on the underlying Mediabunny source when constructing it.

Terminal failures remain observable on later ticks, seeks, and preload requests until retrySource(), replaceSource(), an unload, or a changed source definition explicitly resets that source.

Apply playback ranges and errors

High-level adapter hooks accept the same range policy as timeline playback:

const media = useMediabunnyTimelineMedia({
  sources,
  layers,
  playbackOptions: {
    respectInOut: true,
    loop: true,
  },
  onError: (error) => reportPreviewFailure(error.reason, error.message),
});

External clock ticks pass through Core range enforcement. When looping is enabled, the high-level sync hook seeks and restarts the adapter clock at the range start. At the lower-level useTimelineMediaPlayback boundary, looping is enabled by supplying the loop(timelineTime, activeLayers) realignment callback instead of a boolean; this prevents a clock from looping without a way to seek.

onError receives TimelineMediaError. Use its stable reason for program flow and message for logs or UI. Rendering and scheduling failures report sync-failed and pause both clocks, including terminal asynchronous iterator recovery failures. Recoverable iterator failures first block the next synchronization while the fallback controller loads. Rejected loop realignment reports loop-failed through the same callback and also pauses both clocks. Play commands await startup synchronization before returning a discriminated TimelineMediaPlayResult, and media-aware playback-rate commands likewise resolve only after the external clock and active layers are synchronized. This lets callers handle failures without parsing error text or receiving a stale success result. Paused preview seeks share the same adapter-operation ordering as startup, looping, ticks, pause cleanup, and rate changes. This matters when a lazy source load or decoded-frame request finishes after the user has already scrubbed or started playback at a newer position.

If React replaces an adapter instance while it owns playback, the synchronization hook cancels pending startup, stops the previous clock, pauses the timeline, and primes the replacement at the current position. This keeps native element replacement and decoder-controller recreation from leaving the timeline driven by a detached or frozen clock. The packaged hooks supply adapterIdentity automatically. Custom useTimelineMediaSync() integrations should pass their underlying controller as adapterIdentity; inline callback facades may omit it and do not need memoization.

Continue reading

Compose media-aware keyboard transport

Pass your media transport to the existing keyboard scope so Space starts the external clock through the same path as the play button:

<Timeline.KeyboardScope
  commandHandlers={{ togglePlayback: () => (media.playing ? media.pause() : media.play()) }}
>
  {/* Timeline surface */}
</Timeline.KeyboardScope>

Here media is the result of useTimelineMediaSync, useHTMLTimelineMedia, or useMediabunnyTimelineMedia. The scope waits for returned promises before notifying onCommandResult. Adapter errors still reach the media hook’s onError; unexpected thrown or rejected handler errors reach the scope’s onCommandError.