Analysis and Corpus

OpenMontage builds and queries a local, CLIP-indexed corpus of video and image assets for reference-driven productions. The corpus draws from free and open archives such as Archive.org, NASA, and Wikimedia Commons, plus optional stock sources when API keys are present. This enables documentary-style workflows that rely on real footage instead of generated video.

Corpus Layout

Each production that uses analysis maintains its own corpus under the project workspace:

projects/<kebab-name>/corpus/
├── clips/                  # downloaded video and image files
├── thumbnails/
│   └── <clip_id>/
│       └── frame_00.jpg    # evenly spaced frames for preview
├── embeddings.npy          # (N, 512) L2-normalised visual vectors
├── tag_embeddings.npy      # (N, 512) L2-normalised text/tag vectors
└── index.jsonl             # one metadata record per clip

The index.jsonl and NumPy files stay aligned row-by-row. The entire projects/ tree is gitignored.

Core Analysis Tools

Four tools perform the heavy lifting. All run locally and require no paid API keys for basic operation:

  • video_analyzer accepts a URL or local file and returns a video_analysis_brief. Depths are transcript_only, standard, and deep. It chains metadata fetch, transcript extraction, scene detection, keyframe sampling, motion classification, and style profiling.
  • scene_detect wraps PySceneDetect (content and threshold methods) plus FFmpeg to emit start/end timestamps for each scene.
  • frame_sampler extracts JPEG frames by count, explicit timestamps, or scene boundaries.
  • transcriber produces word-level segments using WhisperX. Speaker diarization is available when an HF_TOKEN is configured.

Supporting tools (video_downloader, transcript_fetcher, audio_probe) are invoked automatically by the analyzer.

Reference-Driven Workflows

Pipelines that declare reference_input.supported: true (such as cinematic) activate analysis when a reference video is supplied at the start of a session. The agent:

  1. Runs the analysis tools listed in the pipeline manifest.
  2. Produces a video_analysis_brief containing source metadata, content analysis, structure (scenes and pacing_profile), style profile, narration transcript, keyframes, and replication guidance.
  3. Embeds the reference and user query through CLIP.
  4. Retrieves matching clips from the corpus using fused visual + tag similarity.
  5. Diversifies the selected set and builds edit_decisions that respect the reference's motion and timing characteristics.

Sub-stages (for example, a short sample preview) become active once the video_analysis_brief exists.

Request real-footage behavior explicitly: "use real footage only." The system then avoids paid video generation and falls back to stock or archive clips.

Retrieval and Diversity

The corpus supports:

  • Ranking by text query (visual and tag channels blended).
  • Nearest-neighbour search from a seed clip.
  • Maximal Marginal Relevance diversification to avoid repetitive cuts.

Motion score, duration, and shot-type filters can be applied during retrieval. Results feed directly into the edit stage.

Verification Before Use

Run preflight to confirm the local analysis stack and any configured stock providers are ready:

make preflight

Or inspect the live registry:

python -c "
from tools.tool_registry import registry
import json
registry.discover()
print(json.dumps(registry.provider_menu_summary(), indent=2))

See guides/project-workspace for the full directory contract and guides/running-pipelines for how to start a reference-aware session. For the list of pipelines that support reference input, see reference/available-pipelines.

Updated June 24, 2026