Engineering Blueprint: Sovereign AI Video Production Pipeline (2026)
Date: August 2026
Author: Sentinel Integrations (Otto / Autonomous Video Studio)
Classification: Internal Technical Architecture & Production Standard
Operating Standard: INSAD Framework & Multi-Agent Studio Topology
1. Reference SOTA Technology Matrix (2026)
| Pipeline Layer | Winning Technology | Role in Sentinel Stack |
| :--- | :--- | :--- |
| Anchor Image Generation | FLUX.1 Dev + Nano Banana Pro / LoRA | Frozen high-fidelity "SOUL frame" per shot |
| Subject & Identity Lock | PuLID-FLUX / InstantID / IP-Adapter FaceID | Consistent subject/hardware identity across cuts |
| I2V Motion Engine | Wan 2.1 I2V (Quality) / LTX-Video 2.3 (Speed, 24fps) | Drift-free 5–8s continuous camera shots |
| Camera Trajectory Control | Wan-Camera LoRA / CameraCtrl / MotionCtrl | Explicit 3D camera translation & rotation (JSON) |
| Structural Conditioning | Depth-Anything-V2 / DWPose / Canny | Guarantees layout lock and zero morphing |
| Audio-Visual Alignment | WhisperX (wav2vec2 forced align) | Word-level timestamps for kinetic captions |
| Programmatic Motion | Remotion 4 (TSX/React) + p5.js (Playwright 60fps) | Alpha-channel kinetic typography & data telemetry |
| Master Compositing | FFmpeg filter_complex + sidechain ducker | 5-layer composite with -22dB ducked audio master |
| Director Reasoning | DSPy RAG + local Ollama (qwen2.5-coder:7b) | Compiles markdown briefs into ShotSpec v1 JSON |
2. The Anchor-First (I2V) Workflow: Eliminating Drift
Core Principle: Never allow the video model to invent visual identity from pure text. Always anchor to an approved high-resolution still.
[SOUL.md & Brand Spec] ──┐
[SHOT_LIBRARY.md Specs] ─┼──> DSPy Shot Planner ──> shot.json (Framing, Lens, Move, Mood)
│ │
│ ▼
│ FLUX.1 Dev + Hardware LoRA ──> anchor_hero_XX.png (2048x1152)
│ │
│ ▼
└────> Wan 2.1 I2V / LTX 2.3 ──> raw_clip_XX.mp4 (24fps, 5-8s)
(Anchor + Camera JSON)Operational Rules (DP & Generator Agents):
1. One Anchor Per Shot: Staged at 01_anchors/anchor_hero_XX.png with complete parameter metadata (anchor_meta.json).
2. Anchor Pass-Through Decay: The Image-to-Video node ingests the anchor at strength 1.0 on frame 0, decaying to 0.6 by frame 24 to prevent static freeze-frame syndrome while retaining 100% geometry fidelity.
3. Deterministic Fallback: If diffusion motion jitters, render a Depth-Animated ControlNet pass (Depth-Anything-V2 depth map transformed along a 3D vector) — geometry drift is mathematically impossible on the depth path.
3. Explicit Camera Trajectory Control (Ranked by Determinism)
1. Wan 2.1 Camera-LoRA + Trajectory JSON (Recommended):
`json
{
"camera": {
"type": "dolly_in",
"speed": 0.45,
"start": [0, 0, 0],
"end": [0, 0, 1.2],
"fov": 35
}
}
`
2. CameraCtrl / MotionCtrl ComfyUI Nodes: Accepts 6-DoF Plücker camera matrices exported from Blender or programmatic scripts.
3. AnimateDiff v3 Motion LoRA Library: Lightweight motion tags (zoom-in, pan-left, orbit-right, rack-focus) for rapid ideation.
4. Depth-Anchored 3D Control: Simple Python affine matrix transform applied over a depth map, fed into ControlNet Depth.
4. Automated Audio-Visual Synchronization & Pacing
Kokoro TTS (:8880) ──> scene_XX_voice.mp3
│
▼
WhisperX (:9000) ──> word_timings.json (word_start, word_end, word_text)
│
├─> Remotion / Mograph Engine ──> kinetic_captions.webm (Alpha channel)
├─> Drives exact I2V motion duration: ceil(sum(word_end)) + 1.0s settle buffer
└─> FFmpeg Sidechain Ducker:
[1:a]sidechaincompress=threshold=0.03:ratio=8:attack=5:release=200[bgm_ducked]5. Multi-Layer Master Compositing (FFmpeg Filter Complex)
Compositing Stack (Bottom $\rightarrow$ Top):
1. Layer 0: Raw I2V Background Clip (scene_XX_i2v.mp4, 1080p @ 24/30fps)
2. Layer 1: p5.js / Manim CE Algorithmic Vector Flow (flow_field.webm, Alpha)
3. Layer 2: Kinetic Typography & HUD Callouts (captions.webm, Alpha)
4. Layer 3: Sentinel Integrations Lower-Thirds & Enterprise Badge (Static PNG w/ Alpha)
5. Audio Mix: Kokoro Voiceover (0dB) + Ambient BGM (-22dB ducked) + UI Interaction SFX
ffmpeg -y \
-i bg_clip.mp4 -i mograph_alpha.webm -i captions_alpha.webm -i logo_badge.png \
-i voiceover.mp3 -i bgm_track.mp3 -i sfx_beep.wav \
-filter_complex "
[0:v][1:v]overlay=shortest=1[comp1];
[comp1][2:v]overlay[comp2];
[comp2][3:v]overlay=W-w-40:40[vout];
[5:a]volume=0.35[bgm_vol];
[4:a][bgm_vol]sidechaincompress=threshold=0.05:ratio=6:attack=5:release=250[ducked_mix];
[ducked_mix][6:a]amix=inputs=2:duration=longest[aout]
" -map "[vout]" -map "[aout]" \
-c:v libx264 -crf 17 -preset slow -pix_fmt yuv420p \
-c:a aac -b:a 192k -movflags +faststart \
06_finals/master_composed.mp46. ComfyUI Cloud Headless Payload Standard
{
"prompt": {
"1": {
"class_type": "CheckpointLoaderSimple",
"inputs": {"ckpt_name": "flux1-dev-fp8.safetensors"}
},
"2": {
"class_type": "LoraLoader",
"inputs": {
"model": ["1", 0],
"clip": ["1", 1],
"lora_name": "sentinel_hardware_hero_v1.safetensors",
"strength_model": 0.85,
"strength_clip": 0.85
}
},
"3": {
"class_type": "CLIPTextEncode",
"inputs": {"clip": ["2", 1], "text": "{{POSITIVE_PROMPT}}"}
},
"4": {
"class_type": "LoadImage",
"inputs": {"image": "anchor_{{SHOT_ID}}.png"}
},
"5": {
"class_type": "WanI2VSampler",
"inputs": {
"model": ["2", 0],
"positive": ["3", 0],
"image": ["4", 0],
"camera_trajectory": "{{CAMERA_TRAJECTORY_JSON}}",
"frames": 144,
"cfg": 6.0,
"seed": "{{SEED}}"
}
},
"6": {
"class_type": "VHS_VideoCombine",
"inputs": {
"images": ["5", 0],
"frame_rate": 24,
"filename_prefix": "shot_{{SHOT_ID}}"
}
}
},
"client_id": "sentinel-nexus"
}7. Next Implementation Milestones
1. Standard Camera JSON Presets: Store 12 reusable camera movement definitions (dolly_in.json, orbit_15.json, lateral_scan.json, etc.) under video_staging/cameras/.
2. Audio Sidechain Compressor: Integrate the FFmpeg sidechaincompress filter into compile_video.py for automated background music ducking.
3. Comfy API Jinja Template: Build Jinja2 template wrappers around workflow_api.json ready to fire once Comfy Cloud credentials activate.
4. WhisperX Service Container: Deploy WhisperX forced alignment on Local Inference Node to feed word timestamps directly into the kinetic caption renderer.