GSoC 2026 Project Report - OpenVINO GenAI: Add Image-to-Video Support to LTX Video Generation Pipeline
| Contributor | Aditya Goyal (@goyaladitya05) |
|---|---|
| Organization | OpenVINO Toolkit |
| Mentors | Anna Likholat (@likholat), Stanislav Gonorovskii (@sgonorov) |
| Project size | Large |
| Duration | 12 weeks |
| Project page | summerofcode.withgoogle.com/programs/2026/projects/unPORmcz |
| Progress board | github.com/users/goyaladitya05/projects/7 |
| Medium article | Bringing Image-to-Video Generation to OpenVINO GenAI with LTX-Video |
Table of contents
- The goal
- Background
- Layer 1 - [OpenVINO] Add Image-to-Video support for LTX-Video
- Compatibility - [Video Generation] Support rank-1 and rank-2 timestep inputs in LTX-Video pipeline
- Layer 2 - [Video Generation] Implement Image2Video Pipeline
- Layer 3 - [llm_bench][wwb] Add Image2Video pipeline support
- Using the pipeline
- Challenges and what I learned
- Work beyond the project scope
- Conclusion
- Appendix: complete list of contributions
- Acknowledgements
The goal
OpenVINO GenAI could generate video from a text prompt through the LTX-Video pipeline, but there was no way to condition that generation on an image. This project added Image-to-Video (I2V) generation - producing a video from an input image plus a text prompt - running on Intel CPU and GPU, with the HuggingFace Diffusers LTXImageToVideoPipeline as the numerical reference.
The mechanism is simple to state: VAE-encode the conditioning image once, then pin the resulting latent as frame 0 along the frame axis of the packed latent tensor for the entire denoising loop. Everything downstream of that sentence is where the work was.
Background
OpenVINO is an open-source software toolkit for optimizing and deploying deep learning models. It supports models from popular frameworks and runs inference across CPU, GPU, and NPU.
Optimum Intel is the interface between the Hugging Face Transformers and Diffusers libraries and Intel’s optimization tools. Its OpenVINO integration can optimize models, convert them to OpenVINO IR, and run inference with OpenVINO Runtime.
OpenVINO GenAI is a library of generative AI model pipelines, optimized execution methods, and samples that run on OpenVINO Runtime. It provides lightweight C++ and Python APIs and includes the core functionality needed to run generative models.
Model export
Converts a Hugging Face model to OpenVINO IR.
Pipeline API
Loads the exported model through C++ or Python.
Runtime execution
Runs the graph on Intel CPU or GPU.
LTX-Video is the model: a transformer-based latent diffusion model for video from Lightricks (arXiv:2501.00103), designed to work in a heavily compressed latent space, and supporting both text-to-video and image-to-video generation.
The image-to-video path end to end: the conditioning image is encoded to a latent, combined with noise and text conditioning, denoised by the diffusion transformer, and decoded back into video frames. Source: Official LTX-Video paper (arXiv:2501.00103).
Two tools from the GenAI repository appear throughout this report:
WhoWhatBenchmark (WWB) is an output-similarity harness. It runs the same prompts through a reference implementation and through the implementation under test, then scores how similar the outputs are.
llm_bench is the corresponding performance harness, measuring load time, generation latency, and per-component timings across GenAI pipelines.
| Layer | Pull request | Upstream |
|---|---|---|
| Export / model | [OpenVINO] Add Image-to-Video support for LTX-Video huggingface/optimum-intel#1762 |
Merged (Jul 20, 2026) |
| Compatibility | [Video Generation] Support rank-1 and rank-2 timestep inputs in LTX-Video pipeline openvinotoolkit/openvino.genai#4298 |
Merged (Aug 20, 2026) |
| Dependencies / CI | Update optimum-intel pin to unblock diffusers 0.39.0, drop Flux2Klein workaround openvinotoolkit/openvino.genai#4280 |
Merged (Aug 25, 2026) |
| Runtime pipeline | [Video Generation] Implement Image2Video Pipeline openvinotoolkit/openvino.genai#3907 |
Merged (Sep 4, 2026) |
| Benchmarking / validation | [llm_bench][wwb] Add Image2Video pipeline support openvinotoolkit/openvino.genai#4111 |
Merged (Sep 10, 2026) |
Every planned deliverable is implemented and validated. All five in-scope pull requests are merged.
Layer 1 - [OpenVINO] Add Image-to-Video support for LTX-Video
huggingface/optimum-intel · PR #1762
Adds export and inference support for LTX-Video image-to-video in optimum-intel. It registers the task, implements OVLTXImageToVideoPipeline, and exports the VAE encoder needed to convert the input image into latents. This is a complete Optimum Intel feature in its own right, independent of OpenVINO GenAI, and the foundation for the GenAI pipeline.
The key export change expands timestep from [B] to [B, S]. Frame-0 conditioning tokens use t = 0, while generated tokens use the current denoising timestep.
Frame-0 tokens carry the clean timestep t_c; generated tokens carry the current timestep t. Diagram redrawn from the LTX-Video paper (arXiv:2501.00103).
Older rank-1 exports remain supported by broadcasting only when the compiled model expects rank 2. Image-to-video scored 0.992 to 0.994 across pairwise comparisons; text-to-video remained at 0.986 for both Hugging Face comparisons and 0.999 for Optimum versus GenAI.
Compatibility - [Video Generation] Support rank-1 and rank-2 timestep inputs in LTX-Video pipeline
openvinotoolkit/openvino.genai · PR #4298
After #1762, new LTX-Video exports used a rank-2 timestep, while GenAI’s existing text-to-video path still allocated [1]. This made newly exported models incompatible with the shipped pipeline.
The runtime now follows the compiled model’s input shape: [1] for older exports and [B, S] for current exports. The rank-2 tensor is sized from the classifier-free-guidance-adjusted latent shape, keeping its batch and token dimensions aligned. The fix was split from #3907 so text-to-video compatibility did not depend on the larger image-to-video PR.
Layer 2 - [Video Generation] Implement Image2Video Pipeline
openvinotoolkit/openvino.genai · PR #3907
Adds the C++ and Python Image2VideoPipeline and the VAE encoder it uses. The pipeline encodes one conditioning frame, assigns per-token timesteps, re-pins the frame-0 latent after every denoising step, and exposes a validated strength setting.
The PR also adds bindings and type stubs, compatibility with rank-1 and rank-2 exports, and fixed-seed and strength tests. It contains all project documentation changes and adds working C++ and Python samples. The result is image-conditioned video generation on Intel CPU and GPU.
Layer 3 - [llm_bench][wwb] Add Image2Video pipeline support
openvinotoolkit/openvino.genai · PR #4111
Adds image-to-video support to llm_bench and WhoWhatBenchmark. llm_bench accepts conditioning images through --media and reports their batch size; WWB runs the same inputs through Diffusers, Optimum, and GenAI for pairwise comparison.
Results
WWB similarity, 10 samples, CPU, fp32:
Image-to-Video
| Comparison | Similarity |
|---|---|
| HF Diffusers vs Optimum | 0.994 |
| HF Diffusers vs GenAI | 0.992 |
| Optimum vs GenAI | 0.994 |
Text-to-Video - regression check, identical on master and with these changes
| Comparison | Similarity |
|---|---|
| HF Diffusers vs Optimum | 0.986 |
| HF Diffusers vs GenAI | 0.986 |
| Optimum vs GenAI | 0.999 |
Using the pipeline
Export the model with the new task. Image-to-video requires a VAE encoder alongside the standard text-to-video components, so the exported directory now contains a vae_encoder subdirectory:
optimum-cli export openvino \
--model Lightricks/LTX-Video \
--task image-to-video \
--weight-format fp32 \
ltx_video_ov_i2v/FP32
Then generate, in Python:
import numpy as np
import openvino as ov
import openvino_genai as ov_genai
from PIL import Image
from video_utils import save_video
def read_image(path: str) -> ov.Tensor:
image = Image.open(path).convert("RGB")
return ov.Tensor(np.array(image)) # [H, W, 3], uint8
pipe = ov_genai.Image2VideoPipeline("./ltx_video_ov_i2v/FP32", "GPU")
output = pipe.generate(
read_image("./input.png"),
"A golden retriever in a sunlit bedroom slowly stands up, stretches its front "
"paws forward, yawns wide, then turns its head to look at the camera, warm "
"afternoon window light, cinematic shallow depth of field, photorealistic",
negative_prompt="static, motionless, frozen, blurry, distorted, low quality",
width=704,
height=480,
num_frames=161,
num_inference_steps=50,
guidance_scale=4.0,
frame_rate=25,
generator=ov_genai.CppStdGenerator(42),
)
save_video("genai_image2video.avi", output.video, 25)
The result is a video tensor of shape [num_videos_per_prompt, num_frames, height, width, 3]. Switching between Intel CPU and GPU is a one-word change to the device argument, so application code stays portable.
Equivalent samples ship in the repository for both languages, at samples/python/video_generation/ and samples/cpp/video_generation/. The Python sample also reports load time, generation duration, transformer inference duration, and VAE decoder duration.
Sample
Optimum Intel reference scripts: image-to-video, text-to-video.
A full walkthrough of the feature - setup, parameters, and worked examples - is available on Medium: Bringing Image-to-Video Generation to OpenVINO GenAI with LTX-Video.
Challenges and what I learned
Testing against unreleased exporter changes
Both #3907 and #4111 depended on optimum-intel#1762 before its exporter changes were available in a release. Using optimum-intel main across GenAI was not a simple pin update: openvino.genai#4280 also had to resolve a safetensors conflict with Diffusers 0.39.0, raise the Transformers 4.x test matrix to 4.57.6, and handle the resulting CI failures. Without that repository-wide update, #4111 was blocked. For #3907, I kept the new API tested by installing optimum-intel main only for the affected export cases and isolating those models from the normal pinned cache. The newer exporter also exposed the rank-2 timestep incompatibility fixed by #4298, which #4280 depends on.
A workaround that hid the real bug
My first image-to-video outputs on CPU were grey and badly degraded. bf16 is the default inference precision on bf16-capable hardware, and I was developing on Sapphire Rapids. I worked around it by forcing INFERENCE_PRECISION_HINT: "f32" for the transformer, wrote a comment about cross-attention saturating, and moved on.
The original LTX-Video implementation does not require fp32. PyTorch in bf16 was fine on the same CPU, Intel GPU in fp16 was fine, and - the decisive observation - text-to-video on GenAI master was also broken in bf16. The bug predated my work and was in how the OpenVINO CPU plugin handled bf16 for this model. The precision hint would have hidden that bug behind a permanent performance penalty.
I removed the hint, regenerated samples on GPU in fp16, filed openvino#36715, and pursued the actual root causes separately so they would not sit inside this project’s scope. That work is summarized in the next section.
Work beyond the project scope
The contributions in this section fall outside the agreed GSoC project scope. They were not part of the accepted proposal, were not proposed as extensions by my mentors, and are not required for the Image-to-Video work described above. I pursued the OpenVINO core work independently after the bf16 investigation exposed issues, while the LTX2 integration was a separate independent contribution. They are included only as additional upstream work completed during the program. Any remaining open item in this section should not be treated as incomplete GSoC work.
The short version: LTX-Video’s grey bf16 output on CPU had two independent causes. Affine-free RMSNorm - used by DiT-family diffusion transformers - was never fused on CPU, so its variance math stayed decomposed and got lowered to bf16, compounding over 30 denoising steps into roughly 20% standard-deviation loss. Separately, LTX computes its 3-axis RoPE table at runtime with angles reaching about 1.6e4 radians, where bf16’s 7-bit mantissa spaces representable values further apart than 2π, scrambling the positional encoding entirely. Investigating f16 later surfaced a third, unrelated failure: the T5 text encoder overflows f16’s range in deep blocks, and the resulting Inf becomes NaN at the next normalization, cascading into an all-black video.
| Contribution | Status |
|---|---|
| openvino#36719 - Keep LTX-Video RoPE angles in f32 under bf16 | Merged, Jul 29 |
| openvino#36718 - Fuse affine-free RMSNorm to keep variance math in f32 | Merged, Aug 4 |
| openvino#37041 - Clamp f16 FC output feeding a residual Add | In review |
| openvino#37141 - Native LTX-Video RoPE kernel for CPU | Merged, Aug 7 |
| openvino#37519 - Add JIT kernel for the LTX-Video RoPE executor | Merged, Aug 28 |
| optimum-intel#1885 - Image-to-video support for LTX2 | Merged, Aug 13 |
| openvino#36715 - bf16 grey video (reported) | Resolved, Aug 4 |
| openvino#37039 - f16 NaN output (reported) | Open |
optimum-intel#1885 added LTX2 image-to-video export and inference as a standalone feature for Optimum Intel users, while providing the export foundation for future support in OpenVINO GenAI.
#37141 replaced LTX-Video’s decomposed RoPE path with a native CPU kernel that computes in f32 and rounds once to the output type. It measured 2.05x faster in f32 and 1.58x faster in bf16 than the decomposed path.
As a follow-up to #37141, #37519 adds a JIT kernel for the LTX-Video RoPE executor.
With #36718 and #36719 both in, bf16 LTX-Video is correct end-to-end on CPU at roughly 3.8x the speed of fp32.
Conclusion
The project is complete. Every deliverable proposed at the start of the program has been implemented, tested, validated against the reference implementation, and submitted upstream.
| Proposed deliverable | Outcome |
|---|---|
VAE encoder support in AutoencoderKLLTXVideo |
Delivered - encode() in C++ with Python bindings and stubs (#3907); encoder submodel export (#1762) |
generate(image, prompt, config) overload in the C++ and Python APIs |
Delivered, design revised - shipped as a dedicated Image2VideoPipeline class in C++ and Python rather than an overload on the text-to-video pipeline (#3907) |
| Timestep scheduling | Delivered - per-token [B, S] conditioning at runtime (#3907), enabled by the widened timestep input (#1762), with rank-1/rank-2 compatibility handling (#4298) |
| C++ and Python samples | Delivered - both, with image decoding handled in the C++ sample (#3907) |
WWB and llm_bench updates |
Delivered - image-to-video task in both tools, with loaders for Optimum and GenAI (#4111) |
| Test suite validating C++/Python consistency | Delivered - coverage for encode(), I2V generation, determinism and strength (#3907), plus resize and num_frames validation (#1762) |
| Running on Intel CPU and GPU | Delivered - validated on both |
Image-to-video generation for LTX-Video is available end-to-end: a user can export an LTX-Video model with optimum-intel, load it in OpenVINO GenAI from either C++ or Python, and generate a video from an image and a prompt, on CPU or GPU. Output matches HuggingFace Diffusers at 0.992–0.994 WWB similarity, and the existing text-to-video path is unchanged, with regression measurements to prove it.
All five in-scope pull requests are merged.
Beyond the proposal, the work also produced four merged fixes in OpenVINO core, one further core fix in review, a merged Optimum integration for LTX2, and three bug reports. The bf16 report in #36715 is resolved; the remaining f16 report is addressed by #37041.
Appendix: complete list of contributions
Progress tracked throughout at github.com/users/goyaladitya05/projects/7.
In scope
| Item | Repository | Opened | Status |
|---|---|---|---|
| #1762 - Add Image-to-Video support for LTX-Video | optimum-intel | Jun 3 | Merged Jul 20 |
| #4298 - Support rank-1 and rank-2 timestep inputs | openvino.genai | Aug 11 | Merged Aug 20 |
| #4280 - Update optimum-intel pin to unblock diffusers 0.39.0, drop Flux2Klein workaround | openvino.genai | Aug 6 | Merged Aug 25 |
| #3907 - Implement Image2Video pipeline | openvino.genai | May 25 | Merged Sep 4 |
| #4111 - llm_bench and WWB support for Image2Video | openvino.genai | Jul 8 | Merged Sep 10 |
Beyond scope
| Item | Repository | Opened | Status |
|---|---|---|---|
| #36719 - Keep LTX-Video RoPE angles in f32 under bf16 | openvino | Jul 4 | Merged Jul 29 |
| #36718 - Fuse affine-free RMSNorm to keep variance math in f32 | openvino | Jul 4 | Merged Aug 4 |
| #37041 - Clamp f16 FC output feeding a residual Add | openvino | Jul 23 | In review |
| #37141 - Native LTX-Video RoPE kernel for CPU | openvino | Jul 30 | Merged Aug 7 |
| #37519 - Add JIT kernel for the LTX-Video RoPE executor | openvino | Aug 19 | Merged Aug 28 |
| #1885 - Image-to-video support for LTX2 | optimum-intel | Jul 22 | Merged Aug 13 |
Issues reported
| Item | Repository | Opened | Status |
|---|---|---|---|
| #36715 - LTX-Video bf16 on CPU produces grey, low-quality video | openvino | Jul 3 | Resolved Aug 4 by #36718 and #36719 |
| #37039 - LTX-Video produces all-black NaN output in f16 on CPU | openvino | Jul 23 | Open; addressed by #37041 |
| #4169 - Text2Video WWB similarity far below reference implementations | openvino.genai | Jul 16 | Closed; fixed in #4172 |
All dates are 2026.
Acknowledgements
Thanks to my mentors Anna Likholat and Stanislav Gonorovskii for their guidance throughout the project, and to the Optimum Intel, OpenVINO GenAI, and OpenVINO reviewers for their feedback.