Back to Devoured - May 27, 2026
DEVOURED
Native Multimodal Models (GitHub Repo)

Native Multimodal Models (GitHub Repo)

AI llmresearchopensource GitHub
A new GitHub repository maps the transition in AI from modular multimodal systems to "Native Multimodal Modeling" (NMM), where different modalities are deeply integrated within unified transformer architectures.
What: The "Awesome Native Multimodal Modeling (NMM)" GitHub repo, accompanied by a roadmap paper, categorizes models and research moving beyond late-fusion multimodal approaches towards intrinsically integrated systems with unified backbones, classifying them into Multi-to-Text (M2T), Multi-to-Target (M2G), and Multi-to-Multi (M2M) paradigms.
Why it matters: This resource provides a critical framework for understanding the next generation of AI foundation models, highlighting the shift towards "born-native" multimodal architectures that process diverse sensory inputs more coherently and symmetrically, which is essential for developing more advanced and integrated AI capabilities.
Takeaway: Developers and researchers interested in the future of multimodal AI should explore this GitHub repository and its associated roadmap paper for an overview of current models and architectural trends.
Deep dive
  • The "Awesome Native Multimodal Modeling (NMM)" GitHub repository and its companion paper track the evolution of multimodal AI models.
  • It focuses on the structural transition from "Modular Assembly" (late-fusion/grafted compositions) to "Native Multimodal Modeling," where multiple modalities are intrinsically integrated.
  • NMMs use unified transformer spaces or joint backbones, moving beyond systems blind to raw sensory signals.
  • The repository introduces an NMM Architectural Taxonomy based on Integration Depth (mid-fusion vs. early-fusion) and Functional Input–Output Duality.
  • M2T (Multi-to-Text): Models that ground cross-modal inputs (e.g., images, audio) into purely linguistic responses for reasoning (e.g., LLaVA, DeepSeek-VL, Qwen-Image, GLM-4.5V).
  • M2G (Multi-to-Target): Models that directly synthesize modality-specific outputs (e.g., video, speech) through native representations (e.g., Wan 2.2-T2V-A14B, HunyuanVideo, OmniVoice).
  • M2M (Multi-to-Multi): A unified paradigm where understanding and generation coexist symmetrically within a single network, treating them equivalently within a unified network.
  • The roadmap also covers technical dimensions like architectural patterns, data curricula, training strategies, inference/deployment, and evaluation benchmarks.
  • It serves as a curated reading list and model zoo for researchers and developers in the field.
Decoder
  • Multimodal Modeling: AI models that can process and understand information from multiple modalities, such as text, images, audio, and video.
  • Modular Assembly (Late-Fusion/Grafted Compositions): An older approach where different unimodal models (e.g., separate vision encoder, text encoder) are trained and then combined, often through shallow projectors, typically suffering from a lack of intrinsic integration between modalities.
  • Native Multimodal Modeling (NMM): An approach where multiple modalities are intrinsically integrated into a unified transformer space or joint backbone, processing raw sensory signals together from earlier stages.
  • Transformer: A neural network architecture, popularized for natural language processing, that uses self-attention mechanisms to weigh the importance of different parts of the input data.
  • Mid-fusion: A type of multimodal integration where features from different modalities are combined at an intermediate stage of the model, after some initial unimodal processing.
  • Early-fusion: A type of multimodal integration where raw or lightly processed data from different modalities are combined at the very beginning of the model's processing pipeline.
  • M2T (Multi-to-Text): Multimodal models where the input can be various modalities (e.g., image, text) but the primary output is text (e.g., visual question answering).
  • M2G (Multi-to-Target): Multimodal models where the input can be various modalities and the output is a specific target modality other than text (e.g., text-to-video, audio-to-image).
  • M2M (Multi-to-Multi): Multimodal models designed for symmetric understanding and generation across multiple modalities, treating them equivalently within a unified network.
Original article

Awesome Native Multimodal Modeling (NMM)

A curated reading list & model zoo for the era of Born-Native multimodal foundation models.

📄 Companion paper: "Toward Native Multimodal Modeling: A Roadmap"

This repository systematically tracks the structural transition from Modular Assembly — late-fusion / grafted compositions that suffer from a fundamental blindness to raw sensory signals — to Native Multimodal Modeling (NMM), where multiple modalities are intrinsically integrated into a unified transformer space or joint backbone.

⭐ Star this repo to track the latest landmark works. PRs are warmly welcomed for any model we may have missed.

🗺️ The NMM Architectural Taxonomy

We formalize the NMM ecosystem through a dual-dimensional lens based on Integration Depth (mid-fusion vs. early-fusion) and Functional Input–Output Duality:

# Paradigm Input → Output Core Idea
🟦 M2T — Multi-to-Text multimodal → text Ground cross-modal inputs into purely linguistic responses for reasoning.
🟩 M2G — Multi-to-Target multimodal → modality-specific Direct synthesis of modality-specific outputs through native representations to achieve temporal & acoustic coherence.
🟪 M2M — Multi-to-Multi multimodal → multimodal A unified paradigm where understanding and generation naturally coexist as reciprocal projections within a single network.

🟦 1. Multi-to-Text (M2T) Unimodal Generation

Native scaling frameworks that ground cross-modal inputs into linguistic streams for logical reasoning.

🧱 Late-Fusion Baseline References

Modularly assembled via shallow projectors; blind to raw sensory signals.

🔗 Mid-Fusion (Naturally Interacted Regime)

Foundational pioneers maintaining explicit, modality-aware boundaries.

Massive state-of-the-art evolved mid-fusion architectures:

Scale-driven industrial mid-fusion implementations:

🧬 Dense / Native M2T Scaling

🟩 2. Multi-to-Target (M2G) Scenario-based Generation

Bypassing traditional post-hoc decoders to synthesize photorealistic spatiotemporal physics or continuous speech directly.

🎬 Advanced Video / World Simulators

🎙️ Speech-Centric Native Frameworks

📅 Timeline Milestone Generators

🟪 3. Multi-to-Multi (M2M) Symmetric Modeling

Omni-directional unified spaces establishing a symmetric paradigm where comprehension and generation natively coexist.

🔥 Early-Fusion (Native Convergent Regime)

Born-native designs treating all modalities equivalently via one unified backbone & embedding space.

🔮 Early Unified Predictors

  • Moshi ★ [Défossez et al., 2024] — 💻 GitHub · 📄 Paper — Real-time conversational audio-text dual-stream processing.
  • Emu3 / Emu3.5 ★ [BAAI, 2024–2025] — 🌐 Project Page · 📄 Paper — Next-token sequence prediction unifying understanding and synthesis.

🧩 Interleaved Sequence Modeling

🌌 Bidirectional Unification Frontiers

Collapsing representation boundaries.

★ Denotes early exploratory or foundational dual-regime architectures.

🛠️ The Technical Roadmap Dimensions

Following the systemic structure detailed across Sections §3–§7 of the roadmap paper, the core components of the NMM lifecycle are curated below.

🧩 1. Architecture · §3

  • Integration Depth Mapping — Structural mechanics of joint multimodal backbones vs. single unified transformer spaces.

  • Input–Output Decoupling — Eliminating modality-aware boundaries & shallow projectors.

📊 2. Data Curriculum · §4

  • Interleaved Data Curricula — Pre-training token mixtures combining web-scale text, audio waves, & video streams.

  • Post-Training Engineering — Multi-modal instruction tuning & alignment token datasets.

🎯 3. Training Strategies · §5

  • Multi-Objective Loss Recipes — Unifying continuous-discrete objectives (autoregressive next-token prediction + diffusion steps).

  • Scaling Dynamics — Computed token-allocation strategies to maintain gradient stability at the 1T+ MoE frontier.

⚡ 4. Inference & Deployment · §6

  • Full-Duplex Orchestration — Dynamic KV-cache eviction & multi-scale attention patterns for real-time interaction (<100 ms).

  • Hardware-Native Compilation — Distributed CUDA compute kernels for unified cross-modal token routing.

🧪 5. Evaluation Benchmarks · §7

  • Symmetric Evaluation Matrices — Benchmarking systems capable of examining interleaved multi-modal sequences without suffering from target-modality collapse.

🤝 Contributing

Contributions are very welcome! If a notable native multimodal model is missing or you find an outdated link, please open an Issue or send a Pull Request.

The preferred entry format is:

- <Model Name> [<Authors / Team>, <Year>] — [`💻 GitHub`](https://...) · [`📄 Paper`](https://...)

✍️ Citation

If our formalization, taxonomy, or roadmap framework assists your research, please cite our definitive paper:

@article{TencentYoutuLab2026toward,
  title   = {Toward Native Multimodal Modeling: A Roadmap},
  author  = {Siyu An and Junru Lu and Junnan Dong and others},
  journal = {arXiv preprint},
  year    = {2026}
}
Explore all 49 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi