DEVOURED
How NVIDIA Builds Open Models for the Age of AI
NVIDIA is using a unified, hybrid Transformer-Mamba architecture to build and open-source models across language, robotics, and physical AI.
What: NVIDIA’s open model portfolio includes the Nemotron family for reasoning, Cosmos for world simulation, and Isaac GR00T for robotics. These models leverage a hybrid architecture combining Mamba’s linear-scaling state space layers with traditional Transformer attention, trained using 4-bit quantization (NVFP4) designed specifically for Blackwell-class GPUs.
Why it matters: NVIDIA is betting that by commoditizing high-performance foundation models, it can accelerate the adoption of AI applications that rely on its GPU hardware, effectively letting developers build the market pull for its compute products.
Deep dive
- Model Architecture: Uses a hybrid approach, swapping most dense attention layers with Mamba-based state space layers to keep long-context processing efficient.
- Hardware Integration: Models are pre-trained in 4-bit precision (NVFP4) specifically co-designed for Blackwell GPU hardware.
- Physical AI: Cosmos 3 acts as an omni-model for world simulation, generating scenes and predicting state transitions for robot policies.
- Standardization: Uses a unified backbone across models to minimize redundant engineering, similar to the strategy behind the CUDA software stack.
- Openness Strategy: Distinguishes itself from competitors by releasing not just model weights, but full training datasets, reinforcement learning environments, and specific recipes.
- Cultural Engineering: Operates via internal 'coalitions' rather than top-down mandates, favoring volunteer contributions to keep innovation velocity high.
Decoder
- Mamba: A type of state space model that processes sequences with linear complexity relative to input length, offering an alternative to the quadratic cost of Transformers.
- NVFP4: A 4-bit numerical format used by NVIDIA for neural network weights and activations to reduce memory footprint and increase throughput on specific hardware.
- State Space Model (SSM): An architecture that represents sequences as continuous signals, often used for efficient long-range dependency modeling.
- Mixture-of-Experts (MoE): A technique where only a small subset of the model's parameters (experts) are activated for each input token, allowing for higher total capacity with lower per-token compute costs.
Original article
Full article content is not available for inline reading.
Explore all 42 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi