Run GenAI Locally on NVIDIA Jetson | NVIDIA Jetson AI Lab

This session shows developers how to build and run their first local GenAI app on NVIDIA Jetson using open-source models such as Gemma and Qwen directly on device. We’ll teach different ways of running inference with Ollama, vLLM, and llama.cpp, including when each framework makes sense for rapid local prototyping, production-style serving, or lightweight edge deployment. Along the way, we’ll share best practices for model choice, quantization, and performance tuning.

In this stream, you will learn how to deploy open-source AI models on NVIDIA Jetson — no cloud required, from first launch to production-ready serving.

We'll cover:

• Getting models running on NVIDIA Jetson — spin up popular OSS models (open-source large language models (LLMs) like Gemma and Qwen (LLMs and VLMs) using Ollama or vLLM on Jetson hardware and verify they're working end-to-end.
• Choosing the right inference engine — understand the practical tradeoffs between Ollama for rapid local prototyping, vLLM for higher-throughput serving, and llama.cpp, so you can pick the right tool for your use case.
• NVIDIA Jetson-specific serving strategies — walk through the real decisions behind model choice, containers, and performance tuning tailored for Orin and Thor, including what works, what doesn't, and why.
• Performance fundamentals — get introduced to quantization and speculative decoding: what they are, how they work, and when to reach for them on edge hardware.
• Real-world application — see a live demo of the Reachy Mini assistant, an end-to-end example of OSS models running in real tim