In collaboration with Shutterstock and WPP, NVIDIA demonstrated how you could generate 3D objects and scenes using natural language prompts. Image courtesy of NVIDIA.
August 18, 2024
At SIGGRAPH 2024, NVIDIA CEO Jensen Huang sat down with Wired’s senior writer Lauren Goode to discuss a range of topics, including AI. During the conversation, Huang demonstrated how users could use text or verbal commands to generate 3D objects and scenes in NVIDIA Omniverse™. It was part of NVIDIA’s collaboration with stock-art merchant Shutterstock and the marketing firm WPP.
“We taught AI to speak OpenUSD [3D file format for NVIDIA Omniverse]. So the girl [the user] is speaking to Omniverse, Omniverse can generate USD, then Omniverse uses the USD prompt to find the object from its catalog, and then Generative AI uses these conditions to generate the scene. So the work that you do will be much, much better controlled,” said Huang.
On-demand content generation is now possible with 2D AI programs such as MidJourney, Dall-E, and Microsoft Image Creator. The ability to do the same in 3D is expected to radically change the way people design and model 3D objects in manufacturing, engineering, architecture, films, and games.
Afterward, Huang switched roles to interview his special guest: Meta CEO Mark Zuckerberg. “Just like every business now has an email, in the future, every business will probably have an AI agent,” Zuckerberg said. Echoing this, Huang said, “Everybody can have an AI. In our company, I want every engineer, every software developer to have an AI.”
In Meta's WhatsApp chat program, the “/imagine” prompt allows you to generate images based on text using Meta AI.
NIMs for Omniverse-Powered AI Apps
During SIGGRAPH, NVIDIA announced the release of more NIM microservices or APIs. They allow developers to create and offer applications built on NVIDIA technology components, such as Generative AI and OpenUSD. NVIDIA writes, “The [AI models for OpenUSD development] enable developers to incorporate generative AI copilots and agents into USD workflows, broadening the possibilities in 3D worlds and helping speed the adoption of USD across a new range of industrial sectors, like manufacturing, automotive and robotics.”
Robotic training made easy via NVIDIA Isaac SIM, demonstrated at SIGGRAPH 2024. Image courtesy of NVIDIA.
The new Generative AI NIM uses USD as its 3D language to create objects and scenes based on user input. Though it originated in the film industry, USD is seeing growing adoption in manufacturing and engineering. Siemens, for example, has a partnership with NVIDIA to integrate OpenUSD workflows into its Simcenter portfolio. NVIDIA cofounded the Alliance for Open USD (AOUSD) along with Pixar, Adobe, Apple, and Autodesk.
Professional workstations running NVIDIA Ada Generation GPUs can render USD scenes, and in engineering applications a lot of this rendering work is happening on the desktop. For example, in the Dell Precision 7960 tower workstation, the combination of multicore CPUs and NVIDIA Tensor and CUDA cores in the GPU is essential for processings tasks in 3D while running NVIDIA Omniverse. At SIGGRAPH, Dell presented an Omniverse USD demo on the Dell Precision 5860 tower, and Pixar Renderman running on a Dell Precision 3680 tower.
Previously NVIDIA demonstrated the ability to generate standard warehouse items using natural language prompts with its AI Room Generator Extension. The code is available from GitHub, and is powered by NVIDIA DeepSearch and GPT 4. When creating environments such as warehouses and reception areas, instead of dragging and dropping items from a library, users can use prompts such as “add common items found in a warehouse/reception area.”
According to NVIDIA, the microservices available include “AI models that can generate OpenUSD language to answer user queries, generate OpenUSD Python code, apply materials to 3D objects, and understand 3D space and physics to help accelerate digital twin development.”
NIMs for Robot Developers
At GTC 2024, Huang had discussed his vision for humanoid robots. At SIGGRAPH, Huang revealed a new step to fulfill that vision. The company is releasing microservices for robotics simulation in Isaac Lab and Isaac Sim, OSMO Robot Cloud Compute Orchestration Service, Teleoperated Data Capture Workflow, and more.
“The next wave of AI is robotics and one of the most exciting developments is humanoid robots,” said Huang. “We’re advancing the entire NVIDIA robotics stack, opening access for worldwide humanoid developers and companies to use the platforms, acceleration libraries and AI models best suited for their needs.”
NVIDIA’s Isaac SIM is a virtual robotic training environment built on Omniverse. It includes NVIDIA PhysX 5, a physics engine that makes the 3D objects’ interactions and collisions accurate. NVIDIA OSMO is a cloud-native service to orchestrate and scale complex robotics development workflows across distributed computing resources. It works for both on-premise or cloud setups.
At SIGGRAPH, NVIDIA demonstrated an AI- and Omniverse-enabled workflow on how to generate a vast amount of synthetic motion and perception data based on a small amount of original, real-world data. In the prescribed workflow, a robot trainee mimics and replicates the movements and behaviors of the human model it sees, significantly reducing the time required to train and develop robots. NVIDIA is currently offering early access to its general-purpose foundation model for humanoid robots, called NVIDIA Project GR00T (General Robot 00 Technology).
NVIDIA’s partner Dell offers various GPU-accelerated data center hardware and workstations tailormade for AI-based application development. During SIGGRAPH, Dell highlighted its joint offerings available in the Dell AI Factory with NVIDIA solution - including Dell servers and workstations integrated with NVIDIA GPUs, networking, and software. Among the lineup is the new Dell PowerEdge XE9680L with eight NVIDIA Blackwell GPUs in a smaller form-factor 4U chassis.
Dell said, “The server provides the highest possible rack-scale density for NVIDIA GPUs in an industry standard x86 rack, offering 33% more GPU density per node.1 The platform offers 20% more PCIe Gen. 5 slots and double the North/South network expansion capacity.”
More Dell Coverage
More NVIDIA Coverage
Subscribe to our FREE magazine, FREE email newsletters or both!
