In this blog post, I provide an overview of how Generative AI (GenAI) technology is impacting different areas of game development. By no means, this blog post is meant to cover every work on GenAI in gaming. I will point to some solutions that I am familiar with but there are many more out there.
Somehow, I have created a kind of taxonomy to differentiate the different areas in game development where GenAI is impacting:
I will also mention how GenAI is being used to manage game communities, probably something that is not well known. I will also briefly mention some of the latest works in GenAI that potentially can move from the academy to game studios.
The introduction of Machine Learning and Generative AI in gaming has sparked an authentic revolution in graphics and gaming, comparable to the launch of the first programmable GPU GForce2 by NVIDIA in 2000. Just as GPUs unlocked real-time rendering and complex physics in gaming, GenAI is now enabling automation, efficiency, high quality graphics and unprecedented creative possibilities.
I was lucky to attend SIGGRAPH 2018 when NVIDIA presented DLSS and neural ray-denoising for the first time. Also, I was in the Augmented World Expo when Adobe presented their ML-based solution for animation, showcasing the animation of a wolf running in the forest. We all know how difficult is to animate characters, and animating quadrupeds is even more challenging. That was just an anticipation of all that was about to come in gaming and graphics.
In general, what we can see already today from the use of GenAI in gaming is:
GenAI can help us from the very beginning when we are brainstorming for game ideas, by suggesting new features and game mechanics based on latest trends and player preferences.
GenAI can also provide great help in creating the game story and dialogs, and storyboards, by developing storylines, branching the narratives and creating characters and interactions between them.
The most amazing thing here is that any of us can make use of a LLM to start designing a game as detailed as we want.
We can start by brainstorming with ChatGPT our vision, the type of game we would like to develop, for example, a fantasy adventure game, and then refine the initial idea by exploring different worlds and themes.
Figure 4. Pipeline showcasing main steps of brainstorming with a LLM on designing a game.
Next, we can ask for help to develop the story and the characters: “Can you help me outline a storyline for a game where the hero unravels ancient mysteries? “. Get suggestions for protagonists and antagonists and supporting characters, including their motivations and backgrounds.
We can then move to the gameplay mechanics: “Can you suggest engaging mechanics I can include in an adventure game?”. We brainstorm with the AI about ideas for levels.
At this point we can create a Game Design Document and move to the technical implementation and prototyping, get coding help by asking for sample code for core features and enquire about game engines that suit our project.
Going through a continuous cycle of iteration and refinement, we can save a lot of time and speed up the design process.
Looking at GenAI in game development, it is perhaps where we see more intensive use of GenAI today, during the different stages of the game development pipeline.
GenAI is already helping in automated content creation in games engines. GenAI can generate game assets like levels, maps, characters, and environments based on predefined rules and parameters. This allows designers to quickly generate unique and diverse content without having to create each element manually. For example, AI might generate a new level layout for a platformer game or even create a procedurally generated world in an open-world game.
GenAI is also contributing to generating textures, 3D models, and animations that fit the aesthetic of a game, saving time for artists and designers.
In coding AI is providing great help in programming and scripting. From code completion, error detection and code optimization to generation of full scripts for game engines.
Testing is a crucial part in game development. AI helps in automating playtesting, bug detection, performance monitoring, and user behavior simulations to streamline Quality Assurance processes.
All major game engines are incorporating AI in different stages of development pipeline.
Different game engines are intensively incorporating GenAI in their game development pipeline. It includes coding, animation, texturing, troubleshooting problems, asset creation like 3D objects, music, and more.
In the Unity Project Muse, we can mention:
Roblox:
Unreal integrates AI through different plugins:
There are other independent AI animation tools which are impressive, capable of extracting animation information from any video clip, allowing us to easily build an animation library for our games (see Kinetix and DeepMotion). Others, like SayMotion, generate animations from a text prompt.
Here we can mention several well-known ML-based solutions to improve graphics rendering like NVIDIA DLSS (Deep Learning Super-Sampling) and NVIDIA Optix Ray Denoising technology and Frame Rate Upscaling. AMD FSR 4 (Fidelity Super Resolution) powered by AI has been recently released.
Regarding other applications different from graphics enhancement, the use of GenAI is just starting. We can mention AI-Dungeon’s scripts for crafting custom adventures. Inworld have also developed solutions for AI-driven NPCs.
AI can transform static, predictable character behaviour into dynamic, learning entities. From sophisticated decision-making and adaptive strategies to realistic navigation and engaging dialogue, these technologies create a more immersive and challenging gaming experience. As AI continues to evolve, we can expect even more lifelike and responsive NPCs that push the boundaries of interactive storytelling and gameplay design.
I would like also to mention in this section the latest Arm technology KleidiAI to enable the inference of LLMs with great performance on mobile devices powered by Arm CPUs. When integrated within the game engine, this allows to use LLMs at runtime for different use-cases. For example, verbal interaction, game assistance, runtime strategy, and more.
KleidiAI is an open-source library that provides highly optimized and performant functions, known as micro-kernels, for AI workloads tailored for Arm CPUs.
The application of AI in game communities is not well-known. Game communities are vital for game developers and players. So, GenAI has come also to improve and optimize the management of game communities and improve safety and content moderation.
An example is Roblox’s AI moderation tool Advance Group Wall Filter, that helps detect harmful content and facilitate community interactions.
Their AI-Powered Real-Time Chat Translation tool helps overcoming language barriers between users of different nationalities.
Discord use AI bots to optimize player engagement.
AI also assists in connecting creators with developers for asset exchanges, optimizing this way asset marketplace in game communities.
Looking at recent developments in GenAI, there are many works with potential impact on gaming. From my personal experience attending game and graphics related conferences I have witnessed a substantial increment of papers on the use of GenAI for 3D assets generation, avatars, high quality faces, environments, including realistic cities, editing of generated images and geometry, video overlying, and many more topics relevant for game development.
Things like hair and cloth simulation which are very difficult to render, are getting better. GenAI is impacting animation and lighting, which require great experience to get correctly.
We see also emerging new rendering paradigms like Gaussian Splatting and Neural Radiance Fields finding their ways within game developers, avid for new rendering technologies.
I would like to highlight here a kind of emerging trend. The time from papers to practical tools for game development is shortening. We can see how after short time, original papers presented in conferences see their implementations shared in Hugging Face or in the authors’ code repository. Everybody is hungry to test new models and tolls.
Finally, I would like to share a few thoughts as conclusions which are summarized in the picture below.First, the fact that AI has triggered an authentic revolution in gaming, enabling automation of repetitive tasks like animation, texturing, 3D modeling. This is speeding up developments and helping to reduce costs.
Second, the use of GenAI opens many new opportunities in gaming at runtime, more realistic and smart characters, infinite worlds capable of auto-evolving, new type of interaction with NPCs based on speech, and many other new features that can be implemented at runtime, where the real fun is. Ultimately, it is the player and the gaming experience, all game development is for.
Finally, I would like to highlight the fact that we see the time from papers to implementation shortening each year. So, it is very difficult to make predictions, but I do not see far the moment when we will be able to generate whole complex scenes with all details, 3D assets, characters, animations, visual effects, lighting, everything that usually takes weeks and months of work, all just from a prompt description.
It will be interesting to see the use of new reasoning models in gaming, where the strategy is important. Also, AI can be very useful in the Metaverse for building avatars and virtual environments.
You can get mor detailed information on Arm KleidiAI technology here. There is also a complete section of Learning Paths devoted to AI here, that provides practical examples on how to use KleidiAI to accelerate GenAI workload and run LLMs on different hardware backends.