Did you ever wonder what is happening inside your phone, what the different pieces and components are, how they interact together to make this lump of sand and plastic become the true miracle of technology that is your smartphone?
Pack your things and get ready to depart to our space station, where you will be miniaturized to the size of a processor chip, and teleported inside a smartphone.
If you were at GDC this year you might have experienced this demo, experienced the thrill of being several millimetres tall. In case you missed it, there will be many more opportunities to try Circuit VR by yourself. Alternatively, see the trailer below:
This article is part of a series detailing the making of and the technologies of Circuit VR, so do not hesitate to check them as well: Circuit VR: Implementing Foveated Rendering on GearVR with Mobile multiview and Eye Tracking in Unreal Engine 4.
I will walk you through the decisions and choices we made during the development. How we went from few ideas to the stunning VR demo.
Ideas were not in short supply at the beginning of the project; crazy designs were discussed, but no clear path was to be seen. After countless hours discussing ideas, we set ourselves a limit - one week to prototype and come back with a proposal.
Figure 1: Initial prototype
As you may have understood if you read our artist’s blog, we followed a very iterative process. This allowed maximum flexibility in our design decisions, allowing us to quickly try and test new concepts. We would start with a prototype of the environment, made with simple shapes in order to test the idea. We would then start white-boxing the level with cubes. In layman’s terms, that simply means putting boxes where the walls and objects would be to start visualising volumes in the scene. Once happy with the overall atmosphere, our artist would start producing some first versions of the assets while the engineers would work on getting the interactions and effects in code. We would then iterate and refine the assets and the interactions in a systematic manner.
We were convinced that VR was all about exploration hence keen to let the player discover the environment by himself. In Ice Cave VR, we chose a free-fly approach where, as long as the player is holding a button he will fly in the direction he is pointing at. Tweaking the speed and removing the acceleration made this option possible without making the player feeling dizzy. This method worked well, and was very successful every time we showed the demo. But Ice Cave VR was a regular 3D demo that was later switched to VR. Because of this, we questioned this assumption of was it the best “VR way”?
We tried multiple approaches, such as fixed teleport points, free fly, pre-recorded trajectories, to finally settling on a point to teleport system.
Figure 2: Point to teleport system
When the user is holding the touchpad, an indicator shows the landing position. As soon as he releases the touchpad he will be instantly teleported there. This is a method commonly seen on desktop but not that much on mobile where the experiences tends to be static or more guided.
By choosing such a movement system, we were able to optimize the collisions in our world, in a way only the ground planes (in blue in following illustration) were visible inside the ECC_WorldStaticChannel. To further refine the navmesh, we also set some nav mesh modifier (pink in the following illustration).
Figure 3: Collision volumes
When the user is pressing the touchpad, we fire rays against the ECC_WorldStaticChannel to know where he is pointing, we then project the resulting collision on the navmesh to query if movement is possible.
Figure 4: Navmesh
In code, it would look like this:
// PreTrace against the World Static bool traceHit = GetWorld()->LineTraceSingleByChannel( HitResult, traceStart, traceEnd, ECC_WorldStatic, TraceParams ); // If we hit something, try to project on the navmesh if ( traceHit ) { FNavLocation NavLoc; // Project the trace collision onto the navmesh auto navSystem = GetWorld()->GetNavigationSystem(); if ( navSystem ) { bool navMeshHit = navSystem->ProjectPointToNavigation( HitResult.Location, NavLoc );
One thing you may notice is that all the implementation details will be given in code. As this project was mainly developed by engineers, we made the choice to go for a code only project as it was easier for us to integrate with our code review system as well as our versioning system. Although the example is given in code, this should not stop you from trying to implement them in blueprint, as Epic did an incredible job at making both systems extremely similar in performance and syntax.
Development was now going in the right direction. We had the ability to move around, and we even started to have some levels to move around. But we were missing an essential component, the ability to interact.
We wanted our interaction system to be as versatile as possible, so that we could connect elements together. Acting on one would trigger actions somewhere else. Support for different interaction events, like hover and click, was also required.
At the heart of our interaction system is a line trace mechanism, which at a fixed interval would query a custom collision layer. Querying a custom layer allows us to check collision against a limited number of assets in our scene.
// Trace bool traceHit = GetWorld()->LineTraceSingleByChannel( HitResult, traceStart, traceEnd, TracerCollisionChannel, TraceParams ); // If we hit something, process the hit if( traceHit ) { // Try to cast to an hoverable AActor* hitActor = HitResult.Actor.Get(); UHoverable* hitHoverable = Cast<UHoverable>( hitActor->GetComponentByClass(UHoverable::StaticClass()) );
Once we get the hit, we process it and try to determine if the item was indeed something we can interact with. As you can see this is extremely similar to the movement raycast. We chose to keep them separate as it allowed us to change the frequency thus reducing the number of item that was probed at any given time.
If the cast to Hoverable succeeded, which should be as only Hoverables were allowed in this layer, we start calling the interface functions:
These events are broadcasted to a set of HoverableListener objects that registered themselves in advance to the Hoverable. Note, that a HoverableListener can register to itself if it is also a Hoverable actor. So, to sum things up, Hoverable receives the interactions from a player and then broadcasts them to all the related HoverableListener. HoverableListener will then process and play the related action.
Hoverable.h DECLARE_MULTICAST_DELEGATE( FOnHoverEnter ); class VRCIRCUIT_API UHoverable : public UActorComponent { GENERATED_BODY() protected: //Delegates used to call the listeners of the hoverable events. FOnHoverEnter OnHoverEnter; } Hoverable.cpp void UHoverable::onHoverEnter_Implementation() { IsHovered = true; if ( CallerCanInteract() ) { OnHoverEnter.Broadcast(); } }
In the above example, we also added a way to block interaction, in case we want to only allow them after the player performed a specific interaction.
Figure 4: Collision volume around an interaction
Let us have a look at an example. In the first room, the player needs to interact with the screen in order to progress. After clicking on the screen, it turns on and the pedestal appears.
Figure 5: Hoverable architecture
Our screen thus contains both a HoverableListener, registered to itself, and a Hoverable attached to the object collision box. This collision box is set to collide only in our custom layer.
This collision system made our development easier as we were able to quickly add complex interaction system involving multiple assets by simply adding the needed component to the actor.
As the development moved along we quickly realized we needed to see what the player in the headset was experiencing. Not only was it a great debugging feature, allowing us to understand where the player was having trouble, but it is also a great way to engage with the crowd waiting to try the experience.
Figure 6: Circuit VR at GDC
The easiest way to do this would be to stream the content directly to the screen, a bit like online streaming works. A quick back-of-the-envelope calculation, without compression, brings us to about 188Mb/s for 2048x1024 which would look terrible on a big TV. Although it is possible to stream this over Wi-Fi, this is far from optimal. We could look at optimizing this by compressing the data, but that would add more computation to our GPU, already busy delivering frames in time.
We turned the problem the other way by asking ourselves the question: what would we really need to mirror? Not much is the answer. Only some data are relevant change based on the player inputs. These are, the head rotation, the player location, and potential interaction. And just like that, we dropped our bandwidth requirements to few kilobytes per second.
Network replication is not an easy topic, but Unreal Engine 4 integrates all these things out of the box. Our first step was to set up an actor for the player that included all the elements we wanted to replicate, and flagging them as replicated.
UFUNCTION( NetMulticast, Reliable ) void ActionPressed(); void ActionPressed_Implementation(); UPROPERTY( Replicated ) UCameraComponent *VRCamera;
From there we just had to attach the second player (mirrored device) camera to the first player camera to see what the person in the headset was seeing. We also replicated interactions like click events in order to trigger them as well on the mirrored device.
This last point is one of the drawbacks of this system. As we did not fully replicate any actor in the scene, we had cases of failed synchronization, where if the mirroring player joined after the mirrored player made an interaction, this interaction would not be replicated. This problem was only visual as movement was always replicated.
As we were moving forward in the development, adding more and more details to our scene, we quickly hit a content limit. The more objects we were adding, the slower our application was running. If this effect was barely noticeable in the beginning it became unbearable as soon as we dropped below 30 FPS.
The root of this problem is the number of draw calls we emit. In VR, each and every draw call is doubled. Every item needs to be rendered twice, once for each view. Epic in their official guidelines, advice creators to keep the draw call count around 50 in total, at that time in the project we were running at more than 200 per eyes. If you are wondering how many you are sending, you can get this information by running stat scenerendering in the console. Bear in mind that the numbers are for a single view only.
Figure 7: stat scenerendering
The first thing we did was to activate the multiview extension, which you can learn more about it in our dedicated blog or by reading Daniele’s technical overview of Circuit VR. But multiview didn’t quite solve all of our problems; we were still emitting too many draw calls. As a rule of thumb, you can think that using multiview you can double the number of draw call you can emit.
Figure 8: Single environment scene
To address that issue, we chose to rethink our environment and split the big room into multiple smaller islands. That way we were able to create a much more diverse and exciting experience. This also served our movement strategy by encouraging players to explore, giving them perspectives on new places that they might want to explore next.
Figure 9: Multiple environment split
This decision also allowed us to integrate a powerful level of detail system. In order to save draw calls, each island is streamed, and switches between two different levels of details as the player comes closer to it. At regular intervals, the level streamer actor will query the player location and its distance to the different islands. Based on that distance it will decide whether to load the LOD asset or the proper level. While traditional LOD system switch between meshes only, our system has the ability to stream levels and mesh to give more freedom to our artist as to what to keep or what to remove in the geometry.
Figure 10: LOD area of effect
Changing the shape and disposition of our levels was a neat way to reduce the number of draw call we were submitting yet, we quickly realized that it was only a temporary fix. As we were progressively filling these islands with details, the draw call limit creeped back in and lead us back to the low FPS territory. This time we decided to tackle the content itself.
Instead of removing details and items on our boards we decided to merge them using the merge actor tool in UE4. This tool allows you to combine multiple meshes into a unique mesh. The neat thing is that if they are sharing the same material, all these meshes will be sent as a single draw call. A great example of this technique are the small resistors on the board. They do look similar but they are slightly different. Using texture atlassing and fiddling with the UV, we were able to send all of them as a single draw call.
Figure 11: Small resistors combined into one mesh
Figure 12: Final atlas texture
We didn’t stop with the resistors; many other objects on the board were also packed on the same big texture atlas. Doing this neat little trick that didn’t take much time, we were able to lower our number of draw calls to around 60 per eye.
Reflections are one of the key features of our demo. For artistic reasons, the one shipped inside Unreal didn’t quite answer all of our requirements. We managed to recreate one of our technique previously developed for Ice Cave, to create high resolution reflections based on local cubemaps. I won’t detail it here as you can learn more about it in Attilio’s blog.
At regular intervals throughout the development, we profiled our application on our target device using Streamline. That gave us an idea of where our bottleneck was. After tackling the CPU one by merging assets and adding the level LOD system, our bottleneck moved toward our GPU, and more precisely the fragment side. This usually means that the application is taking too much time to shade a pixel. We chose to address this issue by implementing foveated rendering. You can learn more about this technique on this dedicated blog or on Daniele’s blog.
After one year in development, Circuit VR was finally shown to the public for the first time at GDC and was very well received. All throughout this project, we faced challenges VR developers like you must face on a day to day basis. Internally this helped us better understand the ecosystem and pinpoint areas where improvements were desperately needed. But more than anything, it was an opportunity for us to feed our findings back to the community. Stay tuned in the following weeks and months for comprehensive tutorials and articles covering some of our discoveries and optimizations
After working on this project, I am more than ever convinced that mobile VR is the way forward. For now, graphics capabilities might be seen as limitation, but looking down at what we have in the pipeline for the future: I am confident. Features like VR support in Vulkan will be real game changers for CPU intensive applications, and this is now looking closer than ever. As for now, it’s up to us developers to think about what we are doing and if we can do it more efficiently. I was the first that needed to be convinced that high quality virtual reality content was possible on mobile. Seeing Circuit VR, I am now confident that such high quality content is not something of the future, but is happening today.
If by any chance you stumble upon our booth at an event and you see Circuit VR there, please do have a try at it. I would be delighted to hear your feedbacks on it, and have a chat about it if I am around.