- 1 How to Get Started
- 2 SDKs
- 3 Runtimes
- 4 App Stores
- 5 Wrappers and Injectors
- 6 Resources
- 7 3D models
- 8 References
How to Get Started
- See also: How to Get Started in VR Development
This page serves as a jumping-off point for new developers to find any resources they might need along their path to learning to make virtual reality experiences.
1. Learn about the hardware
Ask yourself: do you want to develop for a computer-driven headset like the Vive, or are you more interested in mobile applications such as GearVR or Google Cardboard? If you don't already own your hardware of choice, do some research and think about what would be best for both your target market and most practical to develop on. If your idea requires motion controls or high-end graphics, stick to computer-driven VR. A list of currently available hardware that is supported by Unity, Unreal, and VR web implementations:
- HTC Vive, $799 - motion controllers ship with product. iFixIt will show you what it looks like when you take it apart.
- Oculus Rift, $599 - motion controllers won't be available until fall. iFixIt has also autopsied a Rift.
- OSVR HDK2 - $399
- Razer Hydra, $599 - general-purpose wired motion tracking controller for PC
(can use a smartphone as HMD)
- Gear VR, $99
- Google Cardboard, available as cheap as free
- Google's Daydream View - $79
(can use a smartphone as HMD)
- Mozilla A-Frame is a markup language (as are HTML and XML) for making cross-platform VR software. To see it in action, visit their site on your smartphone, turn off orientation lock, and press the VR button that appears.
- Vizor is a web app that allows you to construct 3D scenes and view them across numerous platforms, including from mobile devices. Although it isn't as powerful as a game engine or open-source web platform, it is very straightforward and a great way to start creating in VR without an expensive headset. The Vizor blog has several tutorial posts.
- Responsive WebVR is a cross-platform, web-based VR platform available for modification on GitHub. You'll probably want to brush up on Three.js.
2. Learn about the software
Designing for VR is much like designing videogames, as they are both 3D interactive experiences. The difference between designing for VR and designing for traditional video games is that special considerations must be made for the nuanced experiences of presence and immersion, nonlinear storytelling, locomotion which reduces motion sickness, and graphical optimization.
Most VR developers opt to use a game engine (unless they're developing for WebVR, discussed below), and should decide which they want to work with early on. Two most popular are Unreal Engine 4 (UE4) and Unity. Both of these engines are very capable and will become the tool that you rely on most in your development; both have very active communities with vast resources out there to help you. These are freely available software suites capable of managing 3D environments, importing custom assets (such as 3D models, imagery, sounds, video), and programming interactivity or gameplay. Most useful of all, there are an insane number of YouTube tutorials and online guides for both engines, both official and fan-created.
There is no absolute agreement in the VR developer community that Unreal is a "better" engine than Unity, nor vise versa. However, they do have distinct personalities. UE4 is characterized as being more computationally optimized with greater visual fidelity but a steeper learning curve; on the contrary, Unity was designed to be powerful enough to drive commercial-quality games while remaining more intuitive and efficient for beginning designers. Unreal Engine 4 is free to download and use, but as a condition of use they will take 5% of your net revenue (above $3000) per quarter. Unity has multiple versions with different costs, but Unity Personal is free to use. If possible, it might be best to try both of these engines to see what suits you best, however it's hard to go wrong here, both offer a lot of value and are very capable.
Once you've chosen an engine or web application, the next step is to get familiar with it. Learn the basics of whatever programming language your tools use -- whether it's C++ and Blueprints Visual Scripting(UE4), C# (Unity), or a custom markup language for web applications. If you're developing for Android, download Android Studio and try deploying a basic app. For Cardboard and Unity, check out Google's SDK.
The Resources Directory has links and resources that you will find useful when trying to learn how to use an engine. It's best to follow through with some tutorials to get a feel for the engine, how to manipulate objects in space, and so on. Both Unity and Unreal offer built in VR support, so you can preview your creative works directly in VR!
3. Make or find art assets
Now that you have an engine and headset, you'll need artwork, sound, 3D models, and animations to fill out your virtual world. You may find assets online that are free for reuse, rip assets from popular games (if you don't plan to sell your project), make your own from scratch, or modify existing assets to suit your needs. Remember that virtual realities demand visuals and audio that appear "real" when examined close-up from all sides, even if they are stylized or abstract.
The next thing that's needed for VR dev is 3D models. There are two ways to go about this for a beginner.
The first (and easiest) option is to use openly available 3D models while you learn the other areas of VR development. The Unity and Unreal asset stores have easily available models that can be used for this purpose. There are some links below to other websites with openly available assets. This can be critical for the beginner VR dev, as it's very hard to learn so many things at once!
The other option for 3D models is to learn how to make them yourself. Though this is the more difficult option, it's a good choice for the long term, as the time may come when you're making a more intermediate project and want to make your own assets and visual style. There are a few programs which are very useful for 3D modeling.
Even if you decide to find premade assets, you'll probably end up needing to tweak them in 3D modeling software so so that they're just right. Fortunately, there are plenty of resources available online. Professional design software is available at monthly subscription prices comparable to a MMORPG, and there are tutorials for just about every 3D modeling question freely available on YouTube (click for example for YouTube resources). Use the search function on every website you see! If you demand higher-quality education content, consider subscribing to PluralSight. Go through the various subreddits listed on the sidebar here and catch up on conversations in the different VR tech communities, and learn new tips and tricks.
- 3D Modelling and Sculpting:
- Autodesk's Entertainment Creation Suite of software (including Maya, 3ds Max, Motionbuilder, and Mudbox with native export to Unity and UE4) is available to "students" free for three years -- no verification needed. This includes everything you need to make professional models, textures, animations, etc.
- Pixologic ZBrush (from $795, student discounts available) is a 3D sculpting application, which provides more creative flexibility than traditional modelling applications such as Maya or 3ds Max, and is known for its powerful handling of high-polygon and photorealistic models. The functionally-equivalent Autodesk software is Mudbox.
- Blender is an entirely free and open source 3D modelling, animation, and game design suite. It is very powerful with great flexibility, but has a much steeper learning curve than commercially-developed software.
- You can buy and download models and 3D scans at sites like Turbosquid and Sketchfab.
- MODO Indie ($15/month or $300) is an 3D sculpting, painting, and animation tool that caters to game designers and hobbyist artists learning 3D.
- Speedtree ($19/month) is useful for designing procedurally-generated 3D models of trees, plants, and all sorts of branching structures. These can later be extracted with many options for use in photorealistic landscapes.
Photogrammetry (3D scanning)
Like VR, 3D photo scanning is another futuristic technology now available for dirt cheap, and with mobile solutions. 3D scanning using photos involves taking many photographs (usually upwards of thirty) of a real-life object from as many angles as possible. These photographs are then imported into software such as Agisoft Photoscan or one of Autodesk's many solutions (they keep buying up companies, it seems), which generate highly detailed meshes from these photographs. The meshes and their color/diffuse texture map can then be exported and used in a game engine as a regular asset. This YouTube video competently demonstrates the entire process in ten minutes.
- Photogrammetry and 3D-scanning
- Agisoft Photoscan (from $179), a 3D scanning suite which uses DSLR photos as its source.
- Autodesk offers a few different photogrammetry solutions, from the free mobile- and cloud-based 123D Catch to desktop photogrammetry software Remake and Recap 360. Here's a thread which discusses the differences between Autodesk's various photogrammetry solutions.
Audio & Music
Audio in VR doesn't need to be treated very differently from music and sound effects in movies or traditional games. Like with graphics, there is an emphasis on realism and quality. The most immersive audio in VR with be positionally spatialized depending upon the direction the player is facing respective to the sound source; Unity and UE4 must be configured for audio specialization to function correctly.
- Audio production
- Audacity, although simple on the surface, is a powerful and reliable audio editor that is entirely free and open source.
- For royalty-free stock audio, check out The Free Sound Project, the #GameAudioGDC Bundle, and the Oculus Audio Pack.
- Check out Oculus's introduction to audio spatialization and the Oculus Connect talk 3D Audio: Designing Sounds for VR
4. Implement Interactivity
After you've got a feel for the engine and have some artwork to use, the next hurdle is figuring out how you will add interactivity to your projects. I highly suggest first reading up about VR user interface (UI) and user experience (UX) principles -- otherwise your players might end up with achy eyes from poor stereoscopic rendering decisions, or even vomiting from motion sickness. Fortunately, these can be avoiding simply by avoiding locking text to the viewport, or by putting the player camera in a visible capsule (car, spacesuit, cockpit) during movement to reduce nausea. And if you plan to implement hand tracking, it's a good idea to focus on making everything as real as possible -- your research and prototyping will pay off with an impressive sense of presence.
VR UI/UX Resources
- Google's own Cardboard Design Lab is perhaps the fastest introduction to this subject.
- Google Developers - Google I/O 2015 Livefeed - Designing for virtual reality
- Research VR podcast, which covers the developing VR industry and cognitive science, emphasizing the relationship between intentional design and conscious experience.
- Leap Motion's VR Design Best Practices article is full of thoughtful suggestions
- Mitch's VR Lab, based on UE4, is a YouTube series many examples of UI programming and principles.
- Fuseman's Introduction to VR UI in Unity livestream tutorial, which explains concepts useful beyond Unity.
- UE4 HTC Vive - How to interact with a menu using Motion controllers on YouTube
- VR sickness
This requires some kind of scripting language. Unreal Engine 4 has an intuitive, flowchart-like scripting system called Blueprint Visual Scripting, and if you're not yet comfortable programming, this may be useful for you to get started. Check here for a general introduction to Blueprints. Blueprints are powerful enough to do entire projects without having to write a line of code (though you'll be using many programming concepts). Otherwise, both Unreal and Unity have a native programming language. For Unreal, it's C++, and for Unity, it's C#. Many people who are aspiring VR devs have little to no programming experience, so this step can be particularly daunting. We want to help you get past this hurdle also, so we'll be updating this wiki with resources that will teach you the fundamental programming concepts needed for VR dev.
5. Some final tips
If you're a solo VR dev, remember, start small. There's time for more grandiose visions once you have a solid grasp of the fundamentals, so to begin with start with a project that you think is very simple. Even Pong has more to it than you would think at the first glance. Work at it step by step, once you have a few projects up your belt you'll be in a much better position to attack some more complex problems.
SDKs or Software Development Kits are used to build VR apps. An app can either implement OVR or OpenVR or both. This means that the app has access to native functionality in it's corresponding runtime. SDKs do not handle asynchronous timewarp or reprojection, those are handled by the runtime!
- OVR - Made by Oculus VR for the Oculus Rift. Current version (14th May 2016) is 1.3.1 and can access all features of the Oculus Runtime.
Sidenote to SDK's and Unity games: Unity 5.3 currently has optimizations for VR in their native mode. The native mode supports Rift, Gear VR and PSVR, but not SteamVR. A game compiled with Unity 5.3 can use those optimizations with the Oculus SDK but not the OpenVR SDK. The OpenVR SDK has it's own optimizations, which may or may not result in similar performance. However, the upcoming Unity 5.4 will support SteamVR natively and performance should be more or less identical. Please note: this is Unity specific and other engines might have similar or different optimizations for some or all headsets.
- Oculus Runtime is responsible for asynchronous timewarp and handles device detection, display, etc. It (the runtime service) needs to be running for Oculus Home to launch.
If you launch a game on Steam that supports Rift natively (meaning, it has been compiled against the Oculus SDK), you will get asynchronous timewarp and it will run exactly like the game from Oculus Home. It won't use the the SteamVR Runtime, it will use the Oculus Runtime. Only downside is that launching the game is more difficult.
Rendered Image using the OpenVR SDK -> SteamVR Runtime -> Oculus Runtime -> Rift
The Oculus Runtime effectively thinks you are playing a game called "SteamVR", but the content of the "game" is actually the image that the SteamVR runtime got from the OpenVR compiled game. This seems to work rather well and personally I have not noticed any additional latency. In theory you would even get Asynchronous Timewarp, but that would only happen if SteamVR itself isn't responding in time. If an OpenVR game stalls completely, the SteamVR runtime will throw you back into the loading construct and continues to render at 90fps. Since the SteamVR runtime has this fail safe mechanism and all the Oculus Runtime sees is "SteamVR", it will most likely never trigger Asynchronous Timewarp.
App Stores are distribution platforms for software.
- Oculus Home needs to be running for the Rift to work. By default only supports apps from the Oculus Store (checkbox in the settings of the 2d desktop client to enable other sources). It downloads games and runs them. It also handles the Universal Menu on the Xbox button.
- Steam/SteamVR technically does not need to run when launching OpenVR games, but highly recommended (room setup and config is pulled from there). Also handles overlay menu on the Xbox button, or when running on the Rift, it launches by pressing the select/start button in the Oculus Universal Menu.
- Project Cars on Home + Rift -> will run natively
- Project Cars on Steam + Vive -> will run natively
- Project Cars on Steam + Rift + launched as Rift app -> will run natively
- Project Cars on Home + Vive -> won't run
- Project Cars on Steam + Rift + launched as SteamVR app -> will run via SteamVR
Wrappers and Injectors
Oculus Rift apps, or better said apps compiled with the OVR dlls do not render to the HMD directly. They render their images and send them to the runtime. The interface for that are the dlls. Wrappers replace the content of the dlls with functions that are named identically, but redirect the games rendered image to a different runtime. Equally, they provide the game with the rotation and position data of the headset.
- ReVive replaces the content of the dlls with functions that communicate with the SteamVR Runtime. The game thinks it's connected to a Rift, but it is not. The SteamVR runtime is rendering the game and supports reprojection, but not asynchronous timewarp (remember, this is a feature of the Oculus Runtime).
All Oculus games distributed through Oculus Home implement an entitlement check. This means the game is asking Oculus Home if the user is allowed to play it. ReVive does not hack this entitlement check and games need to be downloaded/purchased via Oculus Home. Games need to be started through Home once or the Oculus Service needs to be restarted. Example of the entitlement check failing in Lucky's Tale: Instead of the main menu, ReVive users see a worm on a log showing you it's tongue.
Sidenote: technically one could even write a wrapper that routes OpenVR calls to the Oculus runtime directly, bypassing SteamVR altogether. But since SteamVR can do it already, there is little need for that
- LibOVRWrapper installs dlls into Windows that old Rift games need in order to communicate with the new runtime. It effectively translates old games to the new runtime. [unsure: Do these old games now also use timewarp or do they need to specifically request it via an sdk call?]
- LibOVRWrapper + ReVive allows you to run old Rift games on Vive.
- See also: Resources Directory
Hardware & User Experience
Unreal Engine 4
- Unreal Engine YouTube Channel, full of tutorials, announcements, and stream recordings
- Unreal forums, and the Unreal VR development forum
- C++ (programming language for Unreal Engine 4)
- Blueprints Visual Scripting
- Video tutorials
- Unity3d Videos on YouTube, featuring many tutorials and panels
- Unity forums, and the Unity VR forum
- Unity tutorial section, has beginner and intermediate tutorials for using the engine.
- Microsoft Developer Network - Unity: Developing Your First Game with Unity and C#
- Unity manual
- C# (programming language for Unity)
- Unity and C# Video Tutorials by Sebastian Lague
- Unity tutorials for C#
- Unity documentation for C#
- Unity Coding Tips on YouTube
- Intro to C# programming and Scripting for Game in Unity, a $30 video tutorial series over at Udemy
- Introductory C# lessons with interactive code
- A handful of useful C# tutorials at SourceCodester
- Discussion forum for learning C# at Dream.In.Code
3D Models and Art Assets
- TurboSquid 3D modelling community
- Sketchfab 3D modelling community
- Blender, a free and open source 3D modelling program
- OpenGameArt has a trove of mixed-quality 3D models free for use.
- NASA's 3D Resources gallery has a number of public domain, space-themed 3D models. The files are in a few different formats, including .FBX, .3DS, .OBJ, and the less familiar .STL (stereolithography), but converters are easily found across the web.
- Normal Map Technical Details at the Polycount Wiki
- Texture Maps Explained: Physically-Based Rendering Workflow
- Oculus's Introduction to VR Audio
- Audacity a free, simple-yet-powerful audio editor
- For stock audio, check out The Free Sound Project, the #GameAudioGDC Bundle, and the Oculus Audio Pack
- Nvidia AndroidWorks provides all the SDKs needed to port a Unity or UE4 application to Android
- Android Studio is Android's all-in-one-suite for developing Android applications, and is useful for debugging and many development tasks. It also fulfills required SDKs.
- Google VR SDK for Unity (works for both Cardboard and Daydream)
- Google Cardboard Developer Overview, and the official Cardboard developer community on Google+
- Google Daydream Developer Overview
Additional VR/AR Platforms
- Google web tools
- Mozilla A-Frame
- Google I/O 2016 Livestream- Building High Performance Daydream Apps
- Leap Motion forum, and their development forum
- Leap Motion best practices guide, generally useful for studying user experience
- Yobi3D - 3D model search engine: offers a 3D viewer to see search results in 3D. Also works with Cardboard.