visionOS 2

visionOS 2 is the second-generation version of visionOS, the operating system that runs on the Vision Pro headset. visionOS offers a full three-dimensional interface that starts out with a Home View that is similar to the iPhone's Home Screen. It displays all of the apps that are installed on the Apple Vision Pro, with the visionOS interface able to respond dynamically to natural light and shadows to help users interpret distance and scale.

visionOS is navigated using eye movements, hand gestures, and voice control because it cannot be directly interacted with. Users can, for example, browse through apps by looking at them, and then select one with a tap of the fingers. This navigation scheme has been described as natural and intuitive by Vision Pro users. There are new gestures in visionOS 2, which can be used to access Home View quickly, control volume, check battery level, and open Control Center to get to frequently used features.

visionOS apps can be placed anywhere in virtual space visible to a user, and they can be resized to be ultra large or small. App windows can be stacked on one another, arranged around a room, and resized. Multiple apps can run at the same time, and it is unclear if there is an upper limit beyond what fits in the visual space. Apple describes visionOS as offering an infinite canvas, but users can also opt to focus on a single app that expands fully into the space.

In visionOS 2, there is an option to personalize the Home View by moving apps around for the first time. With a simple pinch and hold gesture, apps can be rearranged so you can put what you use most on the main app screen.

The level of immersion in visionOS is adjustable. Users can either see their content in their actual surroundings through the Apple Vision Pro's cameras, or a virtual "Environment" can make the real world fade away. Apple designed several Environments like Mount Hood that can be activated as a backdrop in visionOS, and there are new Environments in visionOS 2 like Bora Bora.

Apple tweaked Environments so you can see your Magic Keyboard or MacBook keyboard while immersed, and Environments have expanded to Safari. You can expand videos you come across while browsing the web, watching them in full screen in a dedicated environment. This works with embedded videos from YouTube, Netflix, and more.

Apple adapted existing apps like FaceTime, Photos, Messages, Safari, and more to work in a 3D environment. In Messages, Safari, and other apps, users can pull out 3D USDZ files to look at and manipulate virtual objects, and in Photos, images like Panoramas wrap around the wearer.

For the headset wearer, visionOS creates a Persona, or a digital representation that reflects face and hand movements in real time, with the Persona used in FaceTime and other video apps. With visionOS 2, Apple has improved Personas with more accurate skin tones and more vibrant clothing colors, plus you can add a variety of backgrounds and there are more fluid and natural hand movement animations.

In visionOS 2, Apple added a fun feature that lets you create 3D spatial photos from the 2D images in your library. This feature uses depth information and machine learning to add natural-looking depth and dimension. Apple redesigned the Photos app with a design that makes it easier to find the content that you want, plus there is a SharePlay for Photos option so you can see photos, videos, and panoramas at life scale and relive memories when everyone on FaceTime is using a Vision Pro.

With content apps like Apple TV+, visionOS can display TV shows and movies on a personal movie screen that feels 100 feet wide, plus visionOS offers Apple Immersive Videos that are 180-degree high-resolution recordings. visionOS supports Spatial Audio through the Apple Vision Pro's speakers. For sports fans, visionOS 2 will support Multiview for MLS and MLB games, with up to five able to be watched at once.

There is a dedicated App Store for visionOS, housing apps and content from developers. It also provides thousands of iPhone and iPad apps that automatically run on visionOS and that are adapted to the eye and hand controls. With iCloud, content is synced with iPhone, iPad, and Mac.

Bluetooth accessories like keyboards, trackpads, and game controllers work with visionOS, and Apple added support for mice with visionOS 2 so a Bluetooth mouse can be used to navigate while wearing the headset.

visionOS integrates with the Mac to allow the Apple Vision Pro to serve as a display for a Mac. The Mac's screen shows up in the virtual space provided by visionOS for a private, portable 4K display, and it's expanding in visionOS 2. Later in 2024, the Mac Virtual Display will have a larger ultrawide screen that Apple says is equivalent to two 4K displays sitting side-by-side.

For security and authentication, visionOS supports Optic ID, an iris scanning function that works like Touch ID or Face ID. Optic ID can be used for logging into visionOS, making purchases, and logging into websites.

A Travel Mode in visionOS stabilizes the visuals for Vision Pro wearers on airplanes, and in visionOS 2, there's also Travel Mode for trains. Travel Mode is designed to allow people to work privately or watch content while on a plane or a train.

Apple is improving Guest User mode in visionOS 2, which makes it easier for the headset to be shared with others and demonstrated to family and friends. The most recent guest's eye and hand data is saved for 30 days, so there's no need to repeat the setup process every time a guest uses the headset.

Other new features coming in the visionOS 2 update include an option to dictate in Messages just by looking at the mic icon, an AirPlay Receiver mode for mirroring content from nearby iPhones and iPads on the same network, breathing rate integration when doing a Mindfulness session, and collaborative playlists and sing-along sessions with friends in Apple Music.

Note: See an error in this roundup or want to offer feedback? Send us an email here.

visionOS is the operating system Apple designed for the Vision Pro headset, and Apple calls it the first spatial computing OS. While it is reminiscent of iOS, it is designed to be used in a virtual space. Apple came out with visionOS in February, and released visionOS 2 in September.

visionOS features an iOS-like Home View that serves as the main hub, and it is akin to the Home Screen. It's where you go to access all of the content that is available on the Vision Pro, including apps that you have installed. Software is accessed through apps, as on macOS and iOS. So if you want to watch a movie, play a game, or get work done, you need to open the appropriate app.

When the Vision Pro launched, there was no option for customizing the layout of the Home View, but in visionOS 2, you can move apps around and personalize what you see first. To move apps, just use a pinch and hold gesture.

visionOS features a full three-dimensional interface and all of the app icons, windows, and other objects respond dynamically to natural light and shadows to help you figure out the distance and scale of the virtual elements in the room.

App Windows

Apps open up like they do on the Mac, and you have an adjustable app window that you can position anywhere in the space around you. You can make windows big or small, and have as many windows open as you want around you.

You have an "infinite canvas" to work with, as Apple puts it, because all of the space around you is at your disposal. Apps can be right in front of your face, pushed further back in the room, tiled on top of each other, or spread around in the room.

A single app can fill up the entire area in front of you if you want to focus on one activity like a meditation.

Environments

Apple's cameras are designed to show you your surroundings so you can place apps and other digital objects in the room around you. With the Digital Crown, though, you can turn on full virtual reality to block out the real world.

Turning the Digital Crown adjusts your level of immersion, activating what Apple calls "Environments." Environments are virtual backdrops that take you to another place, such as Mount Hood. Apple has described these as full, 3D like places that make the real world fade away.

Available Environments include Bora Bora (new in visionOS 2), Haleakalā, Yosemite, Sky, Joshua Tree, Lake Vrangla, Mount Hood, The Moon, Beach, White Sands, Winter Light, Fall Light, Summer Light, and Spring Light.

With visionOS 2, Apple tweaked environments so you can see your Magic Keyboard or MacBook keyboard while you're immersed, making it easier to type. Environments can also now be used with Safari, so you can expand videos you come across while browsing the web and watch them in a full screen environment.

Safari Environment support works with embedded videos from YouTube, Netflix, and more.

Screen Sharing via AirPlay

Content that is displayed on the Apple Vision Pro can be mirrored to an iPhone, iPad, Mac, Apple TV, or AirPlay-compatible smart TV using AirPlay. Content can be mirrored to any AirPlay-enabled device at up to 720p, which allows the wearer to show people what's on the headset.

iPhone and iPad users who are nearby can AirPlay content to a Vision Pro thanks to an AirPlay Receiver mode Apple added with visionOS 2.

Personas

Because the Vision Pro is worn on the user's head and is not able to capture the user through an outward-facing camera for video calls and other purposes, Apple designed Personas.

Personas are a digital representation of the Vision Pro wearer, with a 3D scan captured using the Vision Pro headset while it is not worn. Personas reflect facial expressions and hand movements in real time, and Apple has been working to improve Personas for the last several months. Personas are primarily used in FaceTime and other video calls.

Personas used to be confined to a tile, but Spatial Personas allow for a more natural feel so users can have the impression of being located in the same physical space as another Vision Pro wearer.

In visionOS 2, Apple improved Personas with more accurate skin tones and more vibrant clothing color options, plus there are new backgrounds to choose and hand movement animations are more fluid and natural.

Controlling visionOS is done through eye movements, hand gestures, and voice commands. You look at something to highlight it, and then use a gesture to interact with it. The available gesture options are listed below.

• Tap - Tapping the thumb and the index finger together signals to the headset that you want to tap on a virtual element on the display that you're looking at. Users have also described this as a pinch, and it is the equivalent to tapping on the screen of an iPhone.
• Double Tap - Tapping twice initiates a double tap gesture.
• Pinch and Hold - A pinch and a hold is similar to a tap and hold gesture, and it does things like highlighting text.
• Pinch and Drag - Pinching and dragging can be used to scroll and to move windows around. You can scroll horizontally or vertically, and if you move your hand faster, you'll be able to scroll faster.
• Zoom - Zoom is one of two main two-handed gestures. You can pinch your fingers together and pull your hands apart to zoom in, and zooming out involves a pushing sort of motion. Window sizes can also be adjusted by dragging at the corners.
• Rotate - Rotate is the other two-handed gesture and based on Apple's chart, it involves pinching the fingers together and rotating the hands to manipulate virtual objects.

In visionOS 2, Apple added new pinch and rotate gestures for getting to the Home View quickly, accessing Control Center, controlling volume, and checking the battery level.

Gestures work with eye movements, and the many cameras in the Vision Pro track where you are looking with great accuracy. Eye position is a key factor in targeting what you want to interact with using hand gestures. As an example, looking at an app icon or on-screen element targets it and highlights it, and then you can follow up with a gesture.

Hand gestures do not need to be grand, and you can keep your hands in your lap. You only need a tiny pinch gesture for the equivalent of a tap, because the cameras can track precise movements.

There is an air typing feature that is designed to let you type on a full virtual keyboard. There is no tactile feedback, with Apple instead focusing on visual feedback when a key is "pressed."

Input Devices

Though visionOS is meant to be controlled with hand gestures and eye movements, it does support Bluetooth mice, keyboards, and trackpads, along with game controllers for playing games.

Mice, keyboards, and trackpads can be used alongside a Mac. Note that support for mice was added in visionOS 2.

Apple reimagined many of its familiar apps for a 3D interface, including FaceTime, Photos, Messages, and Safari.

FaceTime in particular is of interest because it shows the people you're conversing with in large tiles around the room, which you can adjust. People you're chatting with see your "Persona," a scan of your real face that reflects your facial expressions and hand movements in real time so you look like yourself rather than a person wearing a headset.

The Photos app shows you images in larger than life scale, with panoramas wrapping around you, for example, plus you can view 3D photos and videos. There's a built-in camera on the headset so you can capture video that you can watch back in full 3D.

With visionOS 2, your 2D images can be transformed into 3D spatial photos, with Apple using the embedded depth information and machine learning to add natural-looking dimension that makes your 2D photos feel much more real on the Vision Pro.

A SharePlay for Photos option lets you see photos, videos, and panoramas at life scale, and you can relive memories over FaceTime if all participants have a Vision Pro.

Later this year, Vision Pro users will be able to view spatial photos and videos in Safari, as Apple plans to add a feature that will let web developers share spatial photos and videos on their websites. For non-Vision Pro users, the images and videos will look like they do now, but they will be in 3D for Vision Pro wearers.

In the Messages app, visionOS 2 adds a feature for dictating messages simply by looking at the microphone icon, plus the Mindfulness app has support for breathing rate integration. It is able to monitor respiratory rate by using the cameras to watch you breathe.

Apple Music in visionOS 2 supports collaborative playlists and sing-along sessions.

Third-Party Apps

The visionOS App Store launched with 600 third-party Vision Pro-optimized apps available.

iPhone and iPad Apps

There is a dedicated visionOS App Store where you can find apps created specifically for the device, but it allows you to download existing iPhone and iPad apps. iPhone and iPad apps run on visionOS in a 2D environment, so you can use your favorite apps even if they're not updated for a 3D interface.

Immersive Experiences and Entertainment

The Apple TV app includes a selection of content recorded with the Apple Immersive Video format, which offers 180-degree 3D 8K recordings captured with spatial audio.

• Alicia Keys: Rehearsal Room - Provides a glimpse into Alicia Keys' creative process through an intimate rehearsal session.
• Adventure - Follows pioneering athletes as they take on awe-inspiring challenges in some of the world's most sensational locations.
• Wild Life - Allows viewers to get up close and personal with some of the most charismatic creatures on the planet.
• Prehistoric Planet Immersive - Transports users into the daily lives of dinosaurs.

The Vision Pro supports 3D movie and TV show content from a range of providers, with 3D movies available for purchase from Apple or for streaming through services like Disney+. Non-3D content can be watched on a virtual display that can be made to feel as if it's 100 feet wide.

Apple added a Multiview feature in visionOS 2 that's specifically for sports, and you can watch up to five MLS and MLB games.

Gaming

visionOS supports Apple Arcade games that can function with a connected Bluetooth gaming controller, plus Apple is working with gaming partners to create new gaming experiences for the device.

App Store

Like Apple's other operating system updates, visionOS has its own dedicated App Store where users can find third-party apps and games designed for the device.

Visual Search

visionOS includes a Visual Search feature that can be used to get information about an item, detect and interact with text in the world around them, copy and paste printed text from the real world into apps, translate text between 17 different languages, and more.

Real-world text that includes contact information, webpages, and unit conversions and similar information can be acted upon in visionOS. So, for example, if a printed handout has a website link in it, you can scan the link with the Vision Pro, opening up a Safari window to view the website. Or, if a recipe calls for grams and you need ounces, you can convert using the headset.

Real-time text translation is useful for traveling and other instances when you might want to quickly translate what you're seeing in the real world. The Apple Vision Pro headset can automatically detect text and documents, similar to how the iPhone can detect text in photos and allow it to be interacted with.

Travel Mode

Travel Mode can be turned on through the Control Center, and it is designed to stabilize the visuals on the Vision Pro for users that are wearing the headset on an airplane or a train, with the latter support available in visionOS 2.

Guest Users

Guest mode allows a user's family and friends to use the Vision Pro with limited access to certain apps, such as Photos and Safari. The primary user's Apple ID account, settings, Optic ID, passcode, passwords, Apple Pay, Persona, and health data are restricted while the Guest User mode is active.

Apple improved Guest support in visionOS 2 by saving the most recent guest's hand and eye data for 30 days, so there's no need to repeat the setup process each time a guest uses the headset.

The Vision Pro integrates with the Mac, serving as a display for either a notebook or desktop machine through a feature called Mac Virtual Display. You can see all of your Mac apps in a virtual space over AirPlay 2, using a keyboard and mouse as usual to get work done.

Note that you can use the display with a Mac mini or a Mac Studio that does not have its own display. A secondary display is not required.

With Mac Virtual Display turned on, visionOS apps can be used alongside the virtual Mac screen. The Vision Pro itself can be mirrored to a Mac over AirPlay or FaceTime.

There is no feature for using Mac apps individually in a mixed reality workspace, with separate windows for each. The Vision Pro instead serves as a display for a standard Mac desktop, so you can only operate apps within the single virtual desktop.

With visionOS 2, Apple is planning to add support for a larger ultrawide screen that Apple says is like using two 4K displays sitting side-by-side. This feature will come in a software update later in 2024.

• Your Mac and Apple Vision Pro: What You Can Do (and What You Can't)

Rather than Face ID or Touch ID, visionOS uses Optic ID, an all-new iris scanning security feature. Apple uses infrared lights and cameras inside the headset to look at the unique pattern of your iris for authentication purposes.

Optic ID can be used for logging into visionOS, making purchases, and in lieu of passwords in apps and websites.

visionOS 2 launched on September 16.