Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display

Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.

apple vision framework human body pose detection jumping jack
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.

Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.

apple vision framework hand pose detection
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.

apple vision framework hand emoji
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.

Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.

Popular Stories

iPhone SE 4 Thumb 1

iPhone SE 4 With Apple's Own 5G Modem 'Confirmed' to Launch in March

Tuesday November 19, 2024 12:12 pm PST by
Barclays analyst Tom O'Malley and his colleagues recently traveled to Asia to meet with various electronics manufacturers and suppliers. In a research note this week, outlining key takeaways from the trip, the analysts said they have "confirmed" that a fourth-generation iPhone SE with an Apple-designed 5G modem is slated to launch towards the end of the first quarter next year. In line with previo...
General Black Friday Deals 24 Green Tinsel

Apple Black Friday Deals Available Now: AirPods, iPads, and More

Friday November 22, 2024 5:28 am PST by
Black Friday 2024 is less than one week away, and as always the next few days will be the best time of the year to shop for great deals. Right now, this includes big savings on popular Apple products like AirPods, Apple Watch, MacBook Air, iPad, and more. Note: MacRumors is an affiliate partner with some of these vendors. When you click a link and make a purchase, we may receive a small...
at t turbo indicator iphone 16 pro max v0 8hrh7w5f3w1e1

AT&T Turbo Indicator Showing Up in iPhone Status Bar for Subscribers

Wednesday November 20, 2024 3:42 am PST by
AT&T has begun displaying "Turbo" in the iPhone carrier label for customers subscribed to its premium network prioritization service, according to reports on Reddit. The new indicator seems to have started appearing after users updated to iOS 18.1.1, but that could be just coincidence. Image credit: Reddit user No_Highlight7476 The Turbo feature provides enhanced network performance through ...
anker new xmas 1

Anker Kicks Off Massive Black Friday Sale With Up to 50% Off Sitewide, Free Gifts With Purchase, Mystery Boxes, and More

Thursday November 21, 2024 7:53 am PST by
Anker today kicked off its big Black Friday sale, which is set to run through December 9. This sale includes notable discounts on portable chargers, USB-C hubs, cables, and more. Note: MacRumors is an affiliate partner with Anker. When you click a link and make a purchase, we may receive a small payment, which helps us keep the site running. There are a few bonus offers during this event as ...
Apple 2024 Black Friday Shopping Event feature

Apple Announces 2024 Black Friday Event, Offering Up to $200 Gift Card

Thursday November 21, 2024 5:10 am PST by
Apple's annual four-day Black Friday through Cyber Monday shopping event is returning on Friday, November 29 through Monday, December 2 in many countries, including the U.S., Canada, Australia, France, Germany, Italy, Spain, the U.K., and others. During the event, customers can get an Apple gift card with the purchase of an eligible product. In the U.S., for instance, Apple is including gift ...
apple card feature2

Apple Card 3% Daily Cash Back Now Available From Two More Apple Partners

Tuesday November 19, 2024 10:36 am PST by
Apple has partnered with select merchants to offer Apple Card users three percent Daily Cash back on their purchases, and two new companies were added to the partner list today. When purchasing goods and services from Booking.com and ChargePoint, Apple Card users will now get more cash back. Booking.com is a site for reserving flights, cars, cruises, and hotels, while ChargePoint sells...
silo tv show apple tv plus

Apple TV+ Releasing Next Week's 'Silo' Episode Early

Friday November 22, 2024 7:25 am PST by
The next episode of Apple TV+'s award-winning sci-fi series "Silo" will be released early. Apple previously announced that new "Silo" episodes would be released on Fridays, but the third episode of the second season will instead be released on Wednesday, November 27. Apple has likely bumped up the date so that people can watch the episode during the U.S. Thanksgiving holiday on Thursday,...
Generic iOS 18 Feature Real Mock

Apple Releases iOS 18.1.1 and iPadOS 18.1.1 With Security Fixes

Tuesday November 19, 2024 10:10 am PST by
Apple today released iOS 18.1.1 and iPadOS 18.1.1, minor updates to the iOS 18 and iPadOS 18 operating systems that debuted earlier in September. iOS 18.1.1 and iPadOS 18.1.1 come three weeks after the launch of iOS 18.1. The new software can be downloaded on eligible iPhones and iPads over-the-air by going to Settings > General > Software Update. Apple has also released iOS 17.7.2 for...

Top Rated Comments

smuzzy Avatar
58 months ago
When these phones can snap a photo right at the optimum height of a group jump, we will truly be in the future.
Score: 13 Votes (Like | Disagree)
incoherent_1 Avatar
58 months ago
That’s great. I have a hand gesture I’ve been giving to Siri for years. Now maybe she’ll get the message.
Score: 12 Votes (Like | Disagree)
luvbug Avatar
58 months ago

Seems a lot like what Xbox was able to do with the Kinect 10 years ago
That's funny, I didn't think the Xbox was a mobile phone??
Score: 6 Votes (Like | Disagree)
Appleman3546 Avatar
58 months ago
Seems a lot like what Xbox was able to do with the Kinect 10 years ago
Score: 5 Votes (Like | Disagree)
AngerDanger Avatar
58 months ago


Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.

Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
Score: 4 Votes (Like | Disagree)
SDJim Avatar
58 months ago

The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.
I don't think the wizarding community is going to be too happy about this...
Score: 3 Votes (Like | Disagree)