Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.
Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.
The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.
Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.
Wednesday November 13, 2024 2:09 am PST by Tim Hardwick
Apple is set to release iOS 18.2 next month, bringing the second round of Apple Intelligence features to iPhone 15 Pro and iPhone 16 models. This update brings several major advancements to Apple's AI integration, including completely new image generation tools and a range of Visual Intelligence-based enhancements. There are a handful of new non-AI related feature controls incoming as well....
Thursday November 14, 2024 4:19 pm PST by Juli Clover
The M4 MacBook Pro models feature quantum dot display technology, according to display analyst Ross Young. Apple used a quantum dot film instead of a red KSF phosphor film, a change that provides more vibrant, accurate color results.
Young says that Apple has opted for KSF for prior MacBook Pro models because it doesn't use toxic element cadmium (typical for quantum dot) and is more...
Wednesday November 13, 2024 11:01 am PST by Juli Clover
A trio of Apple customers this month filed a class action lawsuit against Apple, accusing the Cupertino company of violating California consumer protection laws and false advertising for continuing to sell AirPods Pro models that had ongoing issues with crackling or static sounds.
A few months after the AirPods Pro came out in October 2019, buyers began to complain about crackling, rattling, ...
Thursday November 14, 2024 2:54 am PST by Tim Hardwick
Google has launched its dedicated Gemini artificial intelligence app for iPhone users, expanding beyond the previous limited integration within the main Google app. The standalone app offers enhanced functionality, including support for Gemini Live and iOS-specific features like Dynamic Island integration.
The new app allows iPhone users to interact with Google's AI through text or voice...
Wednesday November 13, 2024 11:59 am PST by Juli Clover
Apple last week replaced the M3 Max MacBook Pro with the new M4 Max MacBook Pro, and we picked up one of the new high-end MacBook Pro machines to see how it compares to the prior model with both benchmarks and real-world tests.
We tested an M4 Max with a 16-core CPU, 40-core GPU, and 48GB RAM against an M3 Max MacBook Pro with similar specs. The two machines look similar, but the display on...
Thursday November 14, 2024 2:19 pm PST by Juli Clover
With iOS 18, Apple introduced a feature that causes the iPhone to reboot every three days, security researchers have confirmed (via TechCrunch). In a demo video, security researcher Jiska Classen proved that an iPhone left untouched for 72 hours will automatically restart, and Graykey manufacturer also Magnet Forensics wrote a blog post about the feature.
After a reboot, an iPhone is more...
Honestly, this seems like the kinda stuff that'd make Apple AR compelling—being able to draw in midair means you’d also be able to navigate an interface in midair with just your hands.
Using AR/VR without bringing a controller everywhere seems analogous to what set the iPhone apart from other touchscreen phones in 2007; you didn’t need a stylus.
The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing.
I don't think the wizarding community is going to be too happy about this...
