Apple Preparing for Upcoming Siri Onscreen Awareness Feature With New iOS 18.2 API for Developers

Apple is working on more advanced Siri functionality as part of its Apple Intelligence feature set, and to prepare, it has been providing developers with App Intent APIs so apps will be ready for the new capabilities.

Apple Intelligence General Feature 2
With the latest wave of betas, Apple has a new API that lets developers make onscreen content in their apps available to ‌Siri‌ and ‌Apple Intelligence‌. From Apple's documentation:

When a user asks a question about onscreen content or wants to perform an action on it, Siri and Apple Intelligence can retrieve the content to respond to the question and perform the action. If the user explicitly requests it, Siri and Apple Intelligence can send content to supported third-party services. For example, someone could view a website and use Siri to provide a summary by saying or typing a phrase like "Hey Siri, what's this document about?"

In the iOS 18.2 beta, ChatGPT integration with ‌Siri‌ allows users to ask questions about photos and documents, such as PDFs and presentations, and get information about them. You can, for example, ask ‌Siri‌ "what's in this photo?" and ‌Siri‌ will take a screenshot and hand it over to ChatGPT. ChatGPT then relays what's in the image, and the same feature works for PDFs and other documents.

It does not seem that the iOS 18.2 ChatGPT integration is the onscreen awareness functionality that Apple has planned for ‌Siri‌, but it could be related. Apple describes onscreen awareness as the ability for ‌Siri‌ to understand and take action on things on the screen. If someone texts you an address, for example, you'll be able to say "Add this address to their contact card," and ‌Siri‌ will do it. This functionality is not available in iOS 18.2, and ChatGPT is limited to assessing screenshots, but it is somewhat confusing.

Onscreen awareness, like personal context and in-app actions, is a feature that Apple has planned for ‌Siri‌, but it's probably not something that we're getting this year. Many of the ‌Siri‌ features are coming in a future version of iOS 18, and Bloomberg's Mark Gurman has said we can expect to see them in iOS 18.4, an update set to be released in the spring of 2025.

While there are multiple ‌Siri‌ features that won't be coming until next year, Apple is providing developers with APIs in advance so that developers have several months to prepare and so the features are ready for the public when the updates actually come out.

Related Roundups: iOS 18, iPadOS 18
Related Forums: iOS 18, iPadOS 18

Popular Stories

airtag purple

AirTag 2 Rumored to Launch Next Year With These New Features

Sunday November 17, 2024 5:18 am PST by
Apple released the AirTag in April 2021, so it is now three over and a half years old. While the AirTag has not received any hardware updates since then, a new version of the item tracking accessory is rumored to be in development. Below, we recap rumors about a second-generation AirTag. Timing Apple is aiming to release a new AirTag in mid-2025, according to Bloomberg's Mark Gurman....
Magic Mouse Next to Keyboard

No, Apple CEO Tim Cook Didn't Say He Prefers Logitech's MX Master 3 Over the Magic Mouse

Sunday November 17, 2024 3:03 pm PST by
While the Logitech MX Master 3 is a terrific mouse for the Mac, reports claiming that Apple CEO Tim Cook prefers that mouse over the Magic Mouse are false. The Wall Street Journal last month published an interview with Cook, in which he said he uses every Apple product every day. Soon after, The Verge's Wes Davis attempted to replicate using every Apple product in a single day. During that...
New Things Your iPhone Can Do in iOS 18

18 New Things Your iPhone Can Do in iOS 18.2

Wednesday November 13, 2024 2:09 am PST by
Apple is set to release iOS 18.2 next month, bringing the second round of Apple Intelligence features to iPhone 15 Pro and iPhone 16 models. This update brings several major advancements to Apple's AI integration, including completely new image generation tools and a range of Visual Intelligence-based enhancements. There are a handful of new non-AI related feature controls incoming as well....
iPhone 17 Slim Feature Single Camera 1 Redux

'iPhone 17 Air' Rumored to Surpass iPhone 6 as Thinnest iPhone Ever

Monday November 18, 2024 1:07 pm PST by
In a research note with Hong Kong-based investment bank Haitong today, obtained by MacRumors, Apple analyst Jeff Pu said he agrees with a recent rumor claiming that the so-called "iPhone 17 Air" will be around 6mm thick. "We agreed with the recent chatter of an 6mm thickness ultra-slim design of the iPhone 17 Slim model," he wrote. If that measurement proves to be accurate, there would be ...
iPhone 7 Lightning to Headphone Jack Adapter

Apple Seemingly Discontinuing Lightning to Headphone Jack Adapter Introduced Alongside iPhone 7

Sunday November 17, 2024 12:33 pm PST by
It appears that Apple is discontinuing the Lightning to 3.5mm headphone jack adapter that it released alongside the iPhone 7 and iPhone 7 Plus in 2016. The adapter was recently listed as "sold out" on Apple's online store in the U.S. and most other countries, according to MacRumors contributor Aaron Perris. The adapter remains available from Apple in only a handful of countries, such as...
Apple TV 4K hero 221018 feature

It's 2009 Again: Apple is Apparently Reconsidering Making a TV

Sunday November 17, 2024 5:27 am PST by
Between around 2009 and 2011, it was repeatedly rumored that Apple would be releasing a TV, but that obviously never happened. Now, a decade-and-a-half later, Bloomberg's Mark Gurman says the idea is back on the table. In his Power On newsletter today, Gurman briefly mentioned that Apple has been "evaluating" the "idea of making an Apple-branded TV set." He did not provide any further...

Top Rated Comments

smoker68x Avatar
2 weeks ago
Is this going to be a one time universal permission to Siri, or an individual access per app like camera/microphone/location?
Score: 4 Votes (Like | Disagree)
Fraserpatty Avatar
2 weeks ago
This is the type of Siri functionality that I have been waiting for. I hope Apple doesn’t disappoint.
Score: 3 Votes (Like | Disagree)
dannys1 Avatar
2 weeks ago

but it is somewhat confusing.
Is it?
Score: 2 Votes (Like | Disagree)
deeside81 Avatar
2 weeks ago
Siri still doesn’t know my Scottish accent. It’s absolutely useless and doesn’t work with me. After 10 years they’ve had plenty of time to fix Siri not understanding accents. If I put an American accent on it understands me clearly. Rubbish.
Score: 2 Votes (Like | Disagree)
svish Avatar
2 weeks ago
Good to know. Should make Siri much more useful.
Score: 2 Votes (Like | Disagree)
Oestreich Dieter Avatar
2 weeks ago
This is the true AI use case. Not the generative AI stuff. Being able to talk to your iPhone and have it execute commands across any app. Not sure why the negativity, if Apple pulls this off it will be great. Especially being able to set up complex short cuts by just speaking it. Sign me up.
Score: 1 Votes (Like | Disagree)