What is Apple Intelligence? A full breakdown of Siri AI, AI agents, JSON, and structured data

WWDC 2024 first wrote Apple Intelligence into the system. By 2026, it already covers iPhone, iPad, Mac, Apple Watch, and Vision Pro. On the surface: rewrite, summarize, Genmoji, Image Playground, Visual Intelligence. Underneath is something else: personal context, app actions, and screen awareness all have to be catchable by a program. How flagship cloud models get picked: see 2026 AI model war. Apple's line does not fight that leaderboard on parameter count. It fights for this: after the user finishes a sentence, can the system hand over a typed object, not a fluent paragraph.

What it actually is

On Apple's own developer page, Apple Intelligence is a personal intelligence system, driven by a new generation of Apple Foundation Models, that does three things: understand personal context, take actions in apps, and perceive what is on screen. It is not a chat app you download separately, and it is not a Settings toggle that "turns on ChatGPT."

All three have to hold at once for it to count as Apple Intelligence, not a one-off model demo:

1. The model lives in the system, not on a web page. On-device runs Apple Foundation Models; what will not fit, or needs multimodality, goes to Private Cloud Compute. The 2026 Foundation Models framework also allows external packages that conform to the Language Model protocol — the docs name cloud models like Claude and Gemini as things you can hook up.
2. Capabilities have to be registered; you cannot luck into them with a prompt. The app writes content and actions as an App Intents Schema. Siri, Shortcuts, and Spotlight read the same structure. You no longer maintain a phrase table like "Hey Siri, create an event with such-and-such app."
3. The output has to be catchable by the next hop. A sentence for people is fine. Writing a calendar, sending a message, editing the photo on screen — those must be entities and parameters, not "sometime Tuesday afternoon."

Three layers: on-device, cloud, system surfaces

Under the same brand name sit three layers. Mix them, and it feels like "Siri is sometimes an agent, sometimes just a rewrite tool."

Image Playground, Write with Siri, and Genmoji are surface-layer features. What developers actually write is the contract in the middle, plus Foundation Models sessions when they call them directly. Dynamic Profiles let you swap models, tools, and instructions inside one session; the Evaluations framework is how you accept under dynamic conditions, not one demo.

Siri AI: the wake-word era is over

Old Siri was a phrase table plus regex. New Siri (Apple's developer docs write Siri AI) eats a Schema. You do not write "Hey Siri, create a meeting with Calendar" for every phrasing. The system has already been trained for years on these domains: calendar, photos, comms, tasks. The app only declares which domain it belongs to, which entities it has, and which intents it can do.

1. App Intents is the contract

The App Intents framework hooks the app into Apple Intelligence. Two Schemas do different jobs:

• Entity Schema describes content. Entities that conform to IndexedEntity donate into Spotlight's semantic index, so the system can cite your app with attribution instead of dumping an unsourced summary.
• Intent Schema describes actions. Create an event, send a message, open a detail — all typed parameters, not free text. When language understanding and dialects expand, you do not change code.

This is the same class of problem as cloud Function Calling, with a different wrapper. OpenAI writes tool parameters as JSON Schema; Claude writes input_schema; Apple writes calendar_createEvent, system.open inside a system domain. The user hears "book a review tomorrow afternoon"; the calendar app receives title, start time, duration, attendees — the fields must be there, the types must be right.

WWDC 2026 also filled in the test surface: App Intents Testing walks the real system path for Siri, Shortcuts, and Spotlight. You no longer write a UI automation suite to gamble on the wake word.

2. "This" on screen needs an entity too

Registering intents is not enough. Users will point at the screen and say "move this to Thursday" or "reply to the second one." Without a binding, Siri can only guess. The View Annotations API marks visible views onto entities: this message is a MessageEntity, this landmark is a PlaceDescriptor. Once the referent has an object, the intent has parameters.

ValueRepresentation lets an entity travel between apps with a structure the system knows — hand a landmark to Maps for navigation, instead of exporting an untyped string. EntityCollection avoids fully resolving every entity when you tag a thousand photos — the intent receives a set of identifiers, not a prose roster.

Does this count as an AI agent

Apple did not package Siri as a separately sold "agent product." In behavior, it is already an agent loop: understand the goal → pick one or more Intents → fill parameters → clarify if needed → user confirms → execute → hand the result back as an entity. Shortcuts goes further: stitch multi-step automation in natural language. Apps that adopted App Intents enter that workflow, next to system actions like "Use Model."

Against a cloud agent, the difference is the bus, not whether you need structure:

• Common cloud path: the model picks a tool, parameters go through JSON Schema, the runtime can be MCP JSON-RPC. This site's MCP Tool page is writing that contract.
• Apple path: the model (or Siri) picks a system Intent, parameters go through a domain Schema; an on-device session can also hang custom Tools on Foundation Models.

Neither side is "say another paragraph." Missing fields, type drift, enums that fly off — the loop stops, or dirty data lands in the calendar. The DevDay piece already took the cloud loop apart; this piece fills in the same loop on the system bus. How expensive training and inference get: see How much it costs to train a model — on-device models take some inference off the token bill, but the contract layer does not disappear.

@Generable: under the Swift type sits JSON Schema

The Foundation Models framework is a native Swift API. Free text is enough with LanguageModelSession.respond. Production features — recipe fields, a ticket, a landmark pulled from a photo — want an object. Apple's move is not "please return JSON." It is called Guided Generation:

1. Add @Generable to a struct or enum.
2. When you need constraints, add @Guide on the property: description, count, range, regex.
3. Call session.respond(to:generating:). The framework generates with constrained sampling, then parses into your Swift type.

Apple's own docs say it straight: for each Generable type, the framework converts the type and format information into JSON Schema and hands it to the model. The Schema eats context window. Keep property names short, drop @Guide descriptions you do not need, and split hard tasks across sessions, or you drop a contextSizeExceeded. When the shape is only known at runtime, use DynamicGenerationSchema — do not pretend a compile-time type is enough.

This is the same technique as GPT Structured Outputs, Gemini's responseSchema, and Claude's output_config: constrained decoding guarantees the shape. The wrappers differ. The three put the Schema in an HTTP request body; Apple compiles the Schema from a Swift macro, so day to day you touch types, not curly braces. When it hits logs, evals, cross-device sync, or your own backend, the object still has to be catchable by a JSON parser and JSON Schema. Why the contract picks JSON: see Why AI likes JSON.

Streaming is structured too. The framework generates a PartiallyGenerated companion type: fields start optional, then fill in one by one. The UI updates object fields, not a Markdown paragraph that keeps getting longer.

How it maps to GPT / Claude / Gemini / MCP

What they like is the same object, not the same request body. Write the wrapper into the docs; leave the fields in a shared file.

What you can share is the fields: enums, required, booleans, date formats, additionalProperties: false. What you cannot share is the wrapper. Grow a first Schema from the same sample, then hook it to the three HTTP APIs, an MCP tool, and the Swift type you plan to map to @Generable. Tutorial: Generate Structured Output from a sample.

One intent sample that can validate

This is not Apple's private request body. It is the business object all three layers should line up on when Siri needs to create an event. Enums as strings, duration as number, confirmed as boolean. On-device @Generable can generate the same shape; App Intent calendar_createEvent parameters can line up on these fields too.

{
  "source": "apple_intelligence",
  "surface": "siri",
  "intent": {
    "domain": "calendar",
    "schema": "calendar_createEvent",
    "confirmed": true
  },
  "event": {
    "title": "Schema review",
    "starts_on": "2026-09-09",
    "starts_at": "14:00",
    "duration_min": 45,
    "location": "Example Labs",
    "attendees": [
      "alex@example.com"
    ]
  },
  "context": {
    "device": "iphone",
    "on_device": true,
    "on_screen_ref": "this event"
  },
  "confidence": 0.91,
  "needs_confirmation": false
}

With this site's inference rules, that sample yields:

Once the sample is ready, open Structured Output to generate the OpenAI / Claude / Gemini wrappers, then review required and additionalProperties by hand. The same object can also become a tool via Function Calling or MCP Tool: the model is not "talking about a meeting"; it is submitting an intent that must validate.

How to accept it in JSON Toolbox

1. Paste the sample above into JSON Format and confirm it parses.
2. Generate a first draft with Structured Output or JSON → Schema. That is the step @Generable does for you at compile time, just in another language.
3. Take a second set of real output — Siri got the duration wrong, wrote "Thursday" as a relative date — to Schema validation. The first sample is always a bit too clean.
4. When you map an on-device object onto a cloud agent trace, use JSON Diff to see whether a field dropped or a type drifted.
5. Convert to Zod when the frontend needs checks, and to OpenAPI 3.1 when you need API docs.

Everything stays in the browser and is never uploaded.

FAQ

Is Apple Intelligence ChatGPT stuffed into the iPhone?

No. It is system-level personal intelligence: on-device Apple Foundation Models, Private Cloud Compute, and the App Intents contract shared by Siri / Shortcuts / Spotlight. The chat window is only one surface.

How does Siri AI call a third-party app?

Through App Intents. Entity Schema donates content into Spotlight's semantic index; Intent Schema lets Siri act from natural language, with no wake-word table. View Annotations bind on-screen views to entities, so you can say "reschedule this."

Is Apple's structured output JSON?

Developers write Swift types and @Generable / @Guide. The framework compiles those types to JSON Schema at compile time, then generates with constrained sampling. When it hits disk, logs, and cross-device checks, the contract layer is still an object plus a Schema, not a paragraph.

Does this count as an AI agent?

Siri is the conversational surface, not a separately sold agent product. The real loop is: understand → pick an Intent → fill parameters → confirm → execute. Shortcuts can also stitch multi-step workflows in natural language. In and out are entities and intents, not free text.

How does it map to GPT / Claude / Gemini Structured Output?

The three write JSON Schema into an HTTP API. Apple folds the same job into Swift macros and system Schemas. Field contracts can map; wrappers cannot be shared. Grow a Schema from the same sample, then hook it to response_format, @Generable, or an App Intent.

Summary

Apple Intelligence is a personal intelligence system, not a chat skin. Siri AI understands content and takes actions through an App Intents Schema; View Annotations turn "this" into an entity. Foundation Models' @Generable compiles JSON Schema underneath; constrained sampling guarantees the shape. The agent loop — pick a tool, fill parameters, confirm, execute — is the same class of problem as cloud Function Calling / MCP. What survives next week's system update is entities, intents, and fields, not a wake word in a prompt. Grow the contract from one sample; validation, Zod, OpenAPI, and the Swift type you plan to map all follow it.

Related tools