Question 1

How do I test if the Apple On-Device OpenAI server is working?

Accepted Answer

Run the included test_server.py script, which checks health, model availability, SDK compatibility, and demonstrates streaming and multilingual support. Ensure the server is running on the default localhost:11535 port first.

Question 2

What macOS version and settings do I need to run this?

Accepted Answer

You need macOS 26 beta 2 with Apple Intelligence enabled in Settings, and Xcode 26 beta 2 if building from source. The pre-built app requires the macOS version, making it inaccessible for users on stable releases.

Question 3

Can I use this with JavaScript or Node.js OpenAI clients?

Accepted Answer

Yes, any OpenAI-compatible client can point to http://127.0.0.1:11535/v1 as the base URL, since it mimics OpenAI's endpoints. Set the API key to a dummy value like 'not-needed' as shown in the Python example.

Question 4

How does the performance compare to GPT-4 from OpenAI?

Accepted Answer

Apple's on-device models are optimized for local speed and privacy but may have lower accuracy or fewer capabilities for complex tasks compared to GPT-4. They're free and offline, but trade-offs exist in model power and versatility.

Question 5

Is there a way to avoid the rate limiting without restarting?

Accepted Answer

No, the README states that rate limiting is a current limitation of Apple FoundationModels, and restarting the server is the only workaround. This makes it unreliable for applications requiring sustained, high-volume requests.

Question 6

How to integrate this into an existing SwiftUI app?

Accepted Answer

Run the server as a separate app and connect via HTTP requests, or build from source in Xcode to embed the logic. However, direct integration may require modifying the project structure and handling the GUI app lifecycle.

Question 7

Should I use this or Llama.cpp for local AI on Mac?

Accepted Answer

Choose this if you need OpenAI API compatibility and Apple's integrated models for privacy; Llama.cpp offers more model choices and cross-platform support but requires more setup. It depends on whether API ease or flexibility is priority.

Apple On-Device OpenAI

What is Apple On-Device OpenAI?

Overview

Use Cases

Best For

Related Projects

Found a gem we're missing?

Not Ideal For

Pros & Cons

Pros

Cons

Open Source Alternative To

Frequently Asked Questions