Runtime API
The entire Highlight runtime API reference.
To begin using the runtime API, install
The NPM package primarily provides you with types, the actual API is injected at runtime by the Highlight app. Thus, these methods will throw errors if your web app attempts to access them outside of Highlight.
How it works
Your Highlight app runs in its own isolated web view. Before your app loads, we inject a set of APIs into the web view to allow your app to interact with the user’s computer. This is similar to how a regular web browser exposes APIs. However, Highlight APIs are designed to bridge the gap, allowing you to build AI apps without needing to start from scratch with MacOS/Windows exclusive APIs. Instead, with Highlight, you get a uniform API that works across all major operating systems.
Because your app is still a typical web app, you can still use all the standard web APIs you’re used to. This includes things like fetch
, localStorage
, and more.
Types
HighlightContext
The HighlightContext
type allows your application to access all the context from the user’s computer to help assist them with their request. This object is created when the user activates the Highlight hotkey or clicks the Highlight icon.
Inference
Run local inference through Highlight without needing your own backend.
Get Text Prediction
Generate text predictions using an LLM model based on provided messages.
Model Information
The current LLM inference model is Llama 3.1, featuring:
- 405 billion parameters
- Context size of up to 64,000 tokens
Usage
Parameters
-
messages
: An array of message objects, each containing:role
: string (“system”, “user”, or “assistant”)content
: string
-
options
: (Optional) Configuration object
Example
This function returns an async iterable, allowing you to process the prediction in chunks as they arrive.
User
APIs that involve user data.
Get context
Asynchronously fetches the context of the user’s computer. Similar to the behavior of the onContext
event listener. However, you may poll this method to get repeated updates.
Get facts
Returns a list of facts about the user. These facts are set by the user during Highlight onboarding and can later be adjusted in Highlight settings.
Get audio
Returns the latest buffer of audio data from the user’s computer.
Get audio by duration
Returns the audio transcript for the previous duration seconds of time
Get email
Returns a privacy-protected email address of the user (for example: abcdefghij@highlight.email). This email address forwards to the user’s main email address.
You can use this for implementing auth on top of an existing system you already have.
Get screenshot
Returns a Base64 encoded string with the image data from the user’s screen.
Get display screenshots
Returns a list of desktop image objects, one corresponding to each of the user’s displays. Each object includes a Base64 encoded string with the image data for that display. Your app must be granted screenshot permissions by calling requestScreenshotPermission
before you can use this function.
Get windows
Returns a list of the user’s open windows as well as an app icon for that window if available.
Get window screenshot
Returns a Base64 encoded string of an image of the window with the provided title. Window titles can be accessed by calling getWindows
. Your app must be granted screenshot permissions by calling requestScreenshotPermission
before you can use this function.
App Storage
App storage is a secure, isolated place to keep your app’s data safe in Highlight.
We plan on later supporting secure cloud sync (with user opt-in required). Along with other upcoming features (like a unified way to share documents/attachments with other apps), we recommend using app storage over localStorage.
Loading Data
When your app first loads in Highlight, it will asynchronously request data stored on disk via appStorage. This typically happens within milliseconds, but if you want to ensure the data is ready, you can use:
Get Data
To access data, use the get
method:
Set Data
To set data, use the set
method:
Delete Data
To delete data, use the delete
method:
Auth
Sign the user in with Highlight
Signs the user into your app automatically. Returns an access token (JWT) and a refresh token that can be used to validate the session on your backend and keep the user signed in. This access token is also used to make requests to the Highlight web API (coming soon).
The JWT can be decoded to fetch the sub
or subscriber ID (a unique ID that remains consistent across all Highlight apps).
For more information on using Highlight auth, read the authentication article.
Under the hood, using the .signIn() method makes a web request to Highlight’s backend and thus, it is best not to repeatedly call this function on every page load.
Permissions
Request permissions for your Highlight app to grant it additional capabilities.
Request background permission
Requests permission for your Highlight app to run in the background. After requesting permission, you will need to call setBackgroundStatus
to actually set the background status.
Request screenshot permission
Requests permission for your Highlight app to capture screenshots of the user’s displays and windows. This permission must be granted in order to receive responses from getDisplayScreenshots
and getWindowScreenshot
.
Request clipboard read permission
Requests permission for your Highlight app to read the contents of the user’s clipboard. This permission must be granted in order to receive a response from getClipboardContents
.
App
Methods that help your app feel native to Highlight. This category deals with app to Highlight communication.
Determine if your app is running inside of Highlight
You can check to see if your app is running inside of Highlight by using the isRunningInHighlight
method.
Add event listener
Registers an event listener for your app. When this event occurs, the callback function will be called. This function returns a destroy function that you can call to remove the listener when you are done with it.
Desktop Shortcuts
You can check if the user has created a desktop shortcut for you app, and if not you can prompt them to create one.
Get hotkey
Get the hotkey the user should use to bring up the Highlight overlay window. This is helpful when making an onboard presentation for your users.
Open App
Opens a different app by its ID in Highlight.
Set background status
Tells Highlight whether your app should run in the background. You will need to request background permission first.
If set to true, your app will start when Highlight starts and continue running until Highlight is closed.
Calling setBackgroundStatus(true)
will throw an error if you don’t have permission to run in the background.