# SmartSpectra documentation > SmartSpectra SDK developer docs for v3.4.0, the latest stable release. Measure pulse, breathing and more from camera video on iOS, Android, desktop C++ and Node.js. Every page is Markdown. The SmartSpectra SDK is for general wellness use; only SmartSpectra Vital Signs Monitor 1.0 is FDA-cleared. ## SmartSpectra SDK Source: https://presagetech.com/docs/smartspectra-sdk # SmartSpectra SDK This repository hosts SmartSpectra SDK from PresageTech for measuring vitals such as pulse, breathing, and more using a camera. The SDK supports Android, iOS, Node.js and Electron, and C++ for Windows, macOS, and Linux. ## Table of Contents - [SmartSpectra SDK](#smartspectra-sdk) - [Getting Started](#getting-started) - [Build with an AI Assistant](#build-with-an-ai-assistant) - [Features](#features) - [Model Cards and Limitations](#model-cards-and-limitations) - [Telemetry and Privacy](#telemetry-and-privacy) - [Authentication](#authentication) - [Platform-Specific Guides](#platform-specific-guides) - [Android](#android) - [iOS](#ios) - [Node.js and Electron](#nodejs-and-electron) - [Windows / Mac / Linux (C++)](#windows--mac--linux-c) - [Bugs & Troubleshooting](#bugs--troubleshooting) ## Getting Started To get started, follow the instructions for one of our currently supported platforms. Each platform has its own integration guide and example applications to help you get up and running quickly. Documentation is available at . ## Build with an AI Assistant This repo ships `using-smartspectra`, an [Agent Skill](https://agentskills.io) that teaches an AI coding assistant how to build apps with the SDK. The same skill works in both Claude Code and OpenAI Codex: - **Claude Code:** `/plugin marketplace add Presage-Security/SmartSpectra`, then `/plugin install smartspectra-sdk@smartspectra` - **Codex:** `$skill-installer Presage-Security/SmartSpectra/.agents/skills/using-smartspectra` Then ask your assistant to build with SmartSpectra. Full guide: [docs/agent-skill.md](docs/agent-skill.md). Presage also hosts an MCP (Model Context Protocol) server at `https://mcp.presagetech.com/mcp`. Connect it and your assistant can work with your developer account directly — fetch or rotate your API key, register an iOS or Android app ID, download its OAuth config file, and check your plan and credits. Setup: [docs/mcp-server.md](docs/mcp-server.md). ## Features - **Cardiac Waveform** Real-time pulse pleth waveform supporting calculation of pulse rate and heart rate variability. - **Breathing Waveform** Real-time breathing waveform supporting biofeedback and breathing rate. - **Face Analysis** Supporting face-point analysis, iris tracking, blinking detection, talking detection, and facial expression classification. - **Relative Arterial Pressure Waveform** Relative arterial pressure waveform shape. This is not a blood pressure measurement and cannot estimate systolic or diastolic pressure. - **Integrated Quality Control** Confidence and stability metrics providing insight into the confidence in the signal. User feedback on imaging conditions to support successful use. - **Camera Selection** Discover and select cameras by facing or ID on Android, iOS, Node.js, and C++. - **Caller-owned Video** Submit frames from your own camera or decoder with the Android, Swift, Node.js, or C++ custom-input API. - **LLM Insights** Turn the vitals the SDK computes on-device into natural-language analysis from a large language model, on request. See the [LLM Insights overview](docs/llm-insights/index.md). ## Model Cards and Limitations Model cards include: - Relative arterial pressure waveform - Heart rate variability - Pulse rate - Face analysis - Breathing rate - Upper and lower breathing waveforms Review the [SmartSpectra model cards and limitations](docs/model-cards-and-limitations.md) for metric-specific operating ranges, validation notes, and known limitations. ## Telemetry and Privacy The SDK can report a small, aggregate, per-session diagnostic summary so we can measure release quality across the devices and conditions the SDK runs on. It is aggregate-only — no raw video, no measured vitals values, no user or device identifiers — and it is opt-out. See [SDK Telemetry & Privacy](docs/telemetry-and-privacy.md) for exactly what is and isn't collected and how to disable it. ## Authentication - We support API key authentication for Android, iOS, Node.js, and C++. We also support OAuth authentication for iOS and Android. See the platform-specific guides for setup instructions. - Either path can be set up by an AI assistant instead of by hand — see [docs/mcp-server.md](docs/mcp-server.md). ## Platform-Specific Guides ### Android For Android integration, refer to the [Android README](android/README.md). The guide includes: ![SmartSpectra Android quickstart demo](android/media/android-quickstart.gif) - Prerequisites and setup instructions. - Maven setup for stable releases, release candidates, and snapshots. - Integration steps for your app. - Example usage and troubleshooting tips. ### iOS For iOS integration, refer to the [iOS README](swift/README.md). The guide includes: ![SmartSpectra iOS quickstart demo](swift/docs/ios-quickstart.gif) - Prerequisites and setup instructions. - Integration steps for your app using Swift Package Manager. - Example usage and troubleshooting tips. ### Node.js and Electron For Node.js and Electron integration, refer to the [Node.js README](nodejs/README.md). The guide includes: - Supported operating systems and Node.js versions. - npm installation and runtime packaging instructions. - Native camera, Electron renderer, and custom-input workflows. - Example usage and troubleshooting tips. ### Windows / Mac / Linux (C++) For C++ integration on Windows, macOS, and Linux, refer to the [C++ README](cpp/README.md). The guide includes: ![SmartSpectra C++ quickstart demo](cpp/docs/images/cpp-quickstart.gif) - Supported systems and architectures. - Installation via the prebuilt ZIP (Windows), Homebrew (macOS), or apt (Linux). Debian 13 / Trixie supports both `amd64` and `arm64`; follow the [Debian installation and quickstart guide](cpp/docs/linux/debian-13.md). - Build instructions and example applications. - A Linux redistribution path for bundling the published SDK tarball into your own `.deb`: [Redistribute SmartSpectra on Linux](docs/redistribute_smartspectra_on_linux.md). ## Bugs & Troubleshooting For additional support, contact or [submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues). ## Android Quick Start Source: https://presagetech.com/docs/smartspectra-sdk/android --- title: Android Quick Start description: Measure pulse and breathing from the Android camera or caller-supplied frames with the SmartSpectra Android SDK. Kotlin, minSdk 28; install, authenticate, and run. sidebarTitle: Quick Start --- # SmartSpectra Android Quickstart This repo contains two build guides that produce similar user end states: - [Option 1: API Key](docs/option-1-api-key.md) - [Option 2: OAuth](docs/option-2-oauth.md) The only difference in builds is that the API key build gets up and running very fast, but hard-codes your API key. The OAuth build is more suitable for production deployments because it avoids hard-coding your API key. > **Start with the full runnable sample:** [Option 1: API Key](docs/option-1-api-key.md) > walks through a complete camera app. Use [Option 2: OAuth](docs/option-2-oauth.md) > when your app needs OAuth instead. ## Scope These quickstarts intentionally request only: - `SmartSpectraConfig.breathingMetrics` - `SmartSpectraConfig.cardioMetrics` - `MetricType.EXPRESSIONS` Please see the detailed documents for additional features. ## Important Implementation Rules Start by creating a new Android app project named `Cool Vitals` with package name `com.example.coolvitals`. The Quick Start is intended so that the developer can replace `app/src/main/java/com/example/coolvitals/MainActivity.kt` as a full file. - Use `SmartSpectraSdk.shared`. - Use `PreviewView` with `sdk.config.previewSurfaceProvider` for native CameraX preview. - Disable bitmap preview frames with `sdk.config.imageOutputEnabled = false`. - Buffer trace samples locally before drawing charts. - Keep camera permission in the activity flow with `ActivityResultContracts.RequestPermission()`. ## Choose Your Guide Use [Option 1: API Key](docs/option-1-api-key.md) for the fastest manual setup. Use [Option 2: OAuth](docs/option-2-oauth.md) if you need OAuth. Either way, an AI assistant connected to the [SmartSpectra MCP Server](../docs/mcp-server.md) can do the account side for you — fetch your API key, or register your package name and signing certificate fingerprint and download `presage_services.xml`. If your app already captures video, use the [custom-input guide](docs/headless-mode.md#caller-owned-camera-and-custom-input) to submit buffers, Bitmaps, or CameraX images while retaining camera ownership. ## LLM Insights See [LLM Insights](docs/llm-insights.md) for natural-language analysis of the measured vitals. ## Logging SDK log verbosity defaults to warnings and errors only. To change it, set `logLevel` on the config you pass to `initialize`: ```kotlin val config = SmartSpectraConfig().apply { logLevel = SmartSpectraLogLevel.INFO } SmartSpectraSdk.initialize(context, config) ``` Levels are cumulative — `DEBUG`, `INFO`, `WARNING` (default), `ERROR`, `NONE`. The setting covers both the SDK's Kotlin-side logging and the native engine. `DEBUG` cannot restore debug-only statements compiled out of the release engine binary. ## Supported Platforms | Platform | Notes | | ----------------------- | --------------------------------- | | Google Pixel 3 / 3 XL | Tested on Pixel 9, 10 | | Samsung Galaxy S21 | Tested on A16, S21, Galaxy Flip 6 | | Motorola Edge 30 Ultra | | For support: contact or [submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues). ## Android API Reference Source: https://presagetech.com/docs/smartspectra-sdk/android/api-reference --- title: Android API Reference description: API reference for the SmartSpectra Android SDK — Kotlin types, lifecycle methods, metrics, and error codes. sidebarTitle: API Reference --- ## SmartSpectraSdk Entry point for the SmartSpectra SDK. Most apps use the shared singleton initialized by AndroidX Startup, or call [initialize] with a custom [SmartSpectraConfig] before accessing [shared]. ```kotlin class SmartSpectraSdk ``` ### Methods - ```kotlin public suspend fun start() ``` Begin processing the selected input. In custom mode, await this call before submitting frames. Camera capture is the default until [useCustomInput] is called. - ```kotlin public suspend fun stop() ``` Stop processing. Call [start] again to resume. - ```kotlin public fun useCustomInput(frameTransform: FrameTransform = FrameTransform.NONE): CustomInput ``` Select caller-supplied frames while stopped. No camera discovery, capture, or camera permission is required. Replacing the source invalidates older handles. Throws [SmartSpectraException] with INVALID_STATE during processing or startup. - ```kotlin public fun useCamera(selection: CameraSelection) ``` Select SDK camera capture while stopped. Default prefers front, then the first discovered camera. Explicit selections never fall back. Resolution happens when capture starts; unavailable cameras report INPUT_UNAVAILABLE. Throws CONFIGURATION_FAILED for an empty ID and INVALID_STATE while busy. Rejected requests preserve the previous source. Stop/start/reset retain the chosen selection until another source is selected. - ```kotlin @Deprecated("Pass an explicit CameraSelection, such as Default or ById(id).") public fun useCamera() ``` - ```kotlin public suspend fun reset() ``` Stop processing and clear measurement output, retaining the selected input, transform, configuration, and custom-input handle. Call [start] to begin again. Safe to call while stopped; teardown runs off the caller's thread. - ```kotlin public fun requestInsight(text: String): Int ``` Dispatch an on-demand insight request alongside the vitals samples buffered since the last send. Requires an active measurement: the engine owns the insight session for the duration of a session, so call this after [start]. - ```kotlin @SmartSpectraTestingApi public fun setVideoInputEnabled(enabled: Boolean) ``` Enables or disables video-frame input mode for automated testing. While enabled, the SDK does not open the device camera (so no camera hardware or `CAMERA` permission is needed) and expects the caller to supply frames through [addVideoFrame]. Select only while stopped; after stopping, toggle back to `false` to return to normal camera capture. Throws [SmartSpectraException] with INVALID_STATE if processing or startup is active. New integrations should use [useCustomInput], which requires no testing opt-in. - ```kotlin @SmartSpectraTestingApi public fun addVideoFrame(frame: Bitmap, timestampUs: Long) ``` Feeds one decoded video frame into the measurement pipeline while video-frame input mode is active (see [setVideoInputEnabled]). Call after [start] has completed. Decode your recorded clip however you like (for example `MediaMetadataRetriever` or `MediaCodec`) and deliver frames in playback order. Throws [SmartSpectraException] when video input has not been selected or the frame is rejected. Use [CustomInput.sendFrame] for typed per-frame return values. - ```kotlin public suspend fun availableCameras(context: Context): List ``` Discover camera IDs, optional names, and facing without initializing SmartSpectra, authenticating, or starting capture. No permission prompt is shown. Visibility depends on platform permissions; a snapshot does not guarantee a later capture will succeed. Throws INPUT_UNAVAILABLE when discovery fails. Android currently provides no display names. - ```kotlin @JvmStatic @JvmOverloads fun initialize( context: Context, config: SmartSpectraConfig = SmartSpectraConfig(), ): SmartSpectraSdk ``` Initialize the shared SDK instance and apply the provided configuration. ### Properties - ```kotlin val config: SmartSpectraConfig ``` - ```kotlin public val metrics: LiveData = _metrics ``` Latest metrics snapshot (keep-latest). Suitable for scalar state such as pulse rate or status. For the per-sample traces, prefer [metricsFlow]: LiveData can drop intermediate emits under main-thread congestion, losing trace samples. - ```kotlin public val metricsFlow: SharedFlow = _metricsFlow.asSharedFlow() ``` Every metrics emission, in order — lossless streaming. Prefer this over [metrics] for the per-sample traces; deliveries are buffered, so nothing is dropped under consumer main-thread congestion. - ```kotlin public val insight: LiveData = _insight ``` - ```kotlin public val imageOutput: LiveData = _imageOutput ``` Latest engine display frame as an ARGB_8888 [Bitmap], or `null` while image output is disabled. The bitmap is reused from a small internal pool and refilled on later frames. `ImageView.setImageBitmap` keeps the reference (it does not copy) and reads the pixels at draw time, so displaying each frame as it arrives is safe — the pool covers the delivery latency. You MUST copy it (`Bitmap.copy(...)` or draw into your own bitmap) if you retain or cache it, or refresh the view slower than frames arrive; otherwise its pixels change under you. - ```kotlin val processingStatus: LiveData ``` - ```kotlin val validationStatus: LiveData = _validationStatus ``` - ```kotlin val error: LiveData = _error ``` - ```kotlin val version: String ``` - ```kotlin val shared: SmartSpectraSdk ``` Shared SmartSpectra SDK instance. ## SmartSpectraConfig Configuration for [SmartSpectraSdk]. Most apps access configuration through [SmartSpectraSdk.config], or pass a prebuilt config to [SmartSpectraSdk.initialize] before using [SmartSpectraSdk.shared]. ```kotlin public class SmartSpectraConfig() ``` ### Properties - ```kotlin @Deprecated("Use sdk.useCamera(CameraSelection.Front), Back, or ById instead.") public var cameraPosition: CameraPosition ``` Legacy facing preference; defaults to front and updates active capture. Deprecated: use [SmartSpectraSdk.useCamera] with an explicit [CameraSelection]. After typed selection, this preference applies again only after legacy `useCamera()`. - ```kotlin public var apiKey: String? = null ``` - ```kotlin public var logLevel: SmartSpectraLogLevel = SmartSpectraLogLevel.DEFAULT ``` Verbosity of SDK logging — both the SDK's own logging and the native engine. Set it before [SmartSpectraSdk.initialize] for full effect; later changes apply to the SDK's own logging immediately and to the engine when the next measurement session starts. Defaults to [SmartSpectraLogLevel.WARNING] (warnings and errors only). - ```kotlin public var imageOutputEnabled: Boolean = true ``` - ```kotlin public var enableTelemetry: Boolean = true ``` Controls whether the SDK reports anonymous, aggregate usage telemetry. On by default; set `false` to opt out. When enabled, the SDK sends a per-session summary — no raw frames, metric values, file paths, or user/device identifiers — to help Presage improve SDK reliability. Reporting is best-effort and never blocks a measurement. Read when a measurement session starts. - ```kotlin public var previewSurfaceProvider: Preview.SurfaceProvider? ``` Optional CameraX preview surface supplied by a host app. When set, the SDK binds a CameraX [Preview] use case alongside image analysis so the host can display the camera stream through a native `PreviewView` without consuming bitmap image-output frames. - ```kotlin public var requestedMetrics: List? ``` Metrics the SDK requests for authorization and output. The getter never returns `null` — when no list has been set (or after explicitly setting `null`), it falls back to [breathingMetrics]. The setter accepts `null` to reset back to that default. - ```kotlin public val breathingMetrics: List = listOf( MetricType.CHEST_BREATHING, MetricType.ABDOMEN_BREATHING, MetricType.BREATHING_RATE, MetricType.BREATHING_AMPLITUDE, MetricType.APNEA, MetricType.RESPIRATORY_LINE_LENGTH, MetricType.BASELINE, MetricType.INHALE_EXHALE_RATIO, ) ``` Breathing metric bundle. Equivalent to leaving [requestedMetrics] unset. - ```kotlin public val cardioMetrics: List = listOf( MetricType.PULSE_RATE, MetricType.ARTERIAL_PRESSURE_TRACE, MetricType.HRV, ) ``` Cardio metric bundle (pulse rate, arterial pressure trace, HRV). Combine with [breathingMetrics] for the typical "vitals" bundle. - ```kotlin public val faceMetrics: List = listOf( MetricType.FACE_LANDMARKS, MetricType.BLINKING, MetricType.TALKING, MetricType.EXPRESSIONS, ) ``` Face metric bundle (landmarks, blinking, talking, expressions). Requires the face-metrics-enabled bundle. - ```kotlin public val edaMetrics: List = listOf( MetricType.EDA_TRACE, ) ``` EDA Proxy (electrodermal activity) trace metric bundle. ## CameraPosition - `public fun fromLensFacing(@CameraSelector.LensFacing lensFacing: Int): CameraPosition = when (lensFacing)` - `FRONT` - `BACK` ## CustomInput Caller-supplied frame input, obtained from [SmartSpectraSdk.useCustomInput]. The handle survives stop/start and reset; selecting another source invalidates it. Submit after start completes, from a worker thread, in timestamp order. ```kotlin public fun sendFrame(frame: VideoFrame, timestampUs: Long): FrameSubmissionResult ``` Consumes the borrowed pixels before returning. Timestamps are strictly increasing microseconds on one monotonic timeline per run. Gaps greater than two seconds are rejected; stop and start the SDK to begin a fresh timeline after an interruption. Inspect the returned result: timestamp rejections do not end the session or publish a global SDK error. A later frame on the accepted timeline can still be sent. Calls are serialized; no queue of caller-owned buffers is retained. ```kotlin public fun sendFrame(frame: Bitmap, timestampUs: Long): FrameSubmissionResult ``` Copies a bitmap for submission. The caller retains ownership of the bitmap. ```kotlin public fun sendFrame(frame: ImageProxy): FrameSubmissionResult ``` Submits a full YUV_420_888 image, applying its clockwise rotationDegrees metadata before the selected session transform. Uses its capture timestamp converted from nanoseconds to microseconds. The caller must close the image after this returns. Unrotated images borrow the planes directly; rotated images require a pixel copy. Cropped images are rejected. Plane buffer positions and limits are preserved. ## FrameTransform Spatial transform applied to every frame in a custom-input session. - `NONE` - `ROTATE_90_CW` - `ROTATE_90_CCW` - `ROTATE_180` - `MIRROR_HORIZONTAL` - `MIRROR_VERTICAL` ## PixelFormat Pixel layout of a [VideoFrame.Packed] buffer. - `RGB` - `BGR` - `RGBA` - `BGRA` - `NV12` - `NV21` - `YUYV` ## ProcessingStatus - `IDLE` - `STARTING` - `RUNNING` - `STOPPING` - `ERROR` ## ValidationStatus ```kotlin public data class ValidationStatus( val code: ValidationCode, val hint: String, ) ``` ### Properties - ```kotlin val code: ValidationCode ``` - ```kotlin val hint: String ``` ## ValidationCode - `val wireValue: Int` - `OK` - `NO_FACE_FOUND` - `MULTIPLE_FACES_FOUND` - `FACE_NOT_CENTERED` - `FACE_SIZE_OUT_OF_RANGE` - `TOO_DARK` - `TOO_BRIGHT` - `CHEST_NOT_VISIBLE` - `CAMERA_TUNING` - `FRAME_RATE_TOO_LOW` - `EXCESSIVE_MOTION` - `FACE_TOO_CLOSE` - `FACE_TOO_FAR` - `FACE_TOO_HIGH` - `FACE_TOO_LOW` - `FACE_NOT_FORWARD` ## SmartSpectraError A typed error from the SmartSpectra SDK. Lifecycle methods throw [SmartSpectraException] wrapping this type, and async pipeline failures are published on [SmartSpectraSdk.error]. ```kotlin public data class SmartSpectraError( val code: Code, val message: String, val retryable: Boolean = false, ) ``` ### Properties - ```kotlin val code: Code ``` - ```kotlin val message: String ``` - ```kotlin val retryable: Boolean = false ``` ## SmartSpectraError.Code SDK error codes. Raw values are stable across SDK versions and match the C++/Swift wire values. - `INVALID_STATE` - `AUTHENTICATION_FAILED` - `CONFIGURATION_FAILED` - `CREDIT_EXHAUSTED` - `NETWORK_ERROR` - `SERVER_ERROR` - `INPUT_UNAVAILABLE` - `PROCESSING_FAILED` - `FRAME_CONVERSION_FAILED` - `NON_MONOTONIC_TIMESTAMP` - `TIMESTAMP_GAP` ## SmartSpectraException ```kotlin public class SmartSpectraException( public val error: SmartSpectraError, ) : RuntimeException(error.message) ``` ### Properties - ```kotlin val error: SmartSpectraError ``` ## CameraSelection Requested camera: `CameraSelection.Default`, `Front`, `Back`, or `CameraSelection.ById(id)` for an exact ID from [SmartSpectraSdk.availableCameras]. ## CameraSelection.Default Prefer front, otherwise use the first camera in discovery order. ```kotlin public data object CameraSelection.Default : CameraSelection ``` ## CameraSelection.Front ```kotlin public data object CameraSelection.Front : CameraSelection ``` ## CameraSelection.Back ```kotlin public data object CameraSelection.Back : CameraSelection ``` ## CameraSelection.ById An opaque, device-local camera ID. An unavailable ID fails without fallback. ```kotlin public data class CameraSelection.ById(val id: String) : CameraSelection ``` ### Properties - ```kotlin val id: String ``` ## CameraFacing Camera facing, including cameras that have no front/back position. - `FRONT` - `BACK` - `UNKNOWN` ## CameraLensType Best-effort lens classification, independent of digital zoom. Android compares the nominal view with the default camera of the same facing. Missing metadata and logical cameras combining multiple lenses report [UNKNOWN]. - `UNKNOWN` - `WIDE_ANGLE` - `ULTRA_WIDE` - `TELEPHOTO` ## CameraInfo A discoverable camera; availability may change before capture starts. ```kotlin public data class CameraInfo( val id: String, val name: String? = null, val facing: CameraFacing = CameraFacing.UNKNOWN, val lensType: CameraLensType = CameraLensType.UNKNOWN, ) ``` ### Properties - ```kotlin val id: String ``` - ```kotlin val name: String? = null ``` - ```kotlin val facing: CameraFacing = CameraFacing.UNKNOWN ``` - ```kotlin val lensType: CameraLensType = CameraLensType.UNKNOWN ``` ## SmartSpectraLogLevel Verbosity of SDK logging, set via [SmartSpectraConfig.logLevel]. Levels are cumulative: a level shows its own messages plus everything more severe. The setting covers both the SDK logging and the native engine. [DEBUG] cannot restore debug-only statements that were compiled out of the release engine binary. Wire values are stable across SDK versions and match the C++ `SmartSpectraLogLevel` values. - `DEBUG` - `INFO` - `WARNING` - `ERROR` - `NONE` ## FramePlane A borrowed direct buffer with row and pixel strides in bytes. ```kotlin public data class FramePlane( val buffer: ByteBuffer, val rowStride: Int, val pixelStride: Int, ) ``` ### Properties - ```kotlin val buffer: ByteBuffer ``` - ```kotlin val rowStride: Int ``` - ```kotlin val pixelStride: Int ``` ## VideoFrame Caller-owned pixels. Buffers must be direct; only bytes between position and limit are used. Submission leaves those positions and limits unchanged. Keep the buffers alive and unchanged until [CustomInput.sendFrame] returns, then reuse them freely. ## VideoFrame.Packed A single buffer. NV12/NV21 require even dimensions and row stride, with the UV/VU plane immediately after rowStride * height bytes, using the same stride. YUYV requires an even width. Include padding after the final row. ```kotlin public data class VideoFrame.Packed( val buffer: ByteBuffer, val width: Int, val height: Int, val rowStride: Int, val pixelFormat: PixelFormat, ) : VideoFrame ``` ### Properties - ```kotlin val buffer: ByteBuffer ``` - ```kotlin val width: Int ``` - ```kotlin val height: Int ``` - ```kotlin val rowStride: Int ``` - ```kotlin val pixelFormat: PixelFormat ``` ## VideoFrame.Yuv420 Flexible YUV 4:2:0 planes. Y pixel stride must be 1; U and V must share pixel stride 1 or 2. Chroma dimensions round up for odd image dimensions. Padding after a plane's final sample is optional. ```kotlin public data class VideoFrame.Yuv420( val width: Int, val height: Int, val y: FramePlane, val u: FramePlane, val v: FramePlane, ) : VideoFrame ``` ### Properties - ```kotlin val width: Int ``` - ```kotlin val height: Int ``` - ```kotlin val y: FramePlane ``` - ```kotlin val u: FramePlane ``` - ```kotlin val v: FramePlane ``` ## FrameSubmissionResult Result of a frame submission; acceptance does not mean processing has finished. ## FrameSubmissionResult.Accepted ```kotlin public data object FrameSubmissionResult.Accepted : FrameSubmissionResult ``` ## FrameSubmissionResult.Rejected ```kotlin public data class FrameSubmissionResult.Rejected(val error: SmartSpectraError) : FrameSubmissionResult ``` ### Properties - ```kotlin val error: SmartSpectraError ``` ## Headless Mode on Android Source: https://presagetech.com/docs/smartspectra-sdk/android/headless-mode --- title: Headless Mode on Android description: Use SmartSpectra Android lifecycle and metrics LiveData with your own UI, or submit custom camera frames through direct buffers, Bitmap, and CameraX ImageProxy. sidebarTitle: Headless Mode --- # Headless Mode (Android) The SDK doesn't ship UI. `SmartSpectraSdk.shared` exposes LiveData for `processingStatus`, `validationStatus`, `metrics`, `error`, and (optionally) `imageOutput`. Observe them from your Fragment or Activity with `observe(viewLifecycleOwner)`. The sample apps include a measurement UI; your own integration looks however you want. Use this when you want to: - Monitor vitals in the background while the app shows other content - Build a custom measurement UI ## Processing Status Lifecycle states: | Status | Meaning | | --- | --- | | **Idle** | Pipeline is not running | | **Starting** | Pipeline is initializing | | **Running** | Actively measuring — data is flowing | | **Stopping** | Teardown in progress, will return to Idle | | **Error** | Something went wrong | ## Example Use `SmartSpectraSdk.shared` directly for headless processing: ```kotlin import android.Manifest import android.os.Bundle import android.content.pm.PackageManager import android.view.View import android.widget.ImageView import androidx.activity.result.contract.ActivityResultContracts import androidx.core.content.ContextCompat import androidx.fragment.app.Fragment import androidx.lifecycle.lifecycleScope import com.presagetech.smartspectra.CameraSelection import com.presagetech.smartspectra.ProcessingStatus import com.presagetech.smartspectra.SmartSpectraError import com.presagetech.smartspectra.SmartSpectraSdk import kotlinx.coroutines.launch class HeadlessFragment : Fragment() { private val sdk by lazy { SmartSpectraSdk.shared.apply { config.apiKey = "YOUR_API_KEY" useCamera(CameraSelection.Front) config.imageOutputEnabled = true } } private val requestCameraPermission = registerForActivityResult(ActivityResultContracts.RequestPermission()) { granted -> if (granted) { startMonitoring() } else { showCameraPermissionUi() } } override fun onViewCreated(view: View, savedInstanceState: Bundle?) { super.onViewCreated(view, savedInstanceState) val previewImage: ImageView = view.findViewById(R.id.headless_preview_image) sdk.processingStatus.observe(viewLifecycleOwner) { status -> when (status) { ProcessingStatus.IDLE -> showIdleUi() ProcessingStatus.STARTING -> showLoadingUi() ProcessingStatus.RUNNING -> showRecordingUi() ProcessingStatus.STOPPING -> showStoppingUi() ProcessingStatus.ERROR -> showErrorUi() } } sdk.validationStatus.observe(viewLifecycleOwner) { status -> updateStatusHint(status?.hint.orEmpty()) } sdk.imageOutput.observe(viewLifecycleOwner) { bitmap -> previewImage.setImageBitmap(bitmap) } sdk.metrics.observe(viewLifecycleOwner) { metrics -> renderMetrics(metrics) } sdk.error.observe(viewLifecycleOwner) { error -> if (error?.code == SmartSpectraError.Code.INPUT_UNAVAILABLE) { showCameraPermissionUi() } } } private fun startMonitoring() { if (ContextCompat.checkSelfPermission(requireContext(), Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED ) { requestCameraPermission.launch(Manifest.permission.CAMERA) return } viewLifecycleOwner.lifecycleScope.launch { sdk.start() } } private fun stopMonitoring() { viewLifecycleOwner.lifecycleScope.launch { sdk.stop() } } } ``` ## Caller-owned camera and custom input Use `sdk.useCustomInput()` when your app already captures video, such as a camera shared with a video call, or supplies decoded video frames. In this mode the SDK does not discover, open, or control a camera and does not require camera permission. Your app remains responsible for permissions needed by its own capture code. Select the input while stopped, await `start()`, and then submit frames: ```kotlin import com.presagetech.smartspectra.FrameSubmissionResult import com.presagetech.smartspectra.FrameTransform import com.presagetech.smartspectra.SmartSpectraSdk val sdk = SmartSpectraSdk.shared sdk.config.apiKey = "YOUR_API_KEY" val input = sdk.useCustomInput(frameTransform = FrameTransform.NONE) sdk.start() // Suspends until ready; call from a coroutine. try { // Supply an upright Bitmap and its capture timestamp in microseconds. when (val result = input.sendFrame(bitmap, timestampUs)) { FrameSubmissionResult.Accepted -> Unit is FrameSubmissionResult.Rejected -> showFrameError(result.error) } } finally { sdk.stop() } ``` `Accepted` means the frame was submitted, not that its measurement has finished. Continue observing `sdk.metrics`, `sdk.validationStatus`, and `sdk.error` as with SDK-owned capture. A rejected submission carries a typed `SmartSpectraError`; do not assume every submitted frame was accepted. ### Buffers and image ownership Use `VideoFrame.Packed` for a direct pixel buffer, or `VideoFrame.Yuv420` for three YUV planes: ```kotlin import com.presagetech.smartspectra.FramePlane import com.presagetech.smartspectra.VideoFrame val frame = VideoFrame.Yuv420( width = width, height = height, y = FramePlane(yBuffer, yRowStride, 1), u = FramePlane(uBuffer, uRowStride, uvPixelStride), v = FramePlane(vBuffer, vRowStride, uvPixelStride), ) val result = input.sendFrame(frame, timestampUs) ``` For packed pixels, supply `VideoFrame.Packed(buffer, width, height, rowStride, pixelFormat)` with a typed `PixelFormat`. Dimensions describe the supplied pixels; row and pixel strides are measured in bytes. Buffers must be direct `ByteBuffer` instances. Each buffer's current `position` identifies the first byte and its `limit` bounds the readable region. Submission leaves those values unchanged. The caller owns the buffers. Keep their pixels unchanged until `sendFrame()` returns; they can then be reused. The Bitmap overload is a convenience for callers without direct buffers. The CameraX overload, `input.sendFrame(imageProxy)`, borrows an `ImageProxy` and uses its capture timestamp. It accepts only `YUV_420_888` with a full-image crop. Unrotated images borrow the planes directly; rotated images require a Bitmap copy. Always close the image yourself, including after rejection: ```kotlin try { val result = input.sendFrame(imageProxy) if (result is FrameSubmissionResult.Rejected) { showFrameError(result.error) } } finally { imageProxy.close() } ``` ### CameraX capture example The following helper binds a CameraX analyzer and submits each image to the SDK. Configure SDK authentication and grant camera permission before calling `start()`. Supply your `ProcessCameraProvider`, a worker `Executor`, and an error callback. Call `start()` and `stop()` serially from a coroutine, passing your `LifecycleOwner`, `CameraSelector`, and display rotation to `start()`. The error callback runs on the analyzer's worker thread; dispatch UI updates to the main thread. The helper closes each image after submission and unbinds only its own capture use case. Your app owns the supplied executor and its shutdown. ```kotlin import androidx.camera.core.CameraSelector import androidx.camera.core.ImageAnalysis import androidx.camera.lifecycle.ProcessCameraProvider import androidx.lifecycle.LifecycleOwner import com.presagetech.smartspectra.FrameSubmissionResult import com.presagetech.smartspectra.SmartSpectraError import com.presagetech.smartspectra.SmartSpectraSdk import java.util.concurrent.Executor import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.NonCancellable import kotlinx.coroutines.withContext class CustomCameraInput( private val sdk: SmartSpectraSdk, private val provider: ProcessCameraProvider, private val worker: Executor, private val onFrameError: (SmartSpectraError) -> Unit, ) { private var analysis: ImageAnalysis? = null suspend fun start(owner: LifecycleOwner, selector: CameraSelector, displayRotation: Int) { check(analysis == null) { "Capture is already started" } val input = sdk.useCustomInput() sdk.start() try { withContext(Dispatchers.Main.immediate) { val capture = ImageAnalysis.Builder() .setTargetRotation(displayRotation) .setOutputImageRotationEnabled(true) .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_YUV_420_888) .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) .build() analysis = capture capture.setAnalyzer(worker) { image -> try { val result = input.sendFrame(image) if (result is FrameSubmissionResult.Rejected) onFrameError(result.error) } finally { image.close() } } provider.bindToLifecycle(owner, selector, capture) } } catch (error: Exception) { withContext(NonCancellable) { stop() } throw error } } suspend fun stop() { withContext(Dispatchers.Main.immediate) { analysis?.let { capture -> capture.clearAnalyzer() provider.unbind(capture) } analysis = null } sdk.stop() } } ``` ### Orientation and timing With `FrameTransform.NONE`, submit upright pixels in the orientation you want measured. The Bitmap and buffer overloads have no rotation metadata; normalize their pixels or select a fixed `FrameTransform`. The `ImageProxy` overload applies `imageInfo.rotationDegrees` (0, 90, 180, or 270 degrees) first, then the session transform; other rotation values are rejected. Avoid applying the same rotation twice. Use the same orientation convention throughout the run. Supply strictly increasing timestamps in **microseconds**, on one monotonic timeline per run. Values must be nonnegative and less than `Long.MAX_VALUE - 2`. For camera timestamps in nanoseconds, divide by `1_000L`; for decoded video use the clip's presentation timestamps. A gap greater than two seconds is rejected. After a feed interruption that creates such a gap, call `sdk.stop()` and then `sdk.start()` before resuming submission. Do not rewrite live timestamps to hide the gap. ### Switching sources and reusing handles Call `useCustomInput()` and `useCamera(selection)` only while stopped. Source selection during processing throws `SmartSpectraException`. To return to SDK-owned capture, stop, call `sdk.useCamera(CameraSelection.Default)`, then start. To require a specific camera, pass its discovered ID instead. Handles survive stop/start and reset. Selecting another source—including another `useCustomInput()` call—invalidates them. Frames submitted while stopped or through an invalid handle are rejected. Reset stops processing and clears measurement state while preserving the selected input and transform. Await reset, then start before submitting more frames. ## Reading Metrics `sdk.metrics` is the same LiveData property here as in any other integration — there's no separate "headless" API. See [Android Metrics](metrics.md) for the metric request configuration and the field-by-field reading guide. ## Headless Testing in CI on Android Source: https://presagetech.com/docs/smartspectra-sdk/android/headless-testing-in-ci --- title: Headless Testing in CI on Android description: Run a full SmartSpectra measurement on an Android emulator in CI by feeding a recorded video, or smoke-test that the SDK builds and initializes. sidebarTitle: Headless Testing in CI --- # Headless Testing in CI (Android) See [Headless Testing in CI](../../docs/headless-testing-in-ci.md) for the cross-platform overview of what's automatable and why. This page covers the Android specifics. ## What's different on Android The SDK normally measures from the live camera. Its public custom-input API also accepts frames decoded by your app or test. An emulator can run a recorded-video measurement without camera hardware or `CAMERA` permission: ```kotlin val input = sdk.useCustomInput() // Select while stopped. sdk.start() val result = input.sendFrame(bitmap, timestampUs) ``` Unlike the iOS SDK, the Android SDK does not decode the file itself. Your test supplies decoded frames (for example via `MediaMetadataRetriever`, as below, or `MediaCodec`) with strictly increasing microsecond timestamps from the clip. See [custom input](headless-mode.md#caller-owned-camera-and-custom-input) for lifecycle, frame ownership, orientation, and timing requirements. The earlier `setVideoInputEnabled` / `addVideoFrame` helpers remain testing-only and require `@OptIn(SmartSpectraTestingApi::class)`. The public `useCustomInput()` API requires no opt-in. To update an existing test to the public API: | Testing helper | Public custom-input API | | --- | --- | | `sdk.setVideoInputEnabled(true)` | `val input = sdk.useCustomInput()` while stopped | | `sdk.addVideoFrame(bitmap, timestampUs)` | `input.sendFrame(bitmap, timestampUs)`; handle the returned result | | `sdk.setVideoInputEnabled(false)` | Stop, then call `sdk.useCamera(com.presagetech.smartspectra.CameraSelection.Default)` | If you retain the testing helpers, enable video input while stopped and await `start()` before submitting frames. The helpers now throw `SmartSpectraException` on invalid source selection or a rejected frame; they no longer silently discard these errors. The public handle returns `FrameSubmissionResult.Rejected` instead. Two levels of CI coverage, pick per test: 1. **[Video-fed measurement](#option-1-the-video-fed-test)** — a full measurement from a recorded clip, asserting that real readings came out. 2. **[Build-integration smoke](#option-2-the-build-integration-smoke)** — no clip needed; proves the SDK builds, launches, and initializes. ## Option 1: The video-fed test Drive `SmartSpectraSdk.shared` directly, the same way you would for any [headless integration](headless-mode.md), and run it as an instrumented test on an **Android emulator**. Feed the clip, read the `metrics` LiveData, and assert that real readings appeared — a pulse rate and a breathing rate — not their exact values. ```kotlin import android.media.MediaMetadataRetriever import androidx.test.ext.junit.runners.AndroidJUnit4 import androidx.test.platform.app.InstrumentationRegistry import com.presagetech.smartspectra.SmartSpectraConfig import com.presagetech.smartspectra.SmartSpectraSdk import com.presagetech.smartspectra.FrameSubmissionResult import kotlinx.coroutines.runBlocking import org.junit.Assert.assertTrue import org.junit.Test import org.junit.runner.RunWith @RunWith(AndroidJUnit4::class) class VideoMeasurementTest { @Test fun measuresFromRecordedVideo() = runBlocking { val sdk = SmartSpectraSdk.shared sdk.config.apiKey = InstrumentationRegistry.getArguments().getString("smartspectraApiKey").orEmpty() // The default request is breathing-only; ask for cardio too so a // pulse rate can appear. See the metrics guide. sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics val input = sdk.useCustomInput() val retriever = MediaMetadataRetriever() try { sdk.start() // A short clip of a well-lit, mostly still face, bundled in the // test APK's assets (assets are not compressed for .mp4). InstrumentationRegistry.getInstrumentation().context.assets .openFd("face.mp4").use { afd -> retriever.setDataSource(afd.fileDescriptor, afd.startOffset, afd.declaredLength) } val frameCount = retriever.extractMetadata( MediaMetadataRetriever.METADATA_KEY_VIDEO_FRAME_COUNT)!!.toInt() val durationUs = retriever.extractMetadata( MediaMetadataRetriever.METADATA_KEY_DURATION)!!.toLong() * 1_000L val frameIntervalUs = durationUs / frameCount var sawPulse = false var sawBreathing = false for (index in 0 until frameCount) { val frame = retriever.getFrameAtIndex(index) ?: break val result = try { input.sendFrame(frame, index * frameIntervalUs) } finally { frame.recycle() } assertTrue("frame rejected: $result", result is FrameSubmissionResult.Accepted) sdk.metrics.value?.let { m -> if (!sawPulse) sawPulse = m.cardio.pulseRateList.any { it.value > 0f } if (!sawBreathing) sawBreathing = m.breathing.rateList.any { it.value > 0f } } if (sawPulse && sawBreathing) break // Emulators software-render the pipeline: feed no faster than // ~10 wall-clock fps so frames aren't dropped. This is the // delivery rate, not the capture rate — each frame carries its // own recorded timestamp, so the pipeline still sees the clip's // real >=25 fps cadence (see Getting a Good Measurement) and // computed rates are unaffected. Do not confuse this with // lowering the capture frame rate, which would raise // FRAME_RATE_TOO_LOW. Thread.sleep(100) } assertTrue("no pulse reading came out of the recorded clip", sawPulse) assertTrue("no breathing reading came out of the recorded clip", sawBreathing) } finally { try { retriever.release() } finally { // reset() is safe even if start() failed, and preserves custom input. sdk.reset() sdk.useCamera(com.presagetech.smartspectra.CameraSelection.Default) } } } } ``` ## Option 2: The build-integration smoke If you don't have a recorded clip yet (or want a faster job on every push), skip the video calls entirely and keep the check at smoke level — no opt-in needed. This variant runs the normal camera path against the emulator's simulated feed, so it grants the `CAMERA` permission: ```kotlin import android.Manifest import androidx.test.ext.junit.runners.AndroidJUnit4 import androidx.test.platform.app.InstrumentationRegistry import androidx.test.rule.GrantPermissionRule import com.presagetech.smartspectra.SmartSpectraException import com.presagetech.smartspectra.SmartSpectraSdk import kotlinx.coroutines.runBlocking import org.junit.Rule import org.junit.Test import org.junit.runner.RunWith @RunWith(AndroidJUnit4::class) class HeadlessSmokeTest { @get:Rule val cameraPermission: GrantPermissionRule = GrantPermissionRule.grant(Manifest.permission.CAMERA) @Test fun sdkInitializesHeadless() = runBlocking { val sdk = SmartSpectraSdk.shared sdk.config.apiKey = InstrumentationRegistry.getArguments().getString("smartspectraApiKey").orEmpty() try { sdk.start() sdk.stop() } catch (e: SmartSpectraException) { println("SmartSpectra reported: ${e.message}") } } } ``` The emulator's simulated camera feed has no real face in it, so don't assert on a measurement result here: `start()` returning at all — whether it succeeds or throws a typed `SmartSpectraException` — is the smoke signal that the SDK built, launched, and initialized correctly end to end. ## The recorded video Supply your own short clip and keep it in your test assets: - Around **30–60 seconds** of a **well-lit, mostly still face**, framed like a real measurement — long enough for the pipeline to compute rates (a measurement runs about 30 seconds); a clip of only a few seconds won't produce readings. - Any container/codec your decoder handles; MP4 (H.264) with `MediaMetadataRetriever` is a safe choice. - Bitmaps are converted to `ARGB_8888` internally when needed. - Timestamps are **microseconds**, strictly increasing, on one time base for the whole session — derive them from the clip (`index * frameIntervalUs`, or `MediaExtractor` sample times). See [Android Metrics](metrics.md) for which metrics to request and how to read them. ## A CI pipeline, in general terms 1. **Expose the API key** as a job secret and pass it to the test as an instrumentation argument. 2. **Run the instrumented test** on an emulator — the runner needs hardware acceleration (KVM on Linux) for the emulator to boot in CI. 3. **Fail the job** if the test APK doesn't build or the test fails. A minimal, provider-neutral sketch (GitHub Actions) using [Gradle Managed Devices](https://developer.android.com/studio/test/gradle-managed-devices), which provisions and boots the emulator headlessly for you. Declare the device in your app module: ```kotlin // build.gradle.kts android { testOptions { managedDevices { localDevices { create("headlessVideo") { device = "Pixel 8" apiLevel = 34 systemImageSource = "aosp-atd" } } } } } ``` Then run it in CI: ```yaml name: smartspectra-android-headless-video on: [push] jobs: headless: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-java@v4 with: distribution: temurin java-version: 17 - name: Enable KVM for the emulator run: | echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \ | sudo tee /etc/udev/rules.d/99-kvm4all.rules sudo udevadm control --reload-rules sudo udevadm trigger --name-match=kvm - name: Video-fed measurement test run: | ./gradlew headlessVideoDebugAndroidTest \ -Pandroid.testInstrumentationRunnerArguments.smartspectraApiKey="${{ secrets.SMARTSPECTRA_API_KEY }}" ``` ## Limitations - **No offline mode.** Like every SmartSpectra SDK, a measurement authenticates against the SmartSpectra service, so the runner needs network access. - **Don't mix inputs.** Within one session, feed frames exclusively via the custom-input handle. Stop before selecting the camera again. - **Smoke, not accuracy.** A recorded-clip run confirms the integration and model pipeline end to end; it is not an accuracy benchmark. ## LLM Insights on Android Source: https://presagetech.com/docs/smartspectra-sdk/android/llm-insights --- title: LLM Insights on Android description: Ask natural-language questions about a measurement and receive LLM Insights through the SmartSpectra Android SDK, alongside the metrics stream. sidebarTitle: LLM Insights --- # Android LLM Insights Platform-specific usage for the Android SDK. For what LLM Insights are, the request/response model, required metrics, and the privacy notice, see the [LLM Insights overview](../../docs/llm-insights/index.md). The insight API lives on the `SmartSpectraSdk` singleton (`SmartSpectraSdk.shared`). ## Enable the required metrics Insights summarize the buffered vitals, so breathing (the default set) and cardio must both be active. Assign both bundles to `requestedMetrics`: ```kotlin sdk.config.requestedMetrics = buildList { addAll(SmartSpectraConfig.breathingMetrics) addAll(SmartSpectraConfig.cardioMetrics) } ``` `cardioMetrics` includes `MetricType.ARTERIAL_PRESSURE_TRACE`, which drives the on-screen pulse waveform. When `requestedMetrics` is unset the SDK measures breathing only. ## Receive responses Insights are delivered through a `LiveData`, observed on the **main thread**. It carries **both** the auto-fired periodic vitals insights and on-demand responses: ```kotlin public val insight: LiveData ``` ```kotlin sdk.insight.observe(viewLifecycleOwner) { insight -> if (insight == null) return@observe if (insight.requestId != pendingRequestId) return@observe // correlate the reply val text = when { insight.hasAnalysis() -> insight.analysis else -> getString(R.string.insights_chat_error) } showMessage(text) } ``` Match `insight.requestId` against the value returned by `requestInsight` to tell on-demand replies apart from auto-fired vitals (which won't match a request you made). Every insight is currently delivered with `type` == `INSIGHT_TYPE_VITALS`, so correlate on `requestId`, not `type`. ## Request an insight Call `requestInsight` on a running session. It returns the request ID used to correlate the asynchronous response: ```kotlin public fun requestInsight(text: String): Int ``` ```kotlin pendingRequestId = runCatching { sdk.requestInsight("Summarize my current vital signs and flag anything unusual.") } .onFailure { /* dispatch failed */ } .getOrNull() ``` The prompt is combined with the latest buffered metrics when they exist, otherwise sent prompt-only. ## Reading the Insight `Insight` is the generated proto type (`com.presagetech.smartspectra.proto.InsightsProto.Insight`). Branch on `insight.hasAnalysis()` / `insight.hasError()` (exactly one is set), read the text with `insight.analysis` / `insight.error`, and correlate with `insight.requestId`. (Every insight is currently typed `INSIGHT_TYPE_VITALS`, so use `requestId`, not `type`, to distinguish replies.) Full field documentation is in [Data Types → Insight](../../docs/data-types.md#insight). The first auto-fired insight arrives once pulse rate has a stable reading, typically about 15 seconds after the session starts; further snapshots follow as breathing rate and HRV stabilise, then every 60 seconds. Allow at least 15 seconds of valid measurement before an on-demand request can be grounded in the user's physiology. The service may return no insight for a given request, in which case the observer is not called. ## See also - [LLM Insights overview](../../docs/llm-insights/index.md) - [Android API reference](api-reference.md) - [Data Types](../../docs/data-types.md) ## Configuring Metrics on Android Source: https://presagetech.com/docs/smartspectra-sdk/android/metrics --- title: Configuring Metrics on Android description: Request pulse and breathing rates, HRV and EDA metrics from the SmartSpectra Android SDK, and read LiveData results. sidebarTitle: Configuring Metrics --- # Configuring Android Metrics By default, Android measurements request the breathing metric set. Combine it with `SmartSpectraConfig.cardioMetrics` when your app needs pulse rate, arterial-pressure trace, and HRV. ## Breathing and Pulse ### Request Metrics Request the breathing and cardio bundles before calling `start()`: ```kotlin import com.presagetech.smartspectra.proto.MetricTypesProto.MetricType import com.presagetech.smartspectra.SmartSpectraConfig import com.presagetech.smartspectra.SmartSpectraSdk val sdk = SmartSpectraSdk.shared sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics ``` ### Read Metrics Read the latest breathing and pulse samples from `SmartSpectraSdk.metrics`: ```kotlin sdk.metrics.observe(viewLifecycleOwner) { metrics -> val breathingRate = metrics?.breathing?.rateList?.lastOrNull()?.value val chestTrace = metrics?.breathing?.upperTraceList?.lastOrNull()?.value val abdomenTrace = metrics?.breathing?.lowerTraceList?.lastOrNull()?.value val pulseRate = metrics?.cardio?.pulseRateList?.lastOrNull()?.value } ``` Set `requestedMetrics = null` to return to the default breathing-only set. `cardioMetrics` contains `PULSE_RATE`, `ARTERIAL_PRESSURE_TRACE`, and `HRV`. Cardio fields are empty unless you request a cardio metric. ## Metric Update Patterns `SmartSpectraSdk.metrics` publishes the latest SDK metrics payload. Each payload contains the samples that became available since the previous metrics update; it is not guaranteed to contain every requested field every time. | Metric category | Examples | Expected cadence | Empty/null behavior | | --- | --- | --- | --- | | Peak/event-driven rate metrics | `breathing.rateList`, `cardio.pulseRateList`, `cardio.hrvList` | Updated when a new physiological event, cycle, or analysis window produces a value | Lists may be empty between valid updates during active capture | | Frame-driven metrics | `face.expressionList`, `face.landmarksList`, `face.blinkingList`, `face.talkingList`, breathing trace lists | Updated near device frame cadence, with SDK callbacks rate-limited to about 30 Hz | Usually present more continuously when the metric is enabled and the input signal is valid | For example, `metrics?.cardio?.pulseRateList?.lastOrNull()?.value` and `metrics?.breathing?.rateList?.lastOrNull()?.value` may temporarily evaluate to `null` between valid updates. This is expected and does not mean capture stopped or the metric was disabled. By contrast, face expression samples are frame-driven, so `metrics?.face?.expressionList?.lastOrNull()` can appear continuously while face metrics are enabled and the face signal is valid. Recommended UI handling: - Keep the last valid rate sample in app state and update it only when the list contains a new sample. - Show an initial loading or placeholder state until the first valid sample arrives. - Do not overwrite a displayed pulse rate or breathing rate with `null` only because one metrics payload has no new sample. - Clear retained values when a capture session starts, stops, or when your app intentionally changes the requested metric set. - Prefer sample timestamps, and `stable` when present, to decide whether a retained value is fresh enough for your UI. ```kotlin var lastPulseRate: Double? = null sdk.metrics.observe(viewLifecycleOwner) { metrics -> val pulse = metrics ?.cardio ?.pulseRateList ?.lastOrNull { it.timestamp > 0 } ?.value if (pulse != null) { lastPulseRate = pulse pulseRateLabel.text = String.format(Locale.ROOT, "%.0f bpm", pulse) } else if (lastPulseRate == null) { pulseRateLabel.text = "-- bpm" } } ``` ## Advanced Request additional metrics only when your app needs them: ```kotlin sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics + listOf( MetricType.EDA_TRACE, MetricType.FACE_LANDMARKS, MetricType.BLINKING, MetricType.TALKING, MetricType.EXPRESSIONS, ) ``` Read the advanced fields from the same metrics stream: ```kotlin sdk.metrics.observe(viewLifecycleOwner) { metrics -> val pressureTrace = metrics?.cardio?.arterialPressureTraceList?.lastOrNull()?.value val hrvRmssd = metrics?.cardio?.hrvList?.lastOrNull()?.rmssd val edaTrace = metrics?.eda?.traceList?.lastOrNull()?.value val faceLandmarks = metrics?.face?.landmarksList?.lastOrNull()?.valueList val blinking = metrics?.face?.blinkingList?.lastOrNull()?.detected val talking = metrics?.face?.talkingList?.lastOrNull()?.detected val expression = metrics?.face?.expressionList?.lastOrNull() } ``` ### Advanced Payload Classes Android uses the generated protobuf classes. Requested advanced metrics populate these fields: ```kotlin Metrics { breathing: Breathing eda: Eda face: Face cardio: Cardio } Cardio { pulseRateList: List arterialPressureTraceList: List hrvList: List } Hrv { rmssd: Double meanNn: Double sdnn: Double baevsky: Double timestamp: Long confidence: Float stable: Boolean } Eda { traceList: List } Face { landmarksList: List blinkingList: List talkingList: List expressionList: List } ``` EDA Proxy may take longer to produce its first sample than breathing or cardio outputs. See [Data Types](../../docs/data-types.md) for the complete protobuf schema. ## Android Migration Guide Source: https://presagetech.com/docs/smartspectra-sdk/android/migration-guide --- title: Android Migration Guide description: "Migrate the SmartSpectra Android SDK to typed camera selection, public custom video input, specific usage errors, and other release-by-release API changes." sidebarTitle: Migration Guide --- # SmartSpectra Android SDK Migration Guide > Applies to SmartSpectra Android SDK v3.x. > Migrating from a v3.0 release-candidate prior to rc.12, or from v2.x. ## Android SDK v3.4.0 Migration ### Camera selection Camera selection is opt-in. Existing applications retain the front-camera default and the behavior of `config.cameraPosition`, including active-camera changes. That property, `CameraPosition`, and the original `useCamera()` method remain available, with deprecation warnings. To adopt discovery and strict selection, call `useCamera(selection)` while stopped: ```kotlin import com.presagetech.smartspectra.CameraSelection import com.presagetech.smartspectra.SmartSpectraSdk // From a coroutine. Discovery needs only application context, not an SDK instance: val cameras = SmartSpectraSdk.availableCameras(context) // Select an input while stopped; useCamera does not open a camera. sdk.useCamera(CameraSelection.Default) sdk.useCamera(CameraSelection.Front) sdk.useCamera(CameraSelection.Back) cameras.firstOrNull()?.let { camera -> sdk.useCamera(CameraSelection.ById(camera.id)) } ``` Discovery returns `CameraInfo(id, name, facing, lensType)`. IDs are opaque and device-local; `name` is optional and currently null on Android. Facing is `FRONT`, `BACK`, or `UNKNOWN`; external cameras report `UNKNOWN`. Discovery does not prompt for permission or start capture. Visibility depends on platform permissions, and a snapshot does not guarantee that capture will succeed later. Discovery failure throws `SmartSpectraException` with `INPUT_UNAVAILABLE`. `lensType` is `WIDE_ANGLE`, `ULTRA_WIDE`, `TELEPHOTO`, or `UNKNOWN`. Android classification is approximate, relative to the default camera with the same facing. Missing or ambiguous metadata and logical cameras combining multiple lenses report `UNKNOWN`. Classification does not describe digital zoom or change camera selection; continue selecting by `id`. `Default` explicitly opts into preferring front, then the first camera in discovery order. Explicit `Front`, `Back`, and `ById` requests never fall back to a different facing or ID. Camera resolution occurs at startup; an unavailable selection fails `start()` with `INPUT_UNAVAILABLE`. After stopping, select an available camera and start again. `useCamera(selection)` throws `INVALID_STATE` during startup, processing, stopping, or reset, and `CONFIGURATION_FAILED` for an empty ID. A rejected request preserves the previous source. To change cameras, await `stop()`, call `useCamera(selection)`, and call `start()` again. Stop/start/reset retain the selected camera until you choose another source. Typed selection does not switch a running camera implicitly. Once selected, changes to the deprecated `cameraPosition` property do not override it. Calling the deprecated `useCamera()` returns to that property's selection behavior. The demo app's **Select camera** control lists discovered cameras with their name, facing, lens type, and ID. Choose a camera for the next measurement, **Default** to restore automatic selection, or **Refresh** to discover connected cameras again. The checkup, headless, and capture screens share this picker; selection is disabled while processing. Discovery and selection errors appear in the picker, and capture errors appear on the measurement screen. The minimal app also provides a **Select camera** control, with Default and Refresh actions in both portrait and landscape layouts. After a failed Start, **Select camera** remains available in both apps. Opening it awaits `stop()` to recover the SDK from `ERROR` before allowing a new selection. ### Use the public custom-input API Apps that supply frames now use the public `useCustomInput()` API. Select custom input while the SDK is stopped, retain the returned `CustomInput`, await `start()`, and submit frames from a worker thread: ```kotlin val input = sdk.useCustomInput() sdk.start() val result = input.sendFrame(imageProxy) if (result is FrameSubmissionResult.Rejected) { // Handle result.error. } ``` Existing test integrations should replace the `@SmartSpectraTestingApi` `setVideoInputEnabled(true)` and `addVideoFrame(...)` calls with `useCustomInput()` and `CustomInput.sendFrame(...)`. The public handle supports `ImageProxy`, `Bitmap`, and `VideoFrame`. It survives stop/start/reset; selecting another source invalidates it. SDK camera permission is not required for custom input, but the host app remains responsible for permissions required by its own capture code. ### Check custom-input timestamp rejections in the submission result `CustomInput.sendFrame()` returns `FrameSubmissionResult.Rejected` with `NON_MONOTONIC_TIMESTAMP` or `TIMESTAMP_GAP` when it rejects a timestamp. These rejections leave the session active and do not publish a global `sdk.error`. Check each submission result even if you also observe `sdk.error`. A rejected timestamp does not advance the accepted timeline. Correct the next timestamp, or stop and start the SDK to begin a fresh timeline after an interruption. ### Usage failures now report specific errors Usage entitlement checking continues throughout a measurement as before. What changes in v3.4.0 is how a terminal usage failure is reported. Observe `sdk.error` and handle: - `AUTHENTICATION_FAILED` for rejected credentials - `CREDIT_EXHAUSTED` for an explicit quota denial - `NETWORK_ERROR` when entitlement cannot be refreshed in time The SDK also includes short measurements and the final partial interval before `stop()` in usage reporting. No public API changes are required. ## Android SDK v3.3.0 Migration ### Default log verbosity is now warnings and errors only The SDK previously emitted informational logcat chatter by default. From v3.3.0 the default level is `SmartSpectraLogLevel.WARNING`, covering both the SDK's Kotlin-side logging and the native engine. If you relied on the informational output, restore it via `SmartSpectraConfig.logLevel`: ```kotlin val config = SmartSpectraConfig().apply { logLevel = SmartSpectraLogLevel.INFO } SmartSpectraSdk.initialize(context, config) ``` ## Edge Metrics Migration The `metricsBuffer` pathway has been removed. Android apps should now read all vitals data from `metrics`. ### What Changed - `MetricsBuffer`-based APIs were removed - On-device `Metrics` is now the single vitals data source ### Field Mappings | Old (`metricsBuffer`) | New (`metrics`) | | --------------------- | --------------- | | `pulse.rateList` | `cardio.pulseRateList` | | `pulse.traceList` | `cardio.arterialPressureTraceList` | | `breathing.rateList` | `breathing.rateList` | | `breathing.upperTraceList` | `breathing.upperTraceList` | | `breathing.lowerTraceList` | `breathing.lowerTraceList` | | `face` | `face` | ### Important Cardio fields now require explicit opt-in. If your app displays pulse rate, arterial pressure trace, or HRV, enable cardio metrics explicitly. ```kotlin import com.presagetech.smartspectra.SmartSpectraConfig val sdk = SmartSpectraSdk.shared sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics ``` **Removed:** - `sdk.metricsBuffer` - `setMetricsBufferObserver()` - `setMetricsBuffer()` - `clearMetricsBuffer()` **Replace with:** ```kotlin // Before: smartSpectraSdk.metricsBuffer.observe(viewLifecycleOwner) { buffer -> val pulse = buffer?.pulse?.rateList?.lastOrNull()?.value } ``` ```kotlin // After: smartSpectraSdk.metrics.observe(viewLifecycleOwner) { metrics -> val pulse = metrics?.cardio?.pulseRateList?.lastOrNull()?.value } ``` ## Protobuf Java/Kotlin Package Rename The generated protobuf classes moved from the old Physiology Java package to the SmartSpectra package: | Before | After | | ------ | ----- | | `com.presage.physiology.proto.*` | `com.presagetech.smartspectra.proto.*` | Update imports for generated types such as `Metrics`, `MetricType`, `StatusCode`, and `Insight`: ```kotlin // Before: import com.presage.physiology.proto.MetricsProto.Metrics import com.presage.physiology.proto.MetricTypesProto.MetricType // After: import com.presagetech.smartspectra.proto.MetricsProto.Metrics import com.presagetech.smartspectra.proto.MetricTypesProto.MetricType ``` The proto package also changed from `presage.physiology` to `presagetech.smartspectra`. The binary wire format is unchanged because field numbers did not change, but `Any.type_url`, JSON `@type` values, and descriptor lookups use the fully-qualified message name. Regenerate or re-emit any persisted `Any`-wrapped or JSON-typed payloads with the new SDK before mixing old and new builds. ## Processing Status Migration `SmartSpectraSdk` now exposes the aligned lifecycle model through `processingStatus`. ### New Enum Values - `IDLE` - `STARTING` - `RUNNING` - `STOPPING` - `ERROR` ### Enum Mapping | Previous value | New value | | -------------- | --------- | | `IDLE` | `IDLE` | | `COUNTDOWN` | `STARTING` | | `RUNNING` | `RUNNING` | | `PREPROCESSED` | `RUNNING` | | `DONE` | `RUNNING` | | `DISABLE` | `IDLE` | | `ERROR` | `ERROR` | ### Semantic Updates - `RUNNING` means the pipeline is active and measurement is running - `STOPPING` is a transient shutdown state on the path back to `IDLE` ```kotlin // After: SmartSpectraSdk.shared.processingStatus.observe(viewLifecycleOwner) { status -> when (status) { ProcessingStatus.IDLE -> showIdleUi() ProcessingStatus.STARTING -> showLoadingUi() ProcessingStatus.RUNNING -> showRecordingUi() ProcessingStatus.STOPPING -> showStoppingUi() ProcessingStatus.ERROR -> showErrorUi() } } ``` ## UI Surface Removal The screening UI (Activities, Fragments, custom Views) no longer ships with the SDK. The SDK is now a pure data/lifecycle plane; integrators own all UI. ### What was removed from the SDK - `SmartSpectraView`, `SmartSpectraButton`, `SmartSpectraResultView` - `SmartSpectraActivity`, `OnboardingTutorialActivity` - All `ui/screening/...` Fragments and custom Views (`CameraProcessFragment`, `ConfigurationErrorFragment`, `PermissionsRequestFragment`, `ScreeningPlotView`, `ExpressionStatusView`, `FaceMetricsStatusView`) - `ui/summary/UploadingFragment` - All UI layouts, drawables, tutorial assets, and screening strings - The two SDK manifest `` declarations - LiveDatas / helpers removed from `SmartSpectraSdk`: `timeLeft`, `hintText`, `roundedFps`, `attachPreview`, `detachPreview`, `flipCamera`, `resetMetrics`. None were public, and the only in-tree consumers were the relocated UI files. The internal `SmartSpectraVitalsProcessor` keeps `timeLeft` / `hintText` for instrumented tests; the SDK class no longer mirrors them. - Config flags removed from `SmartSpectraConfig` (all were `internal`): `showFps`, `showOutputFps`, `showControlsInScreeningView` - Derived helpers `cardioMeasurementsEnabled`, `facialExpressionEnabled`, `edaMeasurementsEnabled` (replicate app-side from `requestedMetrics` if needed) ### Migration paths #### Option A — Copy the reference UI from the demo-app sample The sample at `android/samples/demo-app` (in the public [Presage-Security/SmartSpectra](https://github.com/Presage-Security/SmartSpectra) repo) mirrors the SDK's old package layout. Copy the relevant files into your app and adjust `R.*` references: ```text samples/demo-app/src/main/java/com/presagetech/smartspectra_example/ SmartSpectraView.kt SmartSpectraButton.kt SmartSpectraResultView.kt ui/ OnboardingTutorialActivity.kt SmartSpectraActivity.kt AmbientLightBrightnessController.kt screening/... summary/... samples/demo-app/src/main/res/ (layouts, drawables, strings) samples/demo-app/src/main/AndroidManifest.xml (activity declarations) ``` #### Option B — Build your own UI against `SmartSpectraSdk` Recommended for new integrations. The public surface gives you everything you need: ```kotlin val sdk = SmartSpectraSdk.shared sdk.config.apiKey = "YOUR_API_KEY" sdk.useCamera(CameraSelection.Front) sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics // Drive the lifecycle from your own button. lifecycleScope.launch { sdk.start() } lifecycleScope.launch { sdk.stop() } // Render preview frames from a public LiveData. sdk.imageOutput.observe(this) { bitmap -> findViewById(R.id.preview).setImageBitmap(bitmap) } // React to status / metrics / errors. sdk.processingStatus.observe(this) { renderStatus(it) } sdk.metrics.observe(this) { renderMetrics(it) } sdk.error.observe(this) { renderError(it) } ``` See `samples/minimal-app` for the smallest end-to-end headless example, and `samples/demo-app` for a richer reference UI built on top of the public surface. ### Lost reference UX Mid-session camera flip, hint-text overlays, FPS overlay, the configuration-error route, the bundled onboarding tutorial, and the in-app EULA modals are no longer part of the SDK. The demo-app sample keeps a compatible reference UX; integrators can copy it or roll their own. ## `start()` clears observable state `SmartSpectraSdk.start()` now resets the observable LiveData surface (`metrics`, `imageOutput`, `validationStatus`, `error`, `insight`) to `null` before kicking off processing. Previously, the screening UI inside the SDK called an internal `resetMetrics()` from its `onResume`. With the UI moved out and that internal hook removed, callers re-entering a screening would see the previous session's values flash on screen until fresh metrics arrived. Resetting at the lifecycle boundary (`start()`) is the right place — it keeps the contract uniform regardless of whether the host app toggles processing in place or navigates away and back. If your app accumulates derived state from these LiveData (chart buffers, latest-rate caches, etc.), make sure your own clear logic runs on `start()` or `processingStatus == STARTING` so it stays in sync with the SDK's reset. The `samples/demo-app` and `samples/minimal-app` already follow this pattern. ## Public surface narrowing for cross-platform parity A small batch of Android-only or asymmetric-with-iOS public API is now `internal`. None had a non-cosmetic reason to be public; the changes bring the two SDKs closer to a shared contract. ### Metrics observers → requested metric bundles Older Android releases did not expose public requested-metric selection. Apps attached `setMetricsBufferObserver` for pulse and breathing output, and `setEdgeMetricsObserver` for edge metrics such as dense face landmarks. The current SDK uses `requestedMetrics` directly. Public `@JvmField` bundles on `SmartSpectraConfig` include `breathingMetrics`, `cardioMetrics`, `faceMetrics`, and `edaMetrics`. The `Metrics` suffix matches the C++ SDK's `SmartSpectraConfig::CardioMetrics()` etc.; the `SmartSpectraConfig` namespace matches the iOS Swift surface. ```kotlin // Before sdk.setMetricsBufferObserver { metricsBuffer -> val pulse = metricsBuffer.pulse.rateList.lastOrNull()?.value } // After sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics ``` ### SDK permission screen → host-app responsibility Older integrations usually relied on `SmartSpectraView` / `SmartSpectraButton` to present the camera permission flow. Host apps now drive the runtime camera prompt themselves — the modern pattern is `ActivityResultLauncher`. The SDK still publishes a `SmartSpectraError(code = INPUT_UNAVAILABLE, retryable = true)` to `sdk.error` when the validator flags a missing manifest declaration or when `start()` fails because the runtime permission is denied. ```kotlin // Before setContentView(R.layout.activity_main) val smartSpectraView = findViewById(R.id.smart_spectra_view) // After private val cameraPermissionLauncher = registerForActivityResult( ActivityResultContracts.RequestPermission(), ) { granted -> if (granted) startProcessing() } private fun startProcessing() { if (ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED ) { cameraPermissionLauncher.launch(Manifest.permission.CAMERA) return } lifecycleScope.launch { runCatching { sdk.start() } } } ``` ### ABI checks → SDK-managed error Older SDK UI checked ABI support before showing the measurement view. The current SDK owns that check and publishes `SmartSpectraError(code = CONFIGURATION_FAILED, retryable = false)` to `sdk.error` when initialization detects an unsupported ABI. ```kotlin // Before SmartSpectraView(context, attrs) // SDK view handled unsupported ABI UI // After sdk.error.observe(this) { error -> if (error?.code == SmartSpectraError.Code.CONFIGURATION_FAILED) { showUnsupportedDeviceUi() } } ``` ## Option 1: API Key on Android Source: https://presagetech.com/docs/smartspectra-sdk/android/option-1-api-key --- title: "Option 1: API Key on Android" description: "The fastest SmartSpectra Android setup: add the Gradle dependency, drop in an API key, and run a first measurement on a phone." sidebarTitle: "Option 1: API Key" --- # QuickStart - API Key Use this if you want the fastest manual path. ## What you will change manually You will touch exactly these things: 1. The app Gradle repositories and dependencies 2. `app/src/main/java/com/example/coolvitals/MainActivity.kt` You do not need to create XML layouts or additional Kotlin files. ## Requirements Use a compatible Android toolchain before adding the SDK: - `minSdk` **28** or later - `compileSdk` **36.1** or later - Android Gradle Plugin (AGP) **8.10.1** or later; use an AGP 9.x release when your project uses Gradle 9 - Kotlin **2.2.x** These requirements apply to SmartSpectra 3.2.x. New Android Studio projects can usually satisfy them by updating the generated project-level plugin versions before the first Gradle sync. ## Result you should get At the end, the app should show: - live camera preview - pulse rate, breathing rate, HRV RMSSD, and expression cards - arterial pressure waveform - chest and abdomen breathing waveforms - status text and a start/stop button - one portrait screen with no scrolling ![SmartSpectra Android quickstart demo](../media/android-quickstart.gif) ## Register for your free API Key ### Create an Account 1. Navigate to the Presage [Developer Admin Portal Registration](https://physiology.presagetech.com/auth/register) 2. Click **Register** and fill in your email, password, and other required fields. 3. Check your email for a confirmation link and follow it to activate your account. ### Log In 1. Go to the Presage Developer Admin Portal [Login](https://physiology.presagetech.com/auth/login) 2. Enter your email and password, then click **Submit**. 3. After successful login you will be redirected to your Portal page, where you can manage your API key. ## Step 1 — Create the project In Android Studio, create a new Android app project: 1. Select `File` → `New` → `New Project...` 2. Choose `Empty Activity` 3. Set `Name` to `Cool Vitals` 4. Set `Package name` to `com.example.coolvitals` 5. Set `Language` to `Kotlin` 6. Set `Minimum SDK` to `API 28` or newer 7. Finish creating the project If you already created the project, open it instead. Open your Android app project in Android Studio. ## Step 2 — Add repositories In your project-level `settings.gradle.kts`, make sure the app can resolve AndroidX and SmartSpectra artifacts: ```kotlin dependencyResolutionManagement { repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS) repositories { google() mavenCentral() // Required only for versions ending in -SNAPSHOT. maven { url = uri("https://central.sonatype.com/repository/maven-snapshots/") mavenContent { snapshotsOnly() } } } } ``` ## Step 3 — Add app dependencies In `app/build.gradle.kts`, keep the dependencies generated by the `Empty Activity` template and add only these lines to the existing `dependencies` block: ```kotlin dependencies { implementation("androidx.activity:activity-ktx:1.8.1") implementation("androidx.core:core-ktx:1.18.0") implementation("androidx.lifecycle:lifecycle-runtime-ktx:2.10.0") implementation("androidx.camera:camera-view:1.6.0") implementation("com.presagetech:smartspectra:3.4.0") } ``` `androidx.activity:activity-ktx` supplies `ComponentActivity` and `registerForActivityResult`, both used by the complete activity below. The version above is the release these docs describe. To use a different one, browse the published versions on [Maven Central](https://central.sonatype.com/artifact/com.presagetech/smartspectra/versions). Manual check: - Gradle sync succeeds - AndroidX imports resolve: `PreviewView`, `ContextCompat`, and `lifecycleScope` - SmartSpectra imports resolve: `SmartSpectraSdk` and `MetricType` ## Step 4 — Declare manifest permissions Add these permissions directly inside the root `` element in `app/src/main/AndroidManifest.xml`, before ``: ```xml ``` ## Step 5 — Replace `MainActivity.kt` In Android Studio: 1. Open `app/src/main/java/com/example/coolvitals/MainActivity.kt` 2. Delete everything in the file 3. Paste the full file below 4. Replace `YOUR_API_KEY` with your real API key Paste this entire file: ```kotlin package com.example.coolvitals import android.Manifest import android.content.Context import android.content.pm.PackageManager import android.content.res.ColorStateList import android.graphics.Canvas import android.graphics.Color import android.graphics.Paint import android.graphics.Path import android.graphics.drawable.GradientDrawable import android.os.Bundle import android.view.Gravity import android.view.View import android.view.ViewGroup import android.widget.Button import android.widget.FrameLayout import android.widget.LinearLayout import android.widget.TextView import androidx.activity.ComponentActivity import androidx.activity.result.contract.ActivityResultContracts import androidx.camera.view.PreviewView import androidx.core.content.ContextCompat import androidx.core.view.ViewCompat import androidx.core.view.WindowCompat import androidx.core.view.WindowInsetsCompat import androidx.lifecycle.lifecycleScope import com.presagetech.smartspectra.CameraSelection import com.presagetech.smartspectra.ProcessingStatus import com.presagetech.smartspectra.SmartSpectraConfig import com.presagetech.smartspectra.SmartSpectraSdk import com.presagetech.smartspectra.proto.MetricTypesProto.MetricType import com.presagetech.smartspectra.proto.MetricsProto.ExpressionType import kotlin.math.max import kotlin.math.min import kotlin.math.roundToInt import kotlinx.coroutines.launch class MainActivity : ComponentActivity() { private companion object { const val API_KEY = "YOUR_API_KEY" val TEXT_PRIMARY = Color.WHITE val TEXT_MUTED = 0x80FFFFFF.toInt() val CARD_OVERLAY = 0xE61A2233.toInt() val CORAL = 0xFFFF6B6B.toInt() val TEAL = 0xFF4FCCC4.toInt() val VIOLET = 0xFFA58CF9.toInt() val MINT = 0xFF6EE7B7.toInt() val BLUE = 0xFF60A5FA.toInt() val AMBER = 0xFFFBBF24.toInt() } private val sdk by lazy { SmartSpectraSdk.shared } private lateinit var heartRateLabel: TextView private lateinit var expressionLabel: TextView private lateinit var breathingRateLabel: TextView private lateinit var hrvLabel: TextView private lateinit var chestGraphView: SignalGraphView private lateinit var abdomenGraphView: SignalGraphView private lateinit var arterialPressureGraphView: SignalGraphView private lateinit var statusLabel: TextView private lateinit var validationLabel: TextView private lateinit var toggleButton: Button private var latestChestTimestamp: Long = Long.MIN_VALUE private var latestAbdomenTimestamp: Long = Long.MIN_VALUE private var latestPressureTimestamp: Long = Long.MIN_VALUE private val cameraPermissionLauncher = registerForActivityResult( ActivityResultContracts.RequestPermission(), ) { granted -> if (granted) { startProcessing() } else { statusLabel.text = "Status: Camera required" } } override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) WindowCompat.setDecorFitsSystemWindows(window, false) sdk.config.apiKey = API_KEY sdk.config.imageOutputEnabled = false sdk.useCamera(CameraSelection.Front) sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics + listOf(MetricType.EXPRESSIONS) buildUi() bindSdk() resetMeasurementUi() statusLabel.text = "Status: Idle" } override fun onPause() { super.onPause() lifecycleScope.launch { when (sdk.processingStatus.value) { ProcessingStatus.RUNNING, ProcessingStatus.STARTING, ProcessingStatus.STOPPING, -> runCatching { sdk.stop() } else -> Unit } } } private fun bindSdk() { sdk.processingStatus.observe(this) { updateProcessingStatus(it) } sdk.validationStatus.observe(this) { status -> validationLabel.text = "Validation: ${status?.code?.name?.replace('_', ' ') ?: "--"}" } sdk.error.observe(this) { error -> if (error != null) { statusLabel.text = "Error: ${error.message ?: "Unknown"}" } } sdk.metrics.observe(this) { metrics -> if (metrics == null) return@observe if (metrics.hasCardio()) { val pulse = metrics.cardio.pulseRateList .lastOrNull { it.timestamp > 0 } ?.value ?.roundToInt() if (pulse != null) { heartRateLabel.text = "$pulse bpm" } metrics.cardio.arterialPressureTraceList.forEach { sample -> if (sample.timestamp > latestPressureTimestamp) { latestPressureTimestamp = sample.timestamp arterialPressureGraphView.appendValue(sample.value) } } metrics.cardio.hrvList.lastOrNull()?.rmssd?.let { rmssd -> if (rmssd > 0) { hrvLabel.text = "${(rmssd * 10).roundToInt() / 10.0} ms" } } } if (metrics.hasBreathing()) { if (metrics.breathing.rateCount > 0) { val breathingRate = metrics.breathing.rateList.last().value.roundToInt() breathingRateLabel.text = "$breathingRate bpm" } metrics.breathing.upperTraceList.forEach { sample -> if (sample.timestamp > latestChestTimestamp) { latestChestTimestamp = sample.timestamp chestGraphView.appendValue(sample.value) } } metrics.breathing.lowerTraceList.forEach { sample -> if (sample.timestamp > latestAbdomenTimestamp) { latestAbdomenTimestamp = sample.timestamp abdomenGraphView.appendValue(sample.value) } } } if (metrics.hasFace()) { val expression = metrics.face.expressionList.lastOrNull() ?: return@observe val topScore = expression.scoresList .filter { it.confidence > 0f } .maxByOrNull { it.confidence } ?: return@observe val expressionName = topScore.type.expressionName() ?: return@observe expressionLabel.text = "%-8.8s %3d%%".format(expressionName, topScore.confidence.roundToInt()) } } } private fun buildUi() { val horizontalInset = dp(14) val topInsetSpacing = dp(8) val bottomInsetSpacing = dp(12) val root = FrameLayout(this).apply { background = GradientDrawable( GradientDrawable.Orientation.TOP_BOTTOM, intArrayOf(0xFF0B1020.toInt(), 0xFF05070C.toInt()), ) } val previewView = PreviewView(this).apply { contentDescription = "SmartSpectra preview output" setBackgroundColor(Color.BLACK) scaleType = PreviewView.ScaleType.FILL_CENTER implementationMode = PreviewView.ImplementationMode.COMPATIBLE } sdk.config.previewSurfaceProvider = previewView.surfaceProvider val previewParams = FrameLayout.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, dp(465), Gravity.TOP, ) root.addView(previewView, previewParams) val previewOverlay = View(this).apply { background = GradientDrawable( GradientDrawable.Orientation.TOP_BOTTOM, intArrayOf(0x33000000, 0x00000000, 0xCC05070C.toInt()), ) } val previewOverlayParams = FrameLayout.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, dp(465), Gravity.TOP, ) root.addView( previewOverlay, previewOverlayParams, ) val topPanel = LinearLayout(this).apply { orientation = LinearLayout.VERTICAL setPadding(horizontalInset, topInsetSpacing, horizontalInset, 0) } root.addView( topPanel, FrameLayout.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.WRAP_CONTENT, Gravity.TOP, ), ) val topRow = horizontalRow() statusLabel = statusPill("Status", "Idle", CORAL) validationLabel = statusPill("Validation", "--", AMBER) toggleButton = Button(this).apply { text = "Start" setAllCaps(false) setTextColor(Color.BLACK) backgroundTintList = ColorStateList.valueOf(Color.WHITE) setOnClickListener { toggleProcessing() } } topRow.addView(statusLabel, weightedParams(endMargin = dp(8))) topRow.addView(validationLabel, weightedParams(endMargin = dp(8))) topRow.addView(toggleButton, LinearLayout.LayoutParams(dp(92), dp(42))) topPanel.addView(topRow) val panel = LinearLayout(this).apply { orientation = LinearLayout.VERTICAL setPadding(horizontalInset, 0, horizontalInset, 0) } val panelParams = FrameLayout.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.WRAP_CONTENT, Gravity.BOTTOM, ) root.addView( panel, panelParams, ) heartRateLabel = textView(sizeSp = 23f, color = CORAL, bold = true) breathingRateLabel = textView(sizeSp = 23f, color = TEAL, bold = true) val rateRow = horizontalRow() rateRow.addView(metricCard("Pulse Rate", heartRateLabel, CORAL), weightedParams(endMargin = dp(8))) rateRow.addView(metricCard("Breathing Rate", breathingRateLabel, TEAL), weightedParams()) panel.addView(rateRow, matchWrapBottomMargin(dp(10))) hrvLabel = textView(sizeSp = 23f, color = TEXT_PRIMARY, bold = true) expressionLabel = textView(sizeSp = 20f, color = TEXT_PRIMARY, bold = true).apply { typeface = android.graphics.Typeface.create(android.graphics.Typeface.MONOSPACE, android.graphics.Typeface.BOLD) includeFontPadding = false } val summaryRow = horizontalRow() summaryRow.addView(metricCard("HRV RMSSD", hrvLabel, MINT), weightedParams(endMargin = dp(8))) summaryRow.addView(metricCard("Expression", expressionLabel, AMBER), weightedParams()) panel.addView(summaryRow, matchWrapBottomMargin(dp(10))) arterialPressureGraphView = SignalGraphView(this, VIOLET) panel.addView(waveformCard("Arterial Pressure", arterialPressureGraphView), matchWrapBottomMargin(dp(10))) chestGraphView = SignalGraphView(this, TEAL) abdomenGraphView = SignalGraphView(this, BLUE) val breathingRow = horizontalRow() breathingRow.addView(waveformCard("Chest Waveform", chestGraphView), weightedParams(endMargin = dp(8))) breathingRow.addView(waveformCard("Abdomen Waveform", abdomenGraphView), weightedParams()) panel.addView(breathingRow, matchHeight(dp(126))) ViewCompat.setOnApplyWindowInsetsListener(root) { _, windowInsets -> val systemBars = windowInsets.getInsets(WindowInsetsCompat.Type.systemBars()) val previewTopMargin = systemBars.top + topInsetSpacing previewParams.topMargin = previewTopMargin previewView.layoutParams = previewParams previewOverlayParams.topMargin = previewTopMargin previewOverlay.layoutParams = previewOverlayParams topPanel.setPadding( horizontalInset + systemBars.left, previewTopMargin, horizontalInset + systemBars.right, 0, ) panel.setPadding( horizontalInset + systemBars.left, 0, horizontalInset + systemBars.right, 0, ) panelParams.bottomMargin = systemBars.bottom + bottomInsetSpacing panel.layoutParams = panelParams windowInsets } setContentView(root) ViewCompat.requestApplyInsets(root) } private fun toggleProcessing() { when (sdk.processingStatus.value) { ProcessingStatus.RUNNING -> lifecycleScope.launch { runCatching { sdk.stop() } } ProcessingStatus.STARTING, ProcessingStatus.STOPPING -> Unit else -> startProcessing() } } private fun startProcessing() { if (ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED ) { cameraPermissionLauncher.launch(Manifest.permission.CAMERA) return } lifecycleScope.launch { resetMeasurementUi() runCatching { sdk.start() } .onFailure { statusLabel.text = "Error: ${it.message ?: "Unknown"}" } } } private fun updateProcessingStatus(status: ProcessingStatus?) { when (status) { ProcessingStatus.IDLE -> { statusLabel.text = "Status: Idle" toggleButton.text = "Start" toggleButton.isEnabled = true } ProcessingStatus.STARTING -> { statusLabel.text = "Status: Starting" toggleButton.text = "Starting..." toggleButton.isEnabled = false } ProcessingStatus.RUNNING -> { statusLabel.text = "Status: Running" toggleButton.text = "Stop" toggleButton.isEnabled = true } ProcessingStatus.STOPPING -> { statusLabel.text = "Status: Stopping" toggleButton.text = "Stopping..." toggleButton.isEnabled = false } ProcessingStatus.ERROR -> { toggleButton.text = "Start" toggleButton.isEnabled = true } null -> Unit } } private fun resetMeasurementUi() { latestChestTimestamp = Long.MIN_VALUE latestAbdomenTimestamp = Long.MIN_VALUE latestPressureTimestamp = Long.MIN_VALUE chestGraphView.reset() abdomenGraphView.reset() arterialPressureGraphView.reset() heartRateLabel.text = "-- bpm" expressionLabel.text = "--" breathingRateLabel.text = "-- bpm" hrvLabel.text = "-- ms" } private fun card(buildChildren: LinearLayout.() -> Unit): LinearLayout = LinearLayout(this).apply { orientation = LinearLayout.VERTICAL background = GradientDrawable().apply { setColor(CARD_OVERLAY) cornerRadius = dp(20).toFloat() } setPadding(dp(12), dp(8), dp(12), dp(8)) buildChildren() } private fun metricCard(title: String, value: TextView, accent: Int): LinearLayout = card { addView(textView(title, sizeSp = 12f, color = TEXT_MUTED, bold = true)) addView(value, matchWrapTopMargin(dp(4))) }.apply { background = GradientDrawable().apply { setColor(CARD_OVERLAY) setStroke(dp(1), colorWithAlpha(accent, 70)) cornerRadius = dp(20).toFloat() } } private fun waveformCard(title: String, graphView: SignalGraphView): LinearLayout = card { addView(textView(title, sizeSp = 12f, color = TEXT_PRIMARY, bold = true)) addView(graphView, matchHeight(dp(78)).apply { topMargin = dp(6) }) } private fun statusPill(title: String, value: String, accent: Int): TextView = textView("$title: $value", sizeSp = 12f, color = TEXT_PRIMARY, bold = true).apply { gravity = Gravity.CENTER_VERTICAL setPadding(dp(10), 0, dp(10), 0) background = GradientDrawable().apply { setColor(0x26FFFFFF) setStroke(dp(1), colorWithAlpha(accent, 90)) cornerRadius = dp(18).toFloat() } } private fun horizontalRow(): LinearLayout = LinearLayout(this).apply { orientation = LinearLayout.HORIZONTAL gravity = Gravity.CENTER_VERTICAL } private fun textView( text: String = "", sizeSp: Float, color: Int, bold: Boolean = false, ): TextView = TextView(this).apply { this.text = text textSize = sizeSp setTextColor(color) if (bold) typeface = android.graphics.Typeface.DEFAULT_BOLD } private fun matchWrapBottomMargin(bottomMargin: Int): LinearLayout.LayoutParams = LinearLayout.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.WRAP_CONTENT, ).apply { this.bottomMargin = bottomMargin } private fun matchHeight(height: Int): LinearLayout.LayoutParams = LinearLayout.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, height, ) private fun matchWrapTopMargin(topMargin: Int): LinearLayout.LayoutParams = LinearLayout.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.WRAP_CONTENT, ).apply { this.topMargin = topMargin } private fun weightedParams(endMargin: Int = 0): LinearLayout.LayoutParams = LinearLayout.LayoutParams(0, ViewGroup.LayoutParams.WRAP_CONTENT, 1f).apply { marginEnd = endMargin } private fun dp(value: Int): Int = (value * resources.displayMetrics.density).roundToInt() private fun colorWithAlpha(color: Int, alpha: Int): Int = Color.argb(alpha, Color.red(color), Color.green(color), Color.blue(color)) private fun ExpressionType.expressionName(): String? = when (this) { ExpressionType.ANGRY -> "Angry" ExpressionType.CONTEMPT -> "Contempt" ExpressionType.DISGUST -> "Disgust" ExpressionType.FEAR -> "Fear" ExpressionType.HAPPY -> "Happy" ExpressionType.NEUTRAL -> "Neutral" ExpressionType.SAD -> "Sad" ExpressionType.SURPRISE -> "Surprise" else -> null } } private class SignalGraphView( context: Context, private val graphColor: Int, ) : View(context) { private companion object { const val GRID_ALPHA = 12 const val LINE_ALPHA = 170 } private val samples = ArrayDeque() private val maxPoints = 200 private val inset = 10f private val linePath = Path() private val gridPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply { color = Color.argb(GRID_ALPHA, 255, 255, 255) strokeWidth = 1f style = Paint.Style.STROKE } private val linePaint = Paint(Paint.ANTI_ALIAS_FLAG).apply { color = colorWithAlpha(LINE_ALPHA) strokeWidth = 3.5f style = Paint.Style.STROKE strokeCap = Paint.Cap.ROUND strokeJoin = Paint.Join.ROUND } fun appendValue(value: Float) { samples.addLast(value) while (samples.size > maxPoints) { samples.removeFirst() } invalidate() } fun reset() { samples.clear() invalidate() } override fun onDraw(canvas: Canvas) { super.onDraw(canvas) val sampleCount = samples.size if (sampleCount < 2) return val drawLeft = inset val drawTop = inset val drawRight = width - inset val drawBottom = height - inset val drawWidth = drawRight - drawLeft val drawHeight = drawBottom - drawTop for (gridLine in 1..3) { val y = drawTop + drawHeight * (1f - gridLine / 4f) canvas.drawLine(drawLeft, y, drawRight, y, gridPaint) } var minValue = Float.POSITIVE_INFINITY var maxValue = Float.NEGATIVE_INFINITY samples.forEach { minValue = min(minValue, it) maxValue = max(maxValue, it) } val range = if (maxValue - minValue == 0f) 1f else maxValue - minValue linePath.reset() samples.forEachIndexed { index, value -> val x = drawLeft + drawWidth * index / (sampleCount - 1) val normalized = (value - minValue) / range val y = drawBottom - normalized * drawHeight if (index == 0) { linePath.moveTo(x, y) } else { linePath.lineTo(x, y) } } canvas.drawPath(linePath, linePaint) } private fun colorWithAlpha(alpha: Int): Int = Color.argb(alpha, Color.red(graphColor), Color.green(graphColor), Color.blue(graphColor)) } ``` ## Step 5 — Build and run on a phone In Android Studio: 1. Choose a physical Android device as the run destination 2. Build and run the app 3. Allow camera access when Android asks 4. Tap `Start` 5. Wait a few seconds for camera tuning and signal stabilization ## What success looks like When your program is running, you should see all of these: - `Status` and `Validation` chips are visible at the top - the `Start` button changes to `Stop` after processing starts - the camera preview is below the chips - pulse rate, breathing rate, HRV, and expression cards are visible - the arterial pressure waveform is larger than the breathing waveforms - chest and abdomen waveforms both appear on screen - the screen fits in portrait orientation without scrolling Cards appear immediately but fill in on their own schedules: | Card | First value | Confident value | | --- | --- | --- | | Pulse rate | ~12 s | ~12 s (12-second average) | | Breathing rate | ~10 s | ~30 s (confidence is 0 before) | | HRV | ~30 s | ~60 s (confidence is 0 before) | See [Model cards and limitations](../../docs/model-cards-and-limitations.md). ## Expected API key check The first measurement should start after the camera permission prompt is granted. If startup fails with an authentication error, verify that `API_KEY` is valid and authorized for this app. ## Common manual mistakes If the screen does not match the target state, check these first: - the dependency was added to the wrong Gradle module - Gradle sync did not complete after adding the SmartSpectra dependency - `MainActivity.kt` was only partially replaced - `YOUR_API_KEY` was not replaced with a real key - the app is still running an older installed build on the phone ## Option 2: OAuth on Android Source: https://presagetech.com/docs/smartspectra-sdk/android/option-2-oauth --- title: "Option 2: OAuth on Android" description: SmartSpectra Android setup using OAuth instead of a hard-coded API key. sidebarTitle: "Option 2: OAuth" --- # QuickStart - OAuth Use this if you want to use SmartSpectra OAuth instead of an API key. Android OAuth is currently documented for Play Store releases. For local development, internal QA, or sideloaded debug builds, use [Option 1: API Key](option-1-api-key.md) instead. ## What you will change manually You will touch exactly these things: 1. The app Gradle repositories and dependencies 2. The OAuth app registration in the Presage developer portal 3. The OAuth XML file at `app/src/main/res/xml/presage_services.xml` 4. `app/src/main/java/com/example/coolvitals/MainActivity.kt` You do not need to create XML layouts or additional Kotlin files. ## Requirements Use a compatible Android toolchain before adding the SDK: - `minSdk` **28** or later - `compileSdk` **36.1** or later - Android Gradle Plugin (AGP) **8.10.1** or later; use an AGP 9.x release when your project uses Gradle 9 - Kotlin **2.2.x** These requirements apply to SmartSpectra 3.2.x. New Android Studio projects can usually satisfy them by updating the generated project-level plugin versions before the first Gradle sync. ## Result you should get At the end, the app should show: - live camera preview - pulse rate, breathing rate, HRV RMSSD, and expression cards - arterial pressure waveform - chest and abdomen breathing waveforms - status text and a start/stop button - one portrait screen with no scrolling ![SmartSpectra Android quickstart demo](../media/android-quickstart.gif) ## Step 1 — Create the project In Android Studio, create a new Android app project: 1. Select `File` → `New` → `New Project...` 2. Choose `Empty Activity` 3. Set `Name` to `Cool Vitals` 4. Set `Package name` to `com.example.coolvitals` 5. Set `Language` to `Kotlin` 6. Set `Minimum SDK` to `API 28` or newer 7. Finish creating the project If you already created the project, open it instead. Open your Android app project in Android Studio. ## Step 2 — Add repositories In your project-level `settings.gradle.kts`, make sure the app can resolve AndroidX and SmartSpectra artifacts: ```kotlin dependencyResolutionManagement { repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS) repositories { google() mavenCentral() // Required only for versions ending in -SNAPSHOT. maven { url = uri("https://central.sonatype.com/repository/maven-snapshots/") mavenContent { snapshotsOnly() } } } } ``` ## Step 3 — Add app dependencies In `app/build.gradle.kts`, keep the dependencies generated by the `Empty Activity` template and add only these lines to the existing `dependencies` block: ```kotlin dependencies { implementation("androidx.activity:activity-ktx:1.8.1") implementation("androidx.core:core-ktx:1.18.0") implementation("androidx.lifecycle:lifecycle-runtime-ktx:2.10.0") implementation("androidx.camera:camera-view:1.6.0") implementation("com.presagetech:smartspectra:3.4.0") } ``` `androidx.activity:activity-ktx` supplies `ComponentActivity` and `registerForActivityResult`, both used by the complete activity below. The version above is the release these docs describe. To use a different one, browse the published versions on [Maven Central](https://central.sonatype.com/artifact/com.presagetech/smartspectra/versions). Manual check: - Gradle sync succeeds - AndroidX imports resolve: `PreviewView`, `ContextCompat`, and `lifecycleScope` - SmartSpectra imports resolve: `SmartSpectraSdk` and `MetricType` ## Step 4 — Declare manifest permissions Add these permissions directly inside the root `` element in `app/src/main/AndroidManifest.xml`, before ``: ```xml ``` ## Step 5 — Get `presage_services.xml` This file is not created by Android Studio and is not included automatically when you add the SmartSpectra dependency. You need to get it from Presage first: 1. Sign in to the Presage developer portal: `https://physiology.presagetech.com/auth/login` 2. Open **Account** → **OAuth Registration** 3. Select `Android` 4. Enter your Android App ID. For this quickstart, use `com.example.coolvitals`. 5. Enter the SHA-256 fingerprint for the signing certificate used by the app you will release. 6. Click **Register App ID** 7. Download the Android OAuth config file named `presage_services.xml` If you are testing a sandbox-enabled app registration, make sure the app row shows **Sandbox Status: Enabled**. Click **Enable** if sandbox is not already enabled. An AI assistant connected to the [SmartSpectra MCP Server](../../docs/mcp-server.md) can do this step for you: it registers the app ID and fingerprint (`apps.register`) and fetches `presage_services.xml` (`apps.get_config`) without you opening the portal. Registration asks you to confirm before it takes effect. See Google's [Authenticating Your Client](https://developers.google.com/android/guides/client-auth) guide for Android signing certificate fingerprints. To get the SHA-256 fingerprint from Gradle, run: ```bash ./gradlew signingReport ``` ![Android Studio signing report showing the SHA-256 fingerprint](../media/SHA256Example.jpeg) If you cannot find a download for `presage_services.xml`, stop here. Ask [Presage support](mailto:support@presagetech.com) or your Presage contact for the Android OAuth XML for this app. ## Step 6 — Add `presage_services.xml` to the app In Android Studio: 1. Create `app/src/main/res/xml/` if it does not already exist 2. Put `presage_services.xml` in that directory 3. Confirm the file is packaged with the app target The file should contain the OAuth fields provided by Presage, for example: ```xml true your_client_id your_sub 1.0 ``` ## Step 7 — Replace `MainActivity.kt` In Android Studio: 1. Open `app/src/main/java/com/example/coolvitals/MainActivity.kt` 2. Delete everything in the file 3. Paste the full file below Paste this entire file: ```kotlin package com.example.coolvitals import android.Manifest import android.content.Context import android.content.pm.PackageManager import android.content.res.ColorStateList import android.graphics.Canvas import android.graphics.Color import android.graphics.Paint import android.graphics.Path import android.graphics.drawable.GradientDrawable import android.os.Bundle import android.view.Gravity import android.view.View import android.view.ViewGroup import android.widget.Button import android.widget.FrameLayout import android.widget.LinearLayout import android.widget.TextView import androidx.activity.ComponentActivity import androidx.activity.result.contract.ActivityResultContracts import androidx.camera.view.PreviewView import androidx.core.content.ContextCompat import androidx.core.view.ViewCompat import androidx.core.view.WindowCompat import androidx.core.view.WindowInsetsCompat import androidx.lifecycle.lifecycleScope import com.presagetech.smartspectra.CameraSelection import com.presagetech.smartspectra.ProcessingStatus import com.presagetech.smartspectra.SmartSpectraConfig import com.presagetech.smartspectra.SmartSpectraSdk import com.presagetech.smartspectra.proto.MetricTypesProto.MetricType import com.presagetech.smartspectra.proto.MetricsProto.ExpressionType import kotlin.math.max import kotlin.math.min import kotlin.math.roundToInt import kotlinx.coroutines.launch class MainActivity : ComponentActivity() { private companion object { val TEXT_PRIMARY = Color.WHITE val TEXT_MUTED = 0x80FFFFFF.toInt() val CARD_OVERLAY = 0xE61A2233.toInt() val CORAL = 0xFFFF6B6B.toInt() val TEAL = 0xFF4FCCC4.toInt() val VIOLET = 0xFFA58CF9.toInt() val MINT = 0xFF6EE7B7.toInt() val BLUE = 0xFF60A5FA.toInt() val AMBER = 0xFFFBBF24.toInt() } private val sdk by lazy { SmartSpectraSdk.shared } private lateinit var heartRateLabel: TextView private lateinit var expressionLabel: TextView private lateinit var breathingRateLabel: TextView private lateinit var hrvLabel: TextView private lateinit var chestGraphView: SignalGraphView private lateinit var abdomenGraphView: SignalGraphView private lateinit var arterialPressureGraphView: SignalGraphView private lateinit var statusLabel: TextView private lateinit var validationLabel: TextView private lateinit var toggleButton: Button private var latestChestTimestamp: Long = Long.MIN_VALUE private var latestAbdomenTimestamp: Long = Long.MIN_VALUE private var latestPressureTimestamp: Long = Long.MIN_VALUE private val cameraPermissionLauncher = registerForActivityResult( ActivityResultContracts.RequestPermission(), ) { granted -> if (granted) { startProcessing() } else { statusLabel.text = "Status: Camera required" } } override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) WindowCompat.setDecorFitsSystemWindows(window, false) sdk.config.imageOutputEnabled = false sdk.useCamera(CameraSelection.Front) sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics + listOf(MetricType.EXPRESSIONS) buildUi() bindSdk() resetMeasurementUi() statusLabel.text = "Status: Idle" } override fun onPause() { super.onPause() lifecycleScope.launch { when (sdk.processingStatus.value) { ProcessingStatus.RUNNING, ProcessingStatus.STARTING, ProcessingStatus.STOPPING, -> runCatching { sdk.stop() } else -> Unit } } } private fun bindSdk() { sdk.processingStatus.observe(this) { updateProcessingStatus(it) } sdk.validationStatus.observe(this) { status -> validationLabel.text = "Validation: ${status?.code?.name?.replace('_', ' ') ?: "--"}" } sdk.error.observe(this) { error -> if (error != null) { statusLabel.text = "Error: ${error.message ?: "Unknown"}" } } sdk.metrics.observe(this) { metrics -> if (metrics == null) return@observe if (metrics.hasCardio()) { val pulse = metrics.cardio.pulseRateList .lastOrNull { it.timestamp > 0 } ?.value ?.roundToInt() if (pulse != null) { heartRateLabel.text = "$pulse bpm" } metrics.cardio.arterialPressureTraceList.forEach { sample -> if (sample.timestamp > latestPressureTimestamp) { latestPressureTimestamp = sample.timestamp arterialPressureGraphView.appendValue(sample.value) } } metrics.cardio.hrvList.lastOrNull()?.rmssd?.let { rmssd -> if (rmssd > 0) { hrvLabel.text = "${(rmssd * 10).roundToInt() / 10.0} ms" } } } if (metrics.hasBreathing()) { if (metrics.breathing.rateCount > 0) { val breathingRate = metrics.breathing.rateList.last().value.roundToInt() breathingRateLabel.text = "$breathingRate bpm" } metrics.breathing.upperTraceList.forEach { sample -> if (sample.timestamp > latestChestTimestamp) { latestChestTimestamp = sample.timestamp chestGraphView.appendValue(sample.value) } } metrics.breathing.lowerTraceList.forEach { sample -> if (sample.timestamp > latestAbdomenTimestamp) { latestAbdomenTimestamp = sample.timestamp abdomenGraphView.appendValue(sample.value) } } } if (metrics.hasFace()) { val expression = metrics.face.expressionList.lastOrNull() ?: return@observe val topScore = expression.scoresList .filter { it.confidence > 0f } .maxByOrNull { it.confidence } ?: return@observe val expressionName = topScore.type.expressionName() ?: return@observe expressionLabel.text = "%-8.8s %3d%%".format(expressionName, topScore.confidence.roundToInt()) } } } private fun buildUi() { val horizontalInset = dp(14) val topInsetSpacing = dp(8) val bottomInsetSpacing = dp(12) val root = FrameLayout(this).apply { background = GradientDrawable( GradientDrawable.Orientation.TOP_BOTTOM, intArrayOf(0xFF0B1020.toInt(), 0xFF05070C.toInt()), ) } val previewView = PreviewView(this).apply { contentDescription = "SmartSpectra preview output" setBackgroundColor(Color.BLACK) scaleType = PreviewView.ScaleType.FILL_CENTER implementationMode = PreviewView.ImplementationMode.COMPATIBLE } sdk.config.previewSurfaceProvider = previewView.surfaceProvider val previewParams = FrameLayout.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, dp(465), Gravity.TOP, ) root.addView(previewView, previewParams) val previewOverlay = View(this).apply { background = GradientDrawable( GradientDrawable.Orientation.TOP_BOTTOM, intArrayOf(0x33000000, 0x00000000, 0xCC05070C.toInt()), ) } val previewOverlayParams = FrameLayout.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, dp(465), Gravity.TOP, ) root.addView( previewOverlay, previewOverlayParams, ) val topPanel = LinearLayout(this).apply { orientation = LinearLayout.VERTICAL setPadding(horizontalInset, topInsetSpacing, horizontalInset, 0) } root.addView( topPanel, FrameLayout.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.WRAP_CONTENT, Gravity.TOP, ), ) val topRow = horizontalRow() statusLabel = statusPill("Status", "Idle", CORAL) validationLabel = statusPill("Validation", "--", AMBER) toggleButton = Button(this).apply { text = "Start" setAllCaps(false) setTextColor(Color.BLACK) backgroundTintList = ColorStateList.valueOf(Color.WHITE) setOnClickListener { toggleProcessing() } } topRow.addView(statusLabel, weightedParams(endMargin = dp(8))) topRow.addView(validationLabel, weightedParams(endMargin = dp(8))) topRow.addView(toggleButton, LinearLayout.LayoutParams(dp(92), dp(42))) topPanel.addView(topRow) val panel = LinearLayout(this).apply { orientation = LinearLayout.VERTICAL setPadding(horizontalInset, 0, horizontalInset, 0) } val panelParams = FrameLayout.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.WRAP_CONTENT, Gravity.BOTTOM, ) root.addView( panel, panelParams, ) heartRateLabel = textView(sizeSp = 23f, color = CORAL, bold = true) breathingRateLabel = textView(sizeSp = 23f, color = TEAL, bold = true) val rateRow = horizontalRow() rateRow.addView(metricCard("Pulse Rate", heartRateLabel, CORAL), weightedParams(endMargin = dp(8))) rateRow.addView(metricCard("Breathing Rate", breathingRateLabel, TEAL), weightedParams()) panel.addView(rateRow, matchWrapBottomMargin(dp(10))) hrvLabel = textView(sizeSp = 23f, color = TEXT_PRIMARY, bold = true) expressionLabel = textView(sizeSp = 20f, color = TEXT_PRIMARY, bold = true).apply { typeface = android.graphics.Typeface.create(android.graphics.Typeface.MONOSPACE, android.graphics.Typeface.BOLD) includeFontPadding = false } val summaryRow = horizontalRow() summaryRow.addView(metricCard("HRV RMSSD", hrvLabel, MINT), weightedParams(endMargin = dp(8))) summaryRow.addView(metricCard("Expression", expressionLabel, AMBER), weightedParams()) panel.addView(summaryRow, matchWrapBottomMargin(dp(10))) arterialPressureGraphView = SignalGraphView(this, VIOLET) panel.addView(waveformCard("Arterial Pressure", arterialPressureGraphView), matchWrapBottomMargin(dp(10))) chestGraphView = SignalGraphView(this, TEAL) abdomenGraphView = SignalGraphView(this, BLUE) val breathingRow = horizontalRow() breathingRow.addView(waveformCard("Chest Waveform", chestGraphView), weightedParams(endMargin = dp(8))) breathingRow.addView(waveformCard("Abdomen Waveform", abdomenGraphView), weightedParams()) panel.addView(breathingRow, matchHeight(dp(126))) ViewCompat.setOnApplyWindowInsetsListener(root) { _, windowInsets -> val systemBars = windowInsets.getInsets(WindowInsetsCompat.Type.systemBars()) val previewTopMargin = systemBars.top + topInsetSpacing previewParams.topMargin = previewTopMargin previewView.layoutParams = previewParams previewOverlayParams.topMargin = previewTopMargin previewOverlay.layoutParams = previewOverlayParams topPanel.setPadding( horizontalInset + systemBars.left, previewTopMargin, horizontalInset + systemBars.right, 0, ) panel.setPadding( horizontalInset + systemBars.left, 0, horizontalInset + systemBars.right, 0, ) panelParams.bottomMargin = systemBars.bottom + bottomInsetSpacing panel.layoutParams = panelParams windowInsets } setContentView(root) ViewCompat.requestApplyInsets(root) } private fun toggleProcessing() { when (sdk.processingStatus.value) { ProcessingStatus.RUNNING -> lifecycleScope.launch { runCatching { sdk.stop() } } ProcessingStatus.STARTING, ProcessingStatus.STOPPING -> Unit else -> startProcessing() } } private fun startProcessing() { if (ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED ) { cameraPermissionLauncher.launch(Manifest.permission.CAMERA) return } lifecycleScope.launch { resetMeasurementUi() runCatching { sdk.start() } .onFailure { statusLabel.text = "Error: ${it.message ?: "Unknown"}" } } } private fun updateProcessingStatus(status: ProcessingStatus?) { when (status) { ProcessingStatus.IDLE -> { statusLabel.text = "Status: Idle" toggleButton.text = "Start" toggleButton.isEnabled = true } ProcessingStatus.STARTING -> { statusLabel.text = "Status: Starting" toggleButton.text = "Starting..." toggleButton.isEnabled = false } ProcessingStatus.RUNNING -> { statusLabel.text = "Status: Running" toggleButton.text = "Stop" toggleButton.isEnabled = true } ProcessingStatus.STOPPING -> { statusLabel.text = "Status: Stopping" toggleButton.text = "Stopping..." toggleButton.isEnabled = false } ProcessingStatus.ERROR -> { toggleButton.text = "Start" toggleButton.isEnabled = true } null -> Unit } } private fun resetMeasurementUi() { latestChestTimestamp = Long.MIN_VALUE latestAbdomenTimestamp = Long.MIN_VALUE latestPressureTimestamp = Long.MIN_VALUE chestGraphView.reset() abdomenGraphView.reset() arterialPressureGraphView.reset() heartRateLabel.text = "-- bpm" expressionLabel.text = "--" breathingRateLabel.text = "-- bpm" hrvLabel.text = "-- ms" } private fun card(buildChildren: LinearLayout.() -> Unit): LinearLayout = LinearLayout(this).apply { orientation = LinearLayout.VERTICAL background = GradientDrawable().apply { setColor(CARD_OVERLAY) cornerRadius = dp(20).toFloat() } setPadding(dp(12), dp(8), dp(12), dp(8)) buildChildren() } private fun metricCard(title: String, value: TextView, accent: Int): LinearLayout = card { addView(textView(title, sizeSp = 12f, color = TEXT_MUTED, bold = true)) addView(value, matchWrapTopMargin(dp(4))) }.apply { background = GradientDrawable().apply { setColor(CARD_OVERLAY) setStroke(dp(1), colorWithAlpha(accent, 70)) cornerRadius = dp(20).toFloat() } } private fun waveformCard(title: String, graphView: SignalGraphView): LinearLayout = card { addView(textView(title, sizeSp = 12f, color = TEXT_PRIMARY, bold = true)) addView(graphView, matchHeight(dp(78)).apply { topMargin = dp(6) }) } private fun statusPill(title: String, value: String, accent: Int): TextView = textView("$title: $value", sizeSp = 12f, color = TEXT_PRIMARY, bold = true).apply { gravity = Gravity.CENTER_VERTICAL setPadding(dp(10), 0, dp(10), 0) background = GradientDrawable().apply { setColor(0x26FFFFFF) setStroke(dp(1), colorWithAlpha(accent, 90)) cornerRadius = dp(18).toFloat() } } private fun horizontalRow(): LinearLayout = LinearLayout(this).apply { orientation = LinearLayout.HORIZONTAL gravity = Gravity.CENTER_VERTICAL } private fun textView( text: String = "", sizeSp: Float, color: Int, bold: Boolean = false, ): TextView = TextView(this).apply { this.text = text textSize = sizeSp setTextColor(color) if (bold) typeface = android.graphics.Typeface.DEFAULT_BOLD } private fun matchWrapBottomMargin(bottomMargin: Int): LinearLayout.LayoutParams = LinearLayout.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.WRAP_CONTENT, ).apply { this.bottomMargin = bottomMargin } private fun matchHeight(height: Int): LinearLayout.LayoutParams = LinearLayout.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, height, ) private fun matchWrapTopMargin(topMargin: Int): LinearLayout.LayoutParams = LinearLayout.LayoutParams( ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.WRAP_CONTENT, ).apply { this.topMargin = topMargin } private fun weightedParams(endMargin: Int = 0): LinearLayout.LayoutParams = LinearLayout.LayoutParams(0, ViewGroup.LayoutParams.WRAP_CONTENT, 1f).apply { marginEnd = endMargin } private fun dp(value: Int): Int = (value * resources.displayMetrics.density).roundToInt() private fun colorWithAlpha(color: Int, alpha: Int): Int = Color.argb(alpha, Color.red(color), Color.green(color), Color.blue(color)) private fun ExpressionType.expressionName(): String? = when (this) { ExpressionType.ANGRY -> "Angry" ExpressionType.CONTEMPT -> "Contempt" ExpressionType.DISGUST -> "Disgust" ExpressionType.FEAR -> "Fear" ExpressionType.HAPPY -> "Happy" ExpressionType.NEUTRAL -> "Neutral" ExpressionType.SAD -> "Sad" ExpressionType.SURPRISE -> "Surprise" else -> null } } private class SignalGraphView( context: Context, private val graphColor: Int, ) : View(context) { private companion object { const val GRID_ALPHA = 12 const val LINE_ALPHA = 170 } private val samples = ArrayDeque() private val maxPoints = 200 private val inset = 10f private val linePath = Path() private val gridPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply { color = Color.argb(GRID_ALPHA, 255, 255, 255) strokeWidth = 1f style = Paint.Style.STROKE } private val linePaint = Paint(Paint.ANTI_ALIAS_FLAG).apply { color = colorWithAlpha(LINE_ALPHA) strokeWidth = 3.5f style = Paint.Style.STROKE strokeCap = Paint.Cap.ROUND strokeJoin = Paint.Join.ROUND } fun appendValue(value: Float) { samples.addLast(value) while (samples.size > maxPoints) { samples.removeFirst() } invalidate() } fun reset() { samples.clear() invalidate() } override fun onDraw(canvas: Canvas) { super.onDraw(canvas) val sampleCount = samples.size if (sampleCount < 2) return val drawLeft = inset val drawTop = inset val drawRight = width - inset val drawBottom = height - inset val drawWidth = drawRight - drawLeft val drawHeight = drawBottom - drawTop for (gridLine in 1..3) { val y = drawTop + drawHeight * (1f - gridLine / 4f) canvas.drawLine(drawLeft, y, drawRight, y, gridPaint) } var minValue = Float.POSITIVE_INFINITY var maxValue = Float.NEGATIVE_INFINITY samples.forEach { minValue = min(minValue, it) maxValue = max(maxValue, it) } val range = if (maxValue - minValue == 0f) 1f else maxValue - minValue linePath.reset() samples.forEachIndexed { index, value -> val x = drawLeft + drawWidth * index / (sampleCount - 1) val normalized = (value - minValue) / range val y = drawBottom - normalized * drawHeight if (index == 0) { linePath.moveTo(x, y) } else { linePath.lineTo(x, y) } } canvas.drawPath(linePath, linePaint) } private fun colorWithAlpha(alpha: Int): Int = Color.argb(alpha, Color.red(graphColor), Color.green(graphColor), Color.blue(graphColor)) } ``` ## Step 8 — Build and run on a phone In Android Studio: 1. Choose a physical Android device as the run destination 2. Build and run the app 3. Allow camera access when Android asks 4. Tap `Start` 5. Wait a few seconds for camera tuning and signal stabilization ## What success looks like When your program is running, you should see all of these: - `Status` and `Validation` chips are visible at the top - the `Start` button changes to `Stop` after processing starts - the camera preview is below the chips - pulse rate, breathing rate, HRV, and expression cards are visible - the arterial pressure waveform is larger than the breathing waveforms - chest and abdomen waveforms both appear on screen - the screen fits in portrait orientation without scrolling Cards appear immediately but fill in on their own schedules: | Card | First value | Confident value | | --- | --- | --- | | Pulse rate | ~12 s | ~12 s (12-second average) | | Breathing rate | ~10 s | ~30 s (confidence is 0 before) | | HRV | ~30 s | ~60 s (confidence is 0 before) | See [Model cards and limitations](../../docs/model-cards-and-limitations.md). ## Expected OAuth check If OAuth is wired correctly, the app should start without setting `sdk.config.apiKey`. If startup fails with an authentication error, verify that `presage_services.xml` is present in `app/src/main/res/xml/` and that the OAuth app registration matches the installed app. ## Common manual mistakes If the screen does not match the target state, check these first: - the dependency was added to the wrong Gradle module - Gradle sync did not complete after adding the SmartSpectra dependency - `MainActivity.kt` was only partially replaced - `presage_services.xml` is not in `app/src/main/res/xml/` - the OAuth app registration uses a different package name or signing certificate - the app is still running an older installed build on the phone ## Android Troubleshooting Source: https://presagetech.com/docs/smartspectra-sdk/android/troubleshooting --- title: Android Troubleshooting description: Solutions to common build, runtime, and integration issues with the SmartSpectra Android SDK. sidebarTitle: Troubleshooting --- # Android Troubleshooting ## Build & Setup ### `Manifest merger failed: uses-sdk:minSdkVersion 24 cannot be smaller than version 28` Set `minSdk 28` in your `app/build.gradle`: ```groovy android { defaultConfig { minSdk 28 } } ``` --- ### `AAR requires API 36.1` (or a compile SDK version that is too low) SmartSpectra 3.2.x requires `compileSdk 36.1` or later. Update your app module and Android build tooling, then sync Gradle again: ```kotlin android { compileSdk = 36 compileSdkMinor = 1 } ``` Use AGP 8.10.1 or later and Kotlin 2.2.x. If your project uses Gradle 9, use an AGP 9.x release. See [Option 1: API Key](option-1-api-key.md#requirements) for the complete compatible toolchain. --- ### `Unresolved reference: ComponentActivity` (or similar import errors) The quickstarts extend `ComponentActivity`, which comes from `androidx.activity:activity-ktx`. Confirm that dependency is declared, then check the imports in your activity file: ```kotlin import android.os.Bundle import androidx.activity.ComponentActivity import com.presagetech.smartspectra.SmartSpectraSdk ``` If you are following older material that used `AppCompatActivity`, either switch to `ComponentActivity` or add `androidx.appcompat:appcompat` yourself — the SDK does not pull it in. --- ### `AAPT: error: resource mipmap/ic_launcher not found` Remove icon references from `AndroidManifest.xml` or add the missing drawable resources. A minimal application tag that avoids this: ```xml ``` --- ### General build failures after an SDK update 1. **Clean Project** — Build → Clean Project 2. **Rebuild** — Build → Rebuild Project 3. **Sync Gradle** — File → Sync Project with Gradle Files (or the elephant icon in the toolbar) 4. **Invalidate Caches** — File → Invalidate Caches and Restart If the `R` class stops resolving in the linter, Sync Project with Gradle Files typically fixes it. --- ## Camera & Permissions ### Camera permission denied / measurement won't start When using SDK-owned camera capture, the host app requests Android's runtime camera permission before calling `sdk.start()`. The SDK does not show the permission dialog itself. Custom input does not require camera permission from the SDK; your capture source may require it separately. Common causes: - Testing measurements on an emulator — use [recorded frames](headless-testing-in-ci.md); the simulated camera feed is not suitable for measuring a person's vitals. - Permission was previously denied — guide the user to re-enable camera access in system Settings. - `start()` was called before permission was granted — observe `sdk.error` for `SmartSpectraError(code = INPUT_UNAVAILABLE, retryable = true)`. ### Customizing the permission rationale message Because the host app owns the permission prompt, keep the rationale string in your app resources and show it from your onboarding or permission UI: ```xml Your custom message explaining why camera access is needed. ``` ### Requesting permission before starting the SDK Request camera permission with the modern `ActivityResultLauncher` pattern: ```kotlin import android.Manifest import android.content.pm.PackageManager import android.os.Bundle import androidx.activity.result.ActivityResultLauncher import androidx.activity.result.contract.ActivityResultContracts import androidx.core.content.ContextCompat import androidx.lifecycle.lifecycleScope import kotlinx.coroutines.launch private lateinit var requestCameraPermission: ActivityResultLauncher override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) requestCameraPermission = registerForActivityResult( ActivityResultContracts.RequestPermission() ) { granted -> if (granted) startMeasurement() } } private fun startMeasurement() { if (ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED ) { requestCameraPermission.launch(Manifest.permission.CAMERA) return } lifecycleScope.launch { sdk.start() } } ``` With SDK-owned camera capture, if `start()` is called without permission, the SDK publishes a `SmartSpectraError(code = INPUT_UNAVAILABLE, retryable = true)` to `sdk.error`. Observe that error to surface recovery UI and retry after the user grants access. --- ## Authentication ### Auth errors (401 / 403) 1. Verify the API key string is correct in your code. 2. Confirm your subscription is active at [physiology.presagetech.com](https://physiology.presagetech.com/auth/login). 3. Check that the device has an active internet connection — the SDK requires network access for subscription validation. ### OAuth not working in local or debug builds Android OAuth is currently documented for Play Store releases only. For local development, internal QA, or sideloaded debug builds, use an API key instead. If you're preparing a Play Store release, register the SHA-256 fingerprint for the signing certificate used by that release, then re-download `presage_services.xml`. Run: ```bash keytool -list -v -keystore -alias -storepass | grep SHA256 ``` Register that fingerprint under **Account → OAuth Registration** alongside your package name, then re-download and replace `presage_services.xml`. > **Note:** Each package name can only be registered once. You cannot create multiple OAuth configs for the same package name. --- ## Getting Help - Email: [support@presagetech.com](mailto:support@presagetech.com) - [Submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues) - [Docs and FAQ](https://smartspectra.presagetech.com) - [Developer Admin Portal](https://physiology.presagetech.com/auth/login) ## Get Started with the SmartSpectra C++ SDK Source: https://presagetech.com/docs/smartspectra-sdk/cpp --- title: Get Started with the SmartSpectra C++ SDK description: Measure pulse and breathing from a camera or video file with the SmartSpectra C++ SDK on Linux, macOS, and Windows. C++20 and CMake. sidebarTitle: Overview --- # SmartSpectra C++ SDK Cross-platform C++ SDK for measuring vitals and waveform shapes (pulse, breathing, relative arterial pressure, and more) from a camera. Headless by default with optional preview frames; runs on Linux, macOS, and Windows. ## Supported Platforms | Platform | Status | Notes | | --- | --- | --- | | Ubuntu 22.04 / Mint 21 (amd64) | Experimental | Debian package available | | Ubuntu 22.04 / Mint 21 (arm64) | Experimental | Debian package available | | Ubuntu 24.04 / Mint 22 (amd64) | Experimental | Debian package available via `packages.presagetech.com` apt channel | | Ubuntu 24.04 / Mint 22 (arm64) | Experimental | Debian package available via `packages.presagetech.com` apt channel | | macOS Apple Silicon (14.0+) | Supported | Homebrew package available | | Windows 10 / 11 (x64) | Experimental | ZIP distribution available | | macOS Intel | Not supported | — | | Debian 13 / Trixie (amd64) | Experimental | Debian package available | | Debian 13 / Trixie (arm64) | Experimental | Debian package available | | Debian 12 | Not supported | — | | RHEL 9 / Fedora 41 | Not supported | — | For platforms marked "Not supported" or anything not listed above, contact [support@presagetech.com](mailto:support@presagetech.com) if you have a specific need. ## Camera selection On Linux, macOS, and Windows, discover cameras with `SmartSpectra::AvailableCameras(cameras)` and choose one with `spectra.UseCamera(CameraSelection::ById(camera.id)).Build()`. `Default()`, `Front()`, and `Back()` express a camera preference or requirement. Discovery also exists in the native C++ API on iOS. See the [camera selection migration notes](docs/migration-guide.md#explicit-camera-selection) for platform support, errors, and a complete example. ## Common Prerequisites All platforms need: - **CMake 3.22.1+** - **C++20 compiler** (GCC, Clang, or MSVC 2022) - An **API key** from [physiology.presagetech.com](https://physiology.presagetech.com/auth/login) — an AI assistant connected to the [SmartSpectra MCP Server](../docs/mcp-server.md) can fetch it from your account for you The SDK package is self-contained — you do not need to install protobuf, curl, or OpenSSL separately on any platform. ## Pick your platform Each guide is self-contained: prerequisites → install → first running build. > **Start with a full runnable sample:** choose the quickstart for your > platform below. Each guide includes a complete `hello_vitals.cpp` and > `CMakeLists.txt`. - [**Linux Quickstart (Debian/Ubuntu/Mint)**](docs/linux/index.md) — apt-based install for Debian 13 / Trixie, Ubuntu 22.04 / Mint 21, and Ubuntu 24.04 / Mint 22 (`amd64` + `arm64`) - [**macOS Quickstart**](docs/macos.md) — Homebrew formula, Apple Silicon - [**Windows Quickstart**](docs/windows/index.md) — prebuilt ZIP from GitHub Releases ## Scope The quickstarts intentionally request only the breathing and cardio metric bundles, which is enough to see live values on the console. See the [C++ API reference](docs/api-reference.md) for the full `requested_metrics` catalog and custom-input pipeline. If you need to redistribute a Linux desktop app without requiring end users to configure the Presage apt source, see [Redistribute SmartSpectra on Linux](../docs/redistribute_smartspectra_on_linux.md). ## Going further Once your first build runs: - [Configure which metrics to compute](docs/metrics.md) - [Headless mode](docs/headless-mode.md) — C++ is headless by default; see the guide for video output callbacks - [Redistribute SmartSpectra on Linux](../docs/redistribute_smartspectra_on_linux.md) — bundle the published Linux SDK tarball into your own `.deb` - [Migration guide](docs/migration-guide.md) — upgrading from v1.x or v2.x - [API reference](docs/api-reference.md) - [LLM Insights](docs/llm-insights.md) — natural-language analysis of the measured vitals from an LLM, on request ## Logging SDK log verbosity defaults to warnings and errors only. To change it, set `log_level` on the config before constructing `SmartSpectra`: ```cpp SmartSpectraConfig config; config.api_key = "..."; config.log_level = presage::smartspectra::SmartSpectraLogLevel::kInfo; SmartSpectra spectra(config); ``` Levels are cumulative — `kDebug`, `kInfo`, `kWarning` (default), `kError`, `kNone`. `kDebug` also enables verbose diagnostics where they exist, but it cannot restore debug-only statements compiled out of release binaries. A `GLOG_minloglevel` environment variable always takes precedence — when it is set, the SDK leaves logging untouched. ## Bugs & Troubleshooting - Each platform quickstart above ends with a Troubleshooting section covering platform-specific install, signing, and runtime issues. - For additional support, contact [support@presagetech.com](mailto:support@presagetech.com) or [submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues). ## C++ API Reference Source: https://presagetech.com/docs/smartspectra-sdk/cpp/api-reference --- title: C++ API Reference description: API reference for the SmartSpectra C++ SDK — the SmartSpectra class, callbacks, builders, metrics, and error codes. sidebarTitle: API Reference --- ## SmartSpectra Entry point for video-based vitals measurement: owns the processing pipeline and routes output streams to registered callbacks. Typical usage: ```cpp SmartSpectraConfig config; config.api_key = "..."; config.requested_metrics = SmartSpectraConfig::DefaultSupportedMetrics(); SmartSpectra spectra(config); // 1. Register callbacks (all optional, call before Start). spectra.SetOnValidationStatusChanged([](const ValidationStatus& vs, int64_t ts) { ... }); spectra.SetOnMetrics([](const Metrics& m, int64_t ts) { ... }); spectra.SetOnError([](const SmartSpectraError& e) { ... }); // 2. Configure input source (optional — legacy default is camera index zero). spectra.UseCamera(CameraSelection::Default()).Build(); spectra.UseFile("video.mp4").SetTimestamps("ts.txt").Build(); // 3. Start (auth + graph + source in one call). if (auto err = spectra.Start(); !err.ok()) { LOG(ERROR) << err.FullMessage(); } // 4. Wait (file sources stop at EOF; camera sources run until Stop()). spectra.WaitUntilComplete(); spectra.Stop(); ``` Thread safety: all public methods are thread-safe and may be called from any thread. Callbacks fire on internal worker threads — NOT the calling thread. Do not call any SmartSpectra method from within a callback without external synchronization; the internal mutex is not re-entrant. ### Methods - ```cpp static constexpr std::string_view version = SMART_SPECTRA_VERSION_STRING ``` The SDK version string. - ```cpp void SetOnProcessingStatusChanged(OnProcessingStatusChangedFn cb) ``` Registers a callback for processing-status changes. Register callbacks before Start(). Callbacks run on internal worker threads, so synchronize any state you share with your own threads. - ```cpp void SetOnValidationStatusChanged(OnValidationStatusChangedFn cb) ``` - ```cpp void SetOnMetrics(OnMetricsFn cb) ``` - ```cpp void SetOnAccumulatedMetrics(OnAccumulatedMetricsFn cb) ``` - ```cpp void SetOnVideoOutput(OnVideoOutputFn cb) ``` - ```cpp void SetOnFrameSentThrough(OnFrameSentThroughFn cb) ``` - ```cpp void SetOnError(OnErrorFn cb) ``` Registers a callback invoked when an unrecoverable error occurs. It fires once per error, so it need not be idempotent; the error state clears on the next Start(). - ```cpp void SetOnInsight(OnInsightFn cb) ``` Receives insight responses — both the auto-fired VITALS snapshots (one as each of pulse rate, breathing rate and HRV first stabilises, then one every 60 s) and on-demand responses to RequestInsight(). Insight::type() is always INSIGHT_TYPE_VITALS today (SPEECH and COMBINED are reserved and not emitted), so correlate on-demand replies with Insight::request_id(), not type(). - ```cpp [[nodiscard]] SmartSpectraError Start() ``` Starts a measurement session: authenticates, builds the processing pipeline, and begins processing. Returns the first error encountered, or Ok on success. - ```cpp [[nodiscard]] SmartSpectraError Stop() ``` Stops the active measurement session. Safe to call when no session is running — it returns success immediately. - ```cpp [[nodiscard]] SmartSpectraError Reset() ``` Recovers from an unrecoverable error by tearing down and rebuilding the session. Use this after GetStatus() reports an error; for a normal restart after Stop(), call Start() instead. Source configuration is preserved. - ```cpp bool WaitUntilComplete( std::chrono::milliseconds timeout = std::chrono::milliseconds::max()) ``` Blocks until the session finishes or the timeout elapses. Returns true if it completed normally, false on timeout. File sources finish at end-of-file; camera sources finish when Stop() is called. - ```cpp [[nodiscard]] SmartSpectraError RequestInsight( const std::string& text, int32_t* out_request_id = nullptr) ``` Dispatches an on-demand insight prompt. Requires an active session (call Start() first). The matching Insight response is delivered asynchronously through the OnInsightFn callback; callers can correlate it via Insight::request_id(). `text` is the prompt. When buffered vitals exist at dispatch time the request is combined (prompt + latest metrics); otherwise it is prompt-only. If `out_request_id` is non-null it receives the RequestId assigned to the dispatched request. Returns kInvalidState when no session is active, kProcessingFailed when the request fails to dispatch locally — for example no insight callback registered, a prompt over the size limit, or the session could not be established. This return value covers dispatch only; a server-side failure after dispatch arrives asynchronously on the OnInsightFn callback as Insight::error(). - ```cpp CameraBuilder UseCamera(int device_index = 0) ``` Video source (call before Start). Simple verbs that return builders for optional config. Only one source is active at a time — calling Use* again overwrites the previous choice. ```cpp // Typed camera selection (omit source configuration for legacy index zero): spectra.UseCamera(CameraSelection::Default()).Build(); // 1280x720, 30fps spectra.UseCamera(2).SetResolution(1920, 1080).SetFps(60).Build(); // Video file: spectra.UseFile("video.mp4").Build(); spectra.UseFile("video.mp4").SetTimestamps("ts.txt").SetStartOffset(5000).Build(); // Custom frame push (GStreamer, gRPC, etc.): std::shared_ptr input; if (auto err = spectra.UseCustomInput() .SetFrameTransform(FrameTransform::kRotate90CW) .Build(input); !err.ok()) { ... } if (auto err = spectra.Start(); !err.ok()) { ... } if (auto err = input->Send(frame, timestamp_us); !err.ok()) { ... } // input is a shared_ptr — safe even if spectra is destroyed. ``` All source builders support .SetFrameTransform() for input rotation/mirroring. Deprecated: integer/no-argument selection preserves legacy device indices. Use UseCamera(CameraSelection) and AvailableCameras() for new applications. - ```cpp CameraBuilder UseCamera(CameraSelection selection) ``` Configure a camera by intent or opaque ID. Build() validates the request; Start() resolves and opens it, returning kInputUnavailable if it is absent or cannot be opened. Explicit requests never fall back to another camera. Call while uninitialized or idle; this does not switch a running camera. Supported on Linux/macOS/iOS/Windows. Other platforms accept only Default(). - ```cpp static SmartSpectraError AvailableCameras(std::vector& out) ``` Discover cameras without constructing an SDK instance, authenticating, requesting permission, or starting capture. Replaces out on success and clears it on failure. An empty successful result means no visible cameras. Camera visibility depends on platform permissions; discovery does not guarantee that a camera can be opened later. Supported on Linux/macOS/iOS/Windows; returns kConfigurationFailed on other platforms in this release. - ```cpp VideoFileBuilder UseFile(const std::string& path) ``` - ```cpp CustomInputBuilder UseCustomInput() ``` - ```cpp ProcessingStatus GetStatus() const ``` Returns the current processing status. ## SmartSpectraConfig ### Properties - ```cpp std::string api_key ``` API key authentication (simplest mode — no attestation needed). - ```cpp std::vector requested_metrics ``` - ```cpp bool enable_accumulated_output = false ``` - ```cpp SmartSpectraLogLevel log_level = kDefaultLogLevel ``` Verbosity of SDK logging, applied when the instance initializes. Logging is process-wide: the most recently initialized instance's level is in effect. - ```cpp SdkRuntimeContext runtime_context ``` Low-cardinality labels describing the public SDK/runtime surface. Used only for aggregate SDK telemetry; no stable device/user identifiers. - ```cpp bool enable_telemetry = true ``` Opt out of aggregate SDK telemetry by setting this false. On by default. Telemetry is aggregate-only (no raw frames, metric values, file paths, user identifiers, or device IDs). When enabled, a per-session summary is POSTed to the authenticated continuous server — best-effort, never blocking a measurement session. - ```cpp std::optional video_output_directory ``` Directory for video output files written by graph calculators. - ```cpp static const std::vector& BreathingMetrics() ``` Predefined metric bundles. Each returns the standard set of metrics for its category; pass one (or combine several) into requested_metrics. DefaultSupportedMetrics is what the SDK measures when requested_metrics is left empty. - ```cpp static const std::vector& DefaultSupportedMetrics() ``` - ```cpp static const std::vector& CardioMetrics() ``` - ```cpp static const std::vector& FaceMetrics() ``` - ```cpp static const std::vector& EdaMetrics() ``` - ```cpp void AddMetrics(const std::vector& metrics) ``` Add metrics to requested_metrics with deduplication. - ```cpp void RemoveMetrics(const std::vector& metrics) ``` Remove metrics from requested_metrics. ## CameraBuilder ### Methods - ```cpp CameraBuilder& SetResolution(int width, int height) ``` - ```cpp CameraBuilder& SetFps(int fps) ``` - ```cpp CameraBuilder& SetFrameTransform(FrameTransform transform) ``` - ```cpp [[nodiscard]] SmartSpectraError Build() ``` Applies the configuration and returns the result. ## VideoFileBuilder ### Methods - ```cpp VideoFileBuilder& SetTimestamps(const std::string& path) ``` - ```cpp VideoFileBuilder& SetInterframeDelay(int ms) ``` - ```cpp VideoFileBuilder& SetStartOffset(int ms) ``` - ```cpp VideoFileBuilder& SetMaxDuration(int ms) ``` - ```cpp VideoFileBuilder& SetFrameTransform(FrameTransform transform) ``` - ```cpp [[nodiscard]] SmartSpectraError Build() ``` Applies the configuration and returns the result. ## CustomInputBuilder ### Methods - ```cpp CustomInputBuilder& SetFrameTransform(FrameTransform transform) ``` - ```cpp [[nodiscard]] SmartSpectraError Build(std::shared_ptr& out) ``` Creates the custom input handle. On success `out` holds a valid handle and Ok is returned; on failure `out` is unchanged and the error describes the cause. Building twice from the same builder fails on the second call. ## CustomInput ### Methods - ```cpp virtual SmartSpectraError Send(const FrameBuffer& frame, int64_t timestamp_us) = 0 ``` Submits a frame for processing. The frame is consumed synchronously — its pixels are copied before Send() returns — so you only need to keep `frame` and its backing memory valid for the duration of the call; nothing is retained afterward. (Callers passing borrowed plane pointers, e.g. an Android ImageProxy, rely on this.) Inspect the returned error: kNonMonotonicTimestamp and kTimestampGap reject only this frame, leave the session active, and do not invoke SetOnError. Rejected timestamps do not advance the timeline; stop and start to begin a fresh timeline. ## FrameBuffer Non-owning view of a raw pixel buffer. Passed to CustomInput::Send(). For planar YUV formats (kNV12, kNV21): `data` points to the Y plane; the UV/VU plane immediately follows at `data + stride_bytes * height`. `stride_bytes` is the Y-plane stride; the chroma plane stride is equal. This layout is produced by most camera APIs (AImage, CVPixelBuffer, V4L2). ### Properties - ```cpp const uint8_t* data ``` - ```cpp int width ``` - ```cpp int height ``` - ```cpp int stride_bytes ``` - ```cpp PixelFormat format ``` ## FrameTransform Spatial transform applied to each frame before it enters the graph. - `kNone = 0` - `kRotate90CW = 1` - `kRotate90CCW = 2` - `kRotate180 = 3` - `kMirrorHorizontal = 4` - `kMirrorVertical = 5` ## PixelFormat Pixel format of a raw frame buffer passed to CustomInput::Send(). - `kRGB = 0` - `kBGR = 1` - `kRGBA = 2` - `kBGRA = 3` - `kNV12 = 4` - `kNV21 = 5` - `kYUYV = 6` ## ProcessingStatus Processing lifecycle status, reported via OnProcessingStatusChangedFn and GetStatus(). - `kUninitialized = 0` - `kIdle = 1` - `kStarting = 2` - `kRunning = 3` - `kStopping = 4` - `kError = 5` ## ValidationStatus Measurement-readiness status delivered via OnValidationStatusChangedFn. ### Properties - ```cpp ValidationCode code = ValidationCode::kOk ``` - ```cpp std::string hint ``` ## ValidationCode Measurement-readiness codes delivered via OnValidationStatusChangedFn. - `kOk = 0` - `kNoFaceFound = 1` - `kMultipleFacesFound = 2` - `kFaceNotCentered = 3` - `kFaceSizeOutOfRange = 4` - `kTooDark = 5` - `kTooBright = 6` - `kChestNotVisible = 7` - `kCameraTuning = 10` - `kFrameRateTooLow = 11` - `kExcessiveMotion = 12` - `kFaceTooClose = 13` - `kFaceTooFar = 14` - `kFaceTooHigh = 15` - `kFaceTooLow = 16` - `kFaceNotForward = 17` ## SmartSpectraError ### Properties - ```cpp SmartSpectraErrorCode code = SmartSpectraErrorCode::kOk ``` - ```cpp std::string message ``` - ```cpp bool retryable = false ``` ## SmartSpectraErrorCode Error codes returned by synchronous API calls and delivered via OnErrorFn. - `kOk = 0` - `kInvalidState = 1` - `kAuthenticationFailed = 2` - `kConfigurationFailed = 3` - `kCreditExhausted = 4` - `kNetworkError = 5` - `kServerError = 6` - `kInputUnavailable = 7` - `kProcessingFailed = 8` - `kFrameConversionFailed = 9` - `kNonMonotonicTimestamp = 10` - `kTimestampGap = 11` ## OnProcessingStatusChangedFn All callbacks fire on internal worker threads, not the thread that called Start(). Do not call SmartSpectra methods from within a callback without external synchronization — the internal mutex is not re-entrant. ```cpp using OnProcessingStatusChangedFn = std::function ``` ## OnValidationStatusChangedFn ```cpp using OnValidationStatusChangedFn = std::function ``` ## OnMetricsFn ```cpp using OnMetricsFn = std::function ``` ## OnAccumulatedMetricsFn ```cpp using OnAccumulatedMetricsFn = std::function ``` ## OnVideoOutputFn ```cpp using OnVideoOutputFn = std::function ``` ## OnFrameSentThroughFn ```cpp using OnFrameSentThroughFn = std::function ``` ## OnErrorFn ```cpp using OnErrorFn = std::function ``` ## OnInsightFn Delivered asynchronously after RequestInsight() dispatches successfully and for auto-fired periodic VITALS responses. Match the response to the prompt via Insight::request_id(); the payload is either Insight::analysis() on success or Insight::error() on failure. ```cpp using OnInsightFn = std::function ``` ## CameraSelection A camera request, resolved when capture starts. Explicit requests never fall back to a different facing or ID. This value does not open or reserve a device. ### Methods - ```cpp static CameraSelection Default() ``` Use the platform's preferred camera. On iOS prefer front, then the first discovered camera; on macOS and Windows use discovery order. Linux uses the first compatible V4L2 capture node in numeric device order. - ```cpp static CameraSelection Front() ``` Require a front-facing camera. - ```cpp static CameraSelection Back() ``` Require a back-facing camera. - ```cpp static CameraSelection ById(std::string id) ``` Require this exact ID from AvailableCameras(). An empty ID is invalid. ## CameraInfo A snapshot of a discoverable camera. IDs are opaque, platform-local values; rediscover devices before presenting a picker or reusing a saved ID. ### Properties - ```cpp std::string id ``` - ```cpp std::optional name ``` - ```cpp CameraFacing facing = CameraFacing::kUnknown ``` - ```cpp CameraLensType lens_type = CameraLensType::kUnknown ``` ## CameraFacing Camera direction reported by the platform. External cameras may be unknown. - `kFront` - `kBack` - `kUnknown` ## CameraLensType Best-effort lens classification. Unknown includes unavailable metadata and cameras that combine multiple lenses. It does not describe digital zoom. - `kUnknown` - `kWideAngle` - `kUltraWide` - `kTelephoto` ## CameraSelection::Kind - `kDefault` - `kFront` - `kBack` - `kById` ## SmartSpectraLogLevel Verbosity of SDK logging, set once at startup via SmartSpectraConfig::log_level. Levels are cumulative: a level shows its own messages plus everything more severe. kDebug additionally enables verbose diagnostics. - `kDebug = 0` - `kInfo = 1` - `kWarning = 2` - `kError = 3` - `kNone = 4` ## SdkRuntimeContext Low-cardinality runtime labels for aggregate SDK telemetry. These describe where the public SDK surface is running well/okay/failing without carrying stable device, user, or installation identifiers. ### Properties - ```cpp std::string sdk_binding ``` - ```cpp std::string platform ``` - ```cpp std::string os_version ``` - ```cpp std::string device_model ``` ## Headless Mode on C++ Source: https://presagetech.com/docs/smartspectra-sdk/cpp/headless-mode --- title: Headless Mode on C++ description: The SmartSpectra C++ SDK is headless by default — wire up metric and frame callbacks for custom integrations. sidebarTitle: Headless Mode --- # Headless Mode (C++) The SDK doesn't ship UI. Register lambdas with `SetOnMetrics`, `SetOnVideoOutput`, and `SetOnError`; rendering is your code's job. The C++ sample apps show reference UI implementations. Use this when you want to: - Render metrics in your own UI - Process metrics with no UI at all (logging, server-side, batch) - Feed your own frames instead of the SDK's built-in camera ## Processing Status Lifecycle states, reported via `OnProcessingStatusChangedFn` and `GetStatus()`: | Status | Value | Meaning | | --- | --- | --- | | `kUninitialized` | 0 | SDK constructed but not yet initialized | | `kIdle` | 1 | Pipeline is not running | | `kStarting` | 2 | Pipeline is initializing | | `kRunning` | 3 | Actively measuring — data is flowing | | `kStopping` | 4 | Teardown in progress, will return to `kIdle` | | `kError` | 5 | Something went wrong | ## Example ```cpp namespace spectra = presage::smartspectra; spectra::SmartSpectraConfig config; config.api_key = "YOUR_API_KEY"; config.requested_metrics = spectra::SmartSpectraConfig::BreathingMetrics(); spectra::SmartSpectra sdk(config); sdk.SetOnMetrics([](const spectra::Metrics& metrics, int64_t ts) { // Process metrics }); sdk.SetOnVideoOutput([](const spectra::FrameBuffer& frame, int64_t ts) { // Optional: render frame in your own UI }); if (const auto source_error = sdk.UseCamera(spectra::CameraSelection::Default()) .SetResolution(1280, 720) .SetFps(30) .Build(); !source_error.ok()) { // Handle setup error } else if (const auto err = sdk.Start(); !err.ok()) { // Handle startup error } // ... run until done ... sdk.Stop(); ``` ## Custom frame input For custom frame input instead of the built-in camera: ```cpp std::shared_ptr handle; if (auto err = sdk.UseCustomInput().Build(handle); !err.ok()) { // Handle setup error: err.FullMessage() } // Feed frames manually: // handle->Send(frame, timestamp_us); // timestamp_us must be strictly monotonic. ``` ## Reading Metrics `SetOnMetrics` fires the same way regardless of frame source. See [C++ Metrics](metrics.md) for the metric request configuration and the metric catalog. ## Running Headlessly in CI Feed a recorded video through `UseFile` instead of a live camera to run an unattended measurement in a CI pipeline — see [Headless Testing in CI](../../docs/headless-testing-in-ci.md) for the cross-platform model and prerequisites (API key, network access, your own recorded video). The `minimal_example` sample supports this out of the box: ```bash ./build/samples/minimal_example/minimal_example \ --api_key=YOUR_API_KEY --input_video_path=/path/to/video.mp4 ``` The sample plays the file through the same pipeline as a live camera, prints metrics as they arrive, and exits when the file finishes — a non-zero exit code (or no metrics printed) means the run failed. Use a 30–60 second clip (well-lit, mostly still face) in a widely-supported container/codec such as MP4 (H.264). A clip of only a few seconds will not produce readings: pulse rate needs 12 seconds, breathing rate starts at roughly 10 seconds and is not confident until 30, and HRV starts at roughly 30 seconds and is not confident until 60. A provider-neutral CI step (GitHub Actions), after installing or building the SDK for your platform: ```yaml - name: Headless measurement smoke run: | ./build/samples/minimal_example/minimal_example \ --api_key="${{ secrets.SMARTSPECTRA_API_KEY }}" \ --input_video_path=./test-assets/face.mp4 ``` There is no offline mode — the CI runner needs network access to the SmartSpectra service to authenticate and run a measurement. ## Debian 13 (Trixie) Source: https://presagetech.com/docs/smartspectra-sdk/cpp/linux/debian-13 --- title: Debian 13 (Trixie) description: Install the SmartSpectra C++ SDK on Debian 13 (Trixie) for amd64 and arm64. --- # SmartSpectra C++ Quickstart — Debian 13 (Trixie) > **Important:** SDK metrics are offered for general wellness and informational purposes only. SDK metrics have not been cleared by the FDA and may not be used for medical diagnosis or treatment. > > **Warning — Experimental platform:** Linux support for the SmartSpectra C++ > SDK is experimental. If you have any issues running SmartSpectra, > [contact Presage support](mailto:support@presagetech.com) for assistance. This guide covers the `trixie` apt suite, which supports both `amd64` and `arm64`. For Ubuntu and Linux Mint, choose the matching [Linux installation guide](index.md). ## Installation ### Prerequisites - **CMake 3.22.1 or later** (the version shipped with Debian 13 (Trixie) is sufficient) - **C++20 compiler** such as GCC or Clang - **Vulkan-capable graphics driver** — Linux builds use Vulkan inference by default. The SDK package installs the Vulkan loader dependency through apt, but the host must provide a working Vulkan driver. - **`build-essential`, `cmake`, `curl`, `gpg`, `git`, and `pkg-config`** — used by the build, install, and verify steps below. Install with `sudo apt update && sudo apt install build-essential cmake curl gpg git pkg-config` if they are not already present. - **API key** from [physiology.presagetech.com](https://physiology.presagetech.com/auth/login) ### Add the SDK Install the Presage signing key: ```bash sudo install -d -m 0755 /etc/apt/keyrings curl -fsSL https://packages.presagetech.com/KEY.gpg \ | sudo gpg --dearmor -o /etc/apt/keyrings/presage-archive-keyring.gpg sudo chmod 644 /etc/apt/keyrings/presage-archive-keyring.gpg ``` Add the `trixie` apt source: ```bash echo "deb [signed-by=/etc/apt/keyrings/presage-archive-keyring.gpg] https://packages.presagetech.com/apt/debian trixie main" \ | sudo tee /etc/apt/sources.list.d/presage-technologies.list ``` > **Installing an RC build?** Keep the stable source configured and follow the > [release-candidate channel](#release-candidate-channel) steps below. Install the SDK: ```bash sudo apt update sudo apt install libsmartspectra-dev ``` The `signed-by=` source entry scopes the Presage signing key to the Presage apt repository. APT selects the package matching your system's `dpkg --print-architecture` (`amd64` or `arm64`) automatically. The SmartSpectra SDK package is self-contained. You do not need to install protobuf, curl, OpenSSL, or other SDK runtime libraries separately. Verify that the package is visible to build tools: ```bash pkg-config --modversion SmartSpectra ``` The command prints the installed SDK version (for example, `3.3.0`). If it prints nothing or reports that the package is missing, reinstall `libsmartspectra-dev` and confirm you are on a supported Debian 13 (Trixie) (`amd64` or `arm64`) host. ## Example This quick start creates a minimal CMake project that links against the installed `SmartSpectra::SDK` package and reads from the default camera. You will create exactly these files: 1. `hello_vitals/hello_vitals.cpp` 2. `hello_vitals/CMakeLists.txt` ### Result you should get At the end, the app should show console output with: - a successful CMake configure and build - `Processing... Press Ctrl+C to stop.` - `Cardio metrics:` log lines when cardio metrics are available - `Breathing metrics:` log lines when breathing metrics are available - a clean exit after you stop the sample ![SmartSpectra C++ quickstart demo](../images/cpp-quickstart.gif) ### Step 1 - Get an API key 1. Open the Presage [Developer Admin Portal Registration](https://physiology.presagetech.com/auth/register). 2. Register or log in. 3. Copy your API key from the portal. ### Step 2 - Create the project directory ```bash mkdir hello_vitals cd hello_vitals ``` ### Step 3 - Create `hello_vitals.cpp` Open a new file named `hello_vitals.cpp` in your editor of choice and paste this entire file: This example also lives in `smartspectra/cpp/samples/hello_vitals/`. ```cpp #include #include #include #include #include #include #include #include #include namespace spectra = presage::smartspectra; namespace { volatile std::sig_atomic_t g_stop_requested = 0; void HandleSignal(int) { g_stop_requested = 1; } std::string ResolveApiKey(int argc, char** argv) { if (argc > 1) { return argv[1]; } if (const char* key = std::getenv("SMARTSPECTRA_API_KEY")) { return key; } return {}; } } // namespace int main(int argc, char** argv) { std::signal(SIGINT, HandleSignal); const std::string api_key = ResolveApiKey(argc, argv); if (api_key.empty()) { #if defined(_WIN32) std::cerr << "Usage: .\\hello_vitals.exe YOUR_API_KEY\n" << "or set SMARTSPECTRA_API_KEY=YOUR_API_KEY\n"; #else std::cerr << "Usage: ./hello_vitals YOUR_API_KEY\n" << "or export SMARTSPECTRA_API_KEY=YOUR_API_KEY\n"; #endif return 1; } spectra::SmartSpectraConfig config; config.api_key = api_key; config.requested_metrics = spectra::SmartSpectraConfig::BreathingMetrics(); config.AddMetrics(spectra::SmartSpectraConfig::CardioMetrics()); spectra::SmartSpectra sdk(config); sdk.SetOnMetrics([](const spectra::Metrics& metrics, int64_t) { if (metrics.has_cardio()) { std::cerr << "Cardio metrics: " << metrics.cardio().ShortDebugString() << "\n"; } if (metrics.has_breathing()) { std::cerr << "Breathing metrics: " << metrics.breathing().ShortDebugString() << "\n"; } }); sdk.SetOnValidationStatusChanged( [have_last_status = false, last_code = spectra::ValidationCode::kOk, last_hint = std::string{}](const spectra::ValidationStatus& status, int64_t) mutable { if (have_last_status && status.code == last_code && status.hint == last_hint) { return; } have_last_status = true; last_code = status.code; last_hint = status.hint; std::cerr << "Validation [" << status.code << "]: " << status.hint << "\n"; }); sdk.SetOnError([](const spectra::SmartSpectraError& error) { std::cerr << "Error [" << static_cast(error.code) << "]: " << error.message << "\n"; }); const auto source_error = sdk.UseCamera(spectra::CameraSelection::Default()) .SetResolution(1280, 720).SetFps(30).Build(); if (!source_error.ok()) { std::cerr << "Failed to create camera source: " << source_error.message << "\n"; return 1; } if (const auto err = sdk.Start(); !err.ok()) { std::cerr << "Failed to start: " << err.message << "\n"; return 1; } std::cout << "Processing... Press Ctrl+C to stop.\n"; while (!g_stop_requested) { std::this_thread::sleep_for(std::chrono::milliseconds(200)); } if (const auto err = sdk.Stop(); !err.ok()) { std::cerr << "Stop failed: " << err.message << "\n"; } return 0; } ``` The example requests breathing and cardio metrics. Add `FaceMetrics()` or other metric groups with `config.AddMetrics(...)` when your app expects those outputs. ### Step 4 - Create `CMakeLists.txt` Open a new file named `CMakeLists.txt` in the same directory and paste this entire file: ```cmake cmake_minimum_required(VERSION 3.22.1) project(SmartSpectraHelloVitals CXX) set(CMAKE_CXX_STANDARD 20) set(CMAKE_CXX_STANDARD_REQUIRED ON) find_package(SmartSpectra CONFIG REQUIRED) add_executable(hello_vitals hello_vitals.cpp) target_link_libraries(hello_vitals SmartSpectra::SDK) ``` ### Step 5 - Build ```bash cmake -S . -B build cmake --build build ``` A successful build produces the executable at `build/hello_vitals`. If CMake reports that it cannot locate `SmartSpectra`, rerun `pkg-config --modversion SmartSpectra` to confirm the SDK is installed correctly before continuing. ### Step 6 - Run Pass the API key as an argument: ```bash ./build/hello_vitals YOUR_API_KEY ``` Or set it once in your shell: ```bash export SMARTSPECTRA_API_KEY="YOUR_API_KEY" ./build/hello_vitals ``` Sit centered in front of the webcam, well-lit, and stay reasonably still. The app logs breathing and cardio metrics until you stop it with `Ctrl+C`. If no face is detected the app still runs and exits cleanly, but no metrics callbacks fire. An internet connection is required for subscription validation when using the standard SDK. ## What success looks like When your program is running, you should see all of these: - `Processing... Press Ctrl+C to stop.` prints after launch - the camera starts without a source creation error - `Cardio metrics:` or `Breathing metrics:` logs print while you sit centered and well-lit - the process exits after `Ctrl+C` without a `Stop failed` message ## Expected API key check The first measurement should start after the executable launches with a valid API key argument or `SMARTSPECTRA_API_KEY` environment variable. If startup fails with an authentication error, verify that the key is authorized for this app and that your shell did not include extra quotes or whitespace. ## Common manual mistakes If the console output does not match the target state, check these first: - the Presage apt source was added for the wrong distribution or suite - `libsmartspectra-dev` did not finish installing before CMake was run - the API key argument or `SMARTSPECTRA_API_KEY` environment variable is missing - another app is already using the camera - the binary is an older build from before the latest source change ## Running headless (Docker, CI, no desktop) Current SDK releases run without a desktop session in Docker, CI, and SSH-only environments. No alternate SDK package or additional system-service setup is required. ## Build the Provided Samples The SDK package does not install the sample source code. To build the repository samples against the installed SDK, clone the SmartSpectra repository after installing `libsmartspectra-dev`. Several of the samples use OpenCV for video capture and display, and CMake configures every sample in the tree even when you build a single target, so install the OpenCV development package first. The `hello_vitals` quickstart earlier on this page is a standalone project and does not need it. ```bash sudo apt install libopencv-dev git clone https://github.com/Presage-Security/SmartSpectra.git cd SmartSpectra/cpp/samples cmake -S . -B build -DCMAKE_BUILD_TYPE=Release cmake --build build --target minimal_example ``` Run a sample with your API key: ```bash ./build/minimal_example/minimal_example --api_key=YOUR_API_KEY ``` To run headlessly against a recording instead of the default camera, add `--input_video_path=/path/to/video.mp4`. ## Advanced apt workflows Most users only need the stable `trixie` repository above. Use these when you intentionally need release-candidate packages, version pinning, or repository removal. ### Pinning the installed SDK version To keep a working machine on the currently installed SDK version while you test or stage a rollout, hold the package: ```bash sudo apt-mark hold libsmartspectra-dev ``` Release the hold when you are ready to take SDK updates again: ```bash sudo apt-mark unhold libsmartspectra-dev ``` ### Release-candidate channel Release-candidate builds are published to the parallel `trixie-rc` apt suite signed by the same Presage key: ```bash echo "deb [signed-by=/etc/apt/keyrings/presage-archive-keyring.gpg] https://packages.presagetech.com/apt/debian trixie-rc main" \ | sudo tee /etc/apt/sources.list.d/presage-technologies-rc.list sudo apt update && sudo apt -t trixie-rc install libsmartspectra-dev ``` Keep the stable `trixie` source configured alongside `trixie-rc`; the RC channel does not republish stable releases. ### Returning from RC to stable ```bash sudo apt update sudo apt install --allow-downgrades --reinstall -t trixie libsmartspectra-dev=$(apt-cache madison libsmartspectra-dev | awk '/trixie\/main/ {print $3; exit}') sudo rm -f /etc/apt/sources.list.d/presage-technologies-rc.list sudo rm -f /etc/apt/preferences.d/presage-rc sudo apt update ``` ### Uninstalling the package ```bash sudo apt remove --purge libsmartspectra-dev sudo apt autoremove --purge sudo rm -f /etc/apt/sources.list.d/presage-technologies.list sudo rm -f /etc/apt/sources.list.d/presage-technologies-rc.list sudo rm -f /etc/apt/preferences.d/presage-rc sudo rm -f /etc/apt/keyrings/presage-archive-keyring.gpg sudo rm -f /etc/apt/trusted.gpg.d/presage-technologies.gpg sudo apt update ``` ## Next Steps - [Configure which metrics to compute](../metrics.md) - [Run headless without video output](../headless-mode.md) - [Migration Guide](../migration-guide.md) for upgrading from older SDK versions ## Documentation API reference available at [C++ API Reference](../api-reference.md). ## Troubleshooting If you are upgrading an older C++ integration, start with the [C++ Migration Guide](../migration-guide.md). ### APT cannot find the package Confirm that the host runs Debian 13 (`trixie`) and that `dpkg --print-architecture` prints `amd64` or `arm64`. Check that the Presage source uses `/apt/debian trixie main`, then rerun `sudo apt update`. Use the distribution-specific package; do not configure an Ubuntu suite on Debian. ## Getting Help - Email: [support@presagetech.com](mailto:support@presagetech.com) - [Submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues) - [Docs and FAQ](https://smartspectra.presagetech.com) - [Developer Admin Portal](https://physiology.presagetech.com/auth/login) ## Install the SmartSpectra C++ SDK on Linux Source: https://presagetech.com/docs/smartspectra-sdk/cpp/linux --- title: Install the SmartSpectra C++ SDK on Linux sidebarTitle: Overview description: Install the SmartSpectra C++ SDK on Debian, Ubuntu, and Linux Mint. Pick the guide that matches your distribution. --- # Overview > **Warning — Experimental platform:** Linux support for the SmartSpectra C++ > SDK is experimental. If you have any issues running SmartSpectra, > [contact Presage support](mailto:support@presagetech.com) for assistance. Pick the Presage apt repository and suite that match your host: - [**Ubuntu 22.04 / Mint 21**](ubuntu-22-04.md) — `jammy` suite, `amd64` and `arm64`. - [**Ubuntu 24.04 / Mint 22**](ubuntu-24-04.md) — `noble` suite, `amd64` and `arm64`. - [**Debian 13 / Trixie**](debian-13.md) — `trixie` suite, `amd64` and `arm64`. Each guide is end-to-end: prerequisites, repository setup, a minimal CMake project, headless-host behavior, and the advanced apt workflows (RC channel, pinning, uninstall) for that suite. ## Supported Platforms | Platform | Status | Notes | | -------- | ------ | ----- | | Ubuntu 22.04 / Mint 21 (amd64) | Experimental | Debian package available | | Ubuntu 22.04 / Mint 21 (arm64) | Experimental | Debian package available | | Ubuntu 24.04 / Mint 22 (amd64) | Experimental | Debian package available | | Ubuntu 24.04 / Mint 22 (arm64) | Experimental | Debian package available | | Debian 13 / Trixie (amd64) | Experimental | Debian package available | | Debian 13 / Trixie (arm64) | Experimental | Debian package available | | Debian 12 | Not supported | — | | RHEL 9 / Fedora 41 | Not supported | — | For platforms marked "Not supported" or anything not listed above, contact [support@presagetech.com](mailto:support@presagetech.com) if you have a specific need. ## Ubuntu 22.04 / Mint 21 Source: https://presagetech.com/docs/smartspectra-sdk/cpp/linux/ubuntu-22-04 --- title: Ubuntu 22.04 / Mint 21 description: Install the SmartSpectra C++ SDK on Ubuntu 22.04 or Linux Mint 21 (jammy) for amd64 and arm64. --- # SmartSpectra C++ Quickstart — Ubuntu 22.04 / Mint 21 > **Warning — Experimental platform:** Linux support for the SmartSpectra C++ > SDK is experimental. If you have any issues running SmartSpectra, > [contact Presage support](mailto:support@presagetech.com) for assistance. This guide covers the `jammy` apt suite, which supports both `amd64` and `arm64`. If you are on Ubuntu 24.04 / Mint 22, follow the [Ubuntu 24.04 / Mint 22 guide](ubuntu-24-04.md) instead. ## Installation ### Prerequisites - **CMake 3.22.1 or later** (the version shipped with Ubuntu 22.04 / Mint 21 is sufficient) - **C++20 compiler** such as GCC or Clang - **Vulkan-capable graphics driver** — Linux builds use Vulkan inference by default. The SDK package installs the Vulkan loader dependency through apt, but the host must provide a working Vulkan driver. - **`cmake`, `curl`, `gpg`, and `pkg-config`** — used by the build, install, and verify steps below. Install with `sudo apt install cmake curl gpg pkg-config` if they are not already present. - **API key** from [physiology.presagetech.com](https://physiology.presagetech.com/auth/login) ### Add the SDK Install the Presage signing key: ```bash sudo install -d -m 0755 /etc/apt/keyrings curl -fsSL https://packages.presagetech.com/KEY.gpg \ | sudo gpg --dearmor -o /etc/apt/keyrings/presage-archive-keyring.gpg sudo chmod 644 /etc/apt/keyrings/presage-archive-keyring.gpg ``` Add the `jammy` apt source: ```bash echo "deb [signed-by=/etc/apt/keyrings/presage-archive-keyring.gpg] https://packages.presagetech.com/apt/ubuntu jammy main" \ | sudo tee /etc/apt/sources.list.d/presage-technologies.list ``` > **Installing an RC build?** Keep the same signing-key setup, but use the > `jammy-rc` apt source instead of `jammy`: > > ```bash > echo "deb [signed-by=/etc/apt/keyrings/presage-archive-keyring.gpg] https://packages.presagetech.com/apt/ubuntu jammy-rc main" \ > | sudo tee /etc/apt/sources.list.d/presage-technologies.list > ``` > > Then run the same `sudo apt update` and > `sudo apt install libsmartspectra-dev` commands below. Install the SDK: ```bash sudo apt update sudo apt install libsmartspectra-dev ``` The `signed-by=` source entry scopes the Presage signing key to the Presage apt repository. APT selects the package matching your system's `dpkg --print-architecture` (`amd64` or `arm64`) automatically. The SmartSpectra SDK package is self-contained. You do not need to install protobuf, curl, OpenSSL, or other SDK runtime libraries separately. Verify that the package is visible to build tools: ```bash pkg-config --modversion SmartSpectra ``` The command prints the installed SDK version (for example, `3.3.0`). If it prints nothing or reports that the package is missing, reinstall `libsmartspectra-dev` and confirm you are on a supported Ubuntu 22.04 / Mint 21 (`amd64` or `arm64`) host. ## Example This quick start creates a minimal CMake project that links against the installed `SmartSpectra::SDK` package and reads from the default camera. You will create exactly these files: 1. `hello_vitals/hello_vitals.cpp` 2. `hello_vitals/CMakeLists.txt` ### Result you should get At the end, the app should show console output with: - a successful CMake configure and build - `Processing... Press Ctrl+C to stop.` - `Cardio metrics:` log lines when cardio metrics are available - `Breathing metrics:` log lines when breathing metrics are available - a clean exit after you stop the sample ![SmartSpectra C++ quickstart demo](../images/cpp-quickstart.gif) ### Step 1 - Get an API key 1. Open the Presage [Developer Admin Portal Registration](https://physiology.presagetech.com/auth/register). 2. Register or log in. 3. Copy your API key from the portal. ### Step 2 - Create the project directory ```bash mkdir hello_vitals cd hello_vitals ``` ### Step 3 - Create `hello_vitals.cpp` Open a new file named `hello_vitals.cpp` in your editor of choice and paste this entire file: This example also lives in `smartspectra/cpp/samples/hello_vitals/`. ```cpp #include #include #include #include #include #include #include #include #include namespace spectra = presage::smartspectra; namespace { volatile std::sig_atomic_t g_stop_requested = 0; void HandleSignal(int) { g_stop_requested = 1; } std::string ResolveApiKey(int argc, char** argv) { if (argc > 1) { return argv[1]; } if (const char* key = std::getenv("SMARTSPECTRA_API_KEY")) { return key; } return {}; } } // namespace int main(int argc, char** argv) { std::signal(SIGINT, HandleSignal); const std::string api_key = ResolveApiKey(argc, argv); if (api_key.empty()) { #if defined(_WIN32) std::cerr << "Usage: .\\hello_vitals.exe YOUR_API_KEY\n" << "or set SMARTSPECTRA_API_KEY=YOUR_API_KEY\n"; #else std::cerr << "Usage: ./hello_vitals YOUR_API_KEY\n" << "or export SMARTSPECTRA_API_KEY=YOUR_API_KEY\n"; #endif return 1; } spectra::SmartSpectraConfig config; config.api_key = api_key; config.requested_metrics = spectra::SmartSpectraConfig::BreathingMetrics(); config.AddMetrics(spectra::SmartSpectraConfig::CardioMetrics()); spectra::SmartSpectra sdk(config); sdk.SetOnMetrics([](const spectra::Metrics& metrics, int64_t) { if (metrics.has_cardio()) { std::cerr << "Cardio metrics: " << metrics.cardio().ShortDebugString() << "\n"; } if (metrics.has_breathing()) { std::cerr << "Breathing metrics: " << metrics.breathing().ShortDebugString() << "\n"; } }); sdk.SetOnValidationStatusChanged( [have_last_status = false, last_code = spectra::ValidationCode::kOk, last_hint = std::string{}](const spectra::ValidationStatus& status, int64_t) mutable { if (have_last_status && status.code == last_code && status.hint == last_hint) { return; } have_last_status = true; last_code = status.code; last_hint = status.hint; std::cerr << "Validation [" << status.code << "]: " << status.hint << "\n"; }); sdk.SetOnError([](const spectra::SmartSpectraError& error) { std::cerr << "Error [" << static_cast(error.code) << "]: " << error.message << "\n"; }); const auto source_error = sdk.UseCamera(spectra::CameraSelection::Default()) .SetResolution(1280, 720).SetFps(30).Build(); if (!source_error.ok()) { std::cerr << "Failed to create camera source: " << source_error.message << "\n"; return 1; } if (const auto err = sdk.Start(); !err.ok()) { std::cerr << "Failed to start: " << err.message << "\n"; return 1; } std::cout << "Processing... Press Ctrl+C to stop.\n"; while (!g_stop_requested) { std::this_thread::sleep_for(std::chrono::milliseconds(200)); } if (const auto err = sdk.Stop(); !err.ok()) { std::cerr << "Stop failed: " << err.message << "\n"; } return 0; } ``` The example requests breathing and cardio metrics. Add `FaceMetrics()` or other metric groups with `config.AddMetrics(...)` when your app expects those outputs. ### Step 4 - Create `CMakeLists.txt` Open a new file named `CMakeLists.txt` in the same directory and paste this entire file: ```cmake cmake_minimum_required(VERSION 3.22.1) project(SmartSpectraHelloVitals CXX) set(CMAKE_CXX_STANDARD 20) set(CMAKE_CXX_STANDARD_REQUIRED ON) find_package(SmartSpectra CONFIG REQUIRED) add_executable(hello_vitals hello_vitals.cpp) target_link_libraries(hello_vitals SmartSpectra::SDK) ``` ### Step 5 - Build ```bash cmake -S . -B build cmake --build build ``` A successful build produces the executable at `build/hello_vitals`. If CMake reports that it cannot locate `SmartSpectra`, rerun `pkg-config --modversion SmartSpectra` to confirm the SDK is installed correctly before continuing. ### Step 6 - Run Pass the API key as an argument: ```bash ./build/hello_vitals YOUR_API_KEY ``` Or set it once in your shell: ```bash export SMARTSPECTRA_API_KEY="YOUR_API_KEY" ./build/hello_vitals ``` Sit centered in front of the webcam, well-lit, and stay reasonably still. The app logs breathing and cardio metrics until you stop it with `Ctrl+C`. If no face is detected the app still runs and exits cleanly, but no metrics callbacks fire. An internet connection is required for subscription validation when using the standard SDK. ## What success looks like When your program is running, you should see all of these: - `Processing... Press Ctrl+C to stop.` prints after launch - the camera starts without a source creation error - `Cardio metrics:` or `Breathing metrics:` logs print while you sit centered and well-lit - the process exits after `Ctrl+C` without a `Stop failed` message ## Expected API key check The first measurement should start after the executable launches with a valid API key argument or `SMARTSPECTRA_API_KEY` environment variable. If startup fails with an authentication error, verify that the key is authorized for this app and that your shell did not include extra quotes or whitespace. ## Common manual mistakes If the console output does not match the target state, check these first: - the Presage apt source was added for the wrong Ubuntu or Mint suite - `libsmartspectra-dev` did not finish installing before CMake was run - the API key argument or `SMARTSPECTRA_API_KEY` environment variable is missing - another app is already using the camera - the binary is an older build from before the latest source change ## Running headless (Docker, CI, no desktop) Current SDK releases run without a desktop session in Docker, CI, and SSH-only environments. No alternate SDK package or additional system-service setup is required. ## Build the Provided Samples The SDK package does not install the sample source code. To build the repository samples against the installed SDK, clone the SmartSpectra repository after installing `libsmartspectra-dev`. Several of the samples use OpenCV for video capture and display, and CMake configures every sample in the tree even when you build a single target, so install the OpenCV development package first. The `hello_vitals` quickstart earlier on this page is a standalone project and does not need it. ```bash sudo apt install libopencv-dev git clone https://github.com/Presage-Security/SmartSpectra.git cd SmartSpectra/cpp/samples cmake -S . -B build -DCMAKE_BUILD_TYPE=Release cmake --build build --target minimal_example ``` Run a sample with your API key: ```bash ./build/minimal_example/minimal_example --api_key=YOUR_API_KEY ``` To run headlessly against a recording instead of the default camera, add `--input_video_path=/path/to/video.mp4`. ## Advanced apt workflows Most users only need the stable `jammy` repository above. Use these when you intentionally need release-candidate packages, version pinning, or repository removal. ### Pinning the installed SDK version To keep a working machine on the currently installed SDK version while you test or stage a rollout, hold the package: ```bash sudo apt-mark hold libsmartspectra-dev ``` Release the hold when you are ready to take SDK updates again: ```bash sudo apt-mark unhold libsmartspectra-dev ``` ### Release-candidate channel Release-candidate builds are published to the parallel `jammy-rc` apt suite signed by the same Presage key: ```bash echo "deb [signed-by=/etc/apt/keyrings/presage-archive-keyring.gpg] https://packages.presagetech.com/apt/ubuntu jammy-rc main" \ | sudo tee /etc/apt/sources.list.d/presage-technologies-rc.list sudo apt update && sudo apt -t jammy-rc install libsmartspectra-dev ``` Keep the stable `jammy` source configured alongside `jammy-rc`; the RC channel does not republish stable releases. ### Returning from RC to stable ```bash sudo apt update sudo apt install --reinstall -t jammy libsmartspectra-dev=$(apt-cache madison libsmartspectra-dev | awk '/jammy\/main/ {print $3; exit}') sudo rm -f /etc/apt/sources.list.d/presage-technologies-rc.list sudo rm -f /etc/apt/preferences.d/presage-rc sudo apt update ``` ### Uninstalling the package ```bash sudo apt remove --purge libsmartspectra-dev sudo apt autoremove --purge sudo rm -f /etc/apt/sources.list.d/presage-technologies.list sudo rm -f /etc/apt/sources.list.d/presage-technologies-rc.list sudo rm -f /etc/apt/preferences.d/presage-rc sudo rm -f /etc/apt/keyrings/presage-archive-keyring.gpg sudo rm -f /etc/apt/trusted.gpg.d/presage-technologies.gpg sudo apt update ``` ## Next Steps - [Configure which metrics to compute](../metrics.md) - [Run headless without video output](../headless-mode.md) - [Migration Guide](../migration-guide.md) for upgrading from older SDK versions ## Documentation API reference available at [C++ API Reference](../api-reference.md). ## Troubleshooting If you are upgrading an older C++ integration, start with the [C++ Migration Guide](../migration-guide.md). ### Debian `Signed-By` conflict Older Debian instructions installed the Presage key in `/etc/apt/trusted.gpg.d/` and used a source line without `signed-by=`. If `apt update` reports `E: Conflicting values set for option Signed-By regarding source https://packages.presagetech.com/apt/ubuntu/ jammy`, remove the legacy key copy and run `apt update` again: ```bash sudo rm -f /etc/apt/trusted.gpg.d/presage-technologies.gpg sudo apt update ``` For support: contact [support@presagetech.com](mailto:support@presagetech.com) or [submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues). ## Ubuntu 24.04 / Mint 22 Source: https://presagetech.com/docs/smartspectra-sdk/cpp/linux/ubuntu-24-04 --- title: Ubuntu 24.04 / Mint 22 description: Install the SmartSpectra C++ SDK on Ubuntu 24.04 or Linux Mint 22 (noble) for amd64 and arm64. --- # SmartSpectra C++ Quickstart — Ubuntu 24.04 / Mint 22 > **Warning — Experimental platform:** Linux support for the SmartSpectra C++ > SDK is experimental. If you have any issues running SmartSpectra, > [contact Presage support](mailto:support@presagetech.com) for assistance. This guide covers the `noble` apt suite, which supports both `amd64` and `arm64`. If you are on Ubuntu 22.04 / Mint 21, follow the [Ubuntu 22.04 / Mint 21 guide](ubuntu-22-04.md) instead. ## Installation ### Prerequisites - **CMake 3.22.1 or later** (the version shipped with Ubuntu 24.04 / Mint 22 is sufficient) - **C++20 compiler** such as GCC or Clang - **Vulkan-capable graphics driver** — Linux builds use Vulkan inference by default. The SDK package installs the Vulkan loader dependency through apt, but the host must provide a working Vulkan driver. - **`cmake`, `curl`, `gpg`, and `pkg-config`** — used by the build, install, and verify steps below. Install with `sudo apt install cmake curl gpg pkg-config` if they are not already present. - **API key** from [physiology.presagetech.com](https://physiology.presagetech.com/auth/login) ### Add the SDK Install the Presage signing key: ```bash sudo install -d -m 0755 /etc/apt/keyrings curl -fsSL https://packages.presagetech.com/KEY.gpg \ | sudo gpg --dearmor -o /etc/apt/keyrings/presage-archive-keyring.gpg sudo chmod 644 /etc/apt/keyrings/presage-archive-keyring.gpg ``` Add the `noble` apt source: ```bash echo "deb [signed-by=/etc/apt/keyrings/presage-archive-keyring.gpg] https://packages.presagetech.com/apt/ubuntu noble main" \ | sudo tee /etc/apt/sources.list.d/presage-technologies.list ``` > **Installing an RC build?** Keep the same signing-key setup, but use the > `noble-rc` apt source instead of `noble`: > > ```bash > echo "deb [signed-by=/etc/apt/keyrings/presage-archive-keyring.gpg] https://packages.presagetech.com/apt/ubuntu noble-rc main" \ > | sudo tee /etc/apt/sources.list.d/presage-technologies.list > ``` > > Then run the same `sudo apt update` and > `sudo apt install libsmartspectra-dev` commands below. Install the SDK: ```bash sudo apt update sudo apt install libsmartspectra-dev ``` The `signed-by=` source entry scopes the Presage signing key to the Presage apt repository. APT selects the package matching your system's `dpkg --print-architecture` (`amd64` or `arm64`) automatically. The SmartSpectra SDK package is self-contained. You do not need to install protobuf, curl, OpenSSL, or other SDK runtime libraries separately. Verify that the package is visible to build tools: ```bash pkg-config --modversion SmartSpectra ``` The command prints the installed SDK version (for example, `3.3.0`). If it prints nothing or reports that the package is missing, reinstall `libsmartspectra-dev` and confirm you are on a supported Ubuntu 24.04 / Mint 22 (`amd64` or `arm64`) host. ## Example This quick start creates a minimal CMake project that links against the installed `SmartSpectra::SDK` package and reads from the default camera. You will create exactly these files: 1. `hello_vitals/hello_vitals.cpp` 2. `hello_vitals/CMakeLists.txt` ### Result you should get At the end, the app should show console output with: - a successful CMake configure and build - `Processing... Press Ctrl+C to stop.` - `Cardio metrics:` log lines when cardio metrics are available - `Breathing metrics:` log lines when breathing metrics are available - a clean exit after you stop the sample ![SmartSpectra C++ quickstart demo](../images/cpp-quickstart.gif) ### Step 1 - Get an API key 1. Open the Presage [Developer Admin Portal Registration](https://physiology.presagetech.com/auth/register). 2. Register or log in. 3. Copy your API key from the portal. ### Step 2 - Create the project directory ```bash mkdir hello_vitals cd hello_vitals ``` ### Step 3 - Create `hello_vitals.cpp` Open a new file named `hello_vitals.cpp` in your editor of choice and paste this entire file: This example also lives in `smartspectra/cpp/samples/hello_vitals/`. ```cpp #include #include #include #include #include #include #include #include #include namespace spectra = presage::smartspectra; namespace { volatile std::sig_atomic_t g_stop_requested = 0; void HandleSignal(int) { g_stop_requested = 1; } std::string ResolveApiKey(int argc, char** argv) { if (argc > 1) { return argv[1]; } if (const char* key = std::getenv("SMARTSPECTRA_API_KEY")) { return key; } return {}; } } // namespace int main(int argc, char** argv) { std::signal(SIGINT, HandleSignal); const std::string api_key = ResolveApiKey(argc, argv); if (api_key.empty()) { #if defined(_WIN32) std::cerr << "Usage: .\\hello_vitals.exe YOUR_API_KEY\n" << "or set SMARTSPECTRA_API_KEY=YOUR_API_KEY\n"; #else std::cerr << "Usage: ./hello_vitals YOUR_API_KEY\n" << "or export SMARTSPECTRA_API_KEY=YOUR_API_KEY\n"; #endif return 1; } spectra::SmartSpectraConfig config; config.api_key = api_key; config.requested_metrics = spectra::SmartSpectraConfig::BreathingMetrics(); config.AddMetrics(spectra::SmartSpectraConfig::CardioMetrics()); spectra::SmartSpectra sdk(config); sdk.SetOnMetrics([](const spectra::Metrics& metrics, int64_t) { if (metrics.has_cardio()) { std::cerr << "Cardio metrics: " << metrics.cardio().ShortDebugString() << "\n"; } if (metrics.has_breathing()) { std::cerr << "Breathing metrics: " << metrics.breathing().ShortDebugString() << "\n"; } }); sdk.SetOnValidationStatusChanged( [have_last_status = false, last_code = spectra::ValidationCode::kOk, last_hint = std::string{}](const spectra::ValidationStatus& status, int64_t) mutable { if (have_last_status && status.code == last_code && status.hint == last_hint) { return; } have_last_status = true; last_code = status.code; last_hint = status.hint; std::cerr << "Validation [" << status.code << "]: " << status.hint << "\n"; }); sdk.SetOnError([](const spectra::SmartSpectraError& error) { std::cerr << "Error [" << static_cast(error.code) << "]: " << error.message << "\n"; }); const auto source_error = sdk.UseCamera(spectra::CameraSelection::Default()) .SetResolution(1280, 720).SetFps(30).Build(); if (!source_error.ok()) { std::cerr << "Failed to create camera source: " << source_error.message << "\n"; return 1; } if (const auto err = sdk.Start(); !err.ok()) { std::cerr << "Failed to start: " << err.message << "\n"; return 1; } std::cout << "Processing... Press Ctrl+C to stop.\n"; while (!g_stop_requested) { std::this_thread::sleep_for(std::chrono::milliseconds(200)); } if (const auto err = sdk.Stop(); !err.ok()) { std::cerr << "Stop failed: " << err.message << "\n"; } return 0; } ``` The example requests breathing and cardio metrics. Add `FaceMetrics()` or other metric groups with `config.AddMetrics(...)` when your app expects those outputs. ### Step 4 - Create `CMakeLists.txt` Open a new file named `CMakeLists.txt` in the same directory and paste this entire file: ```cmake cmake_minimum_required(VERSION 3.22.1) project(SmartSpectraHelloVitals CXX) set(CMAKE_CXX_STANDARD 20) set(CMAKE_CXX_STANDARD_REQUIRED ON) find_package(SmartSpectra CONFIG REQUIRED) add_executable(hello_vitals hello_vitals.cpp) target_link_libraries(hello_vitals SmartSpectra::SDK) ``` ### Step 5 - Build ```bash cmake -S . -B build cmake --build build ``` A successful build produces the executable at `build/hello_vitals`. If CMake reports that it cannot locate `SmartSpectra`, rerun `pkg-config --modversion SmartSpectra` to confirm the SDK is installed correctly before continuing. ### Step 6 - Run Pass the API key as an argument: ```bash ./build/hello_vitals YOUR_API_KEY ``` Or set it once in your shell: ```bash export SMARTSPECTRA_API_KEY="YOUR_API_KEY" ./build/hello_vitals ``` Sit centered in front of the webcam, well-lit, and stay reasonably still. The app logs breathing and cardio metrics until you stop it with `Ctrl+C`. If no face is detected the app still runs and exits cleanly, but no metrics callbacks fire. An internet connection is required for subscription validation when using the standard SDK. ## What success looks like When your program is running, you should see all of these: - `Processing... Press Ctrl+C to stop.` prints after launch - the camera starts without a source creation error - `Cardio metrics:` or `Breathing metrics:` logs print while you sit centered and well-lit - the process exits after `Ctrl+C` without a `Stop failed` message ## Expected API key check The first measurement should start after the executable launches with a valid API key argument or `SMARTSPECTRA_API_KEY` environment variable. If startup fails with an authentication error, verify that the key is authorized for this app and that your shell did not include extra quotes or whitespace. ## Common manual mistakes If the console output does not match the target state, check these first: - the Presage apt source was added for the wrong Ubuntu or Mint suite - `libsmartspectra-dev` did not finish installing before CMake was run - the API key argument or `SMARTSPECTRA_API_KEY` environment variable is missing - another app is already using the camera - the binary is an older build from before the latest source change ## Running headless (Docker, CI, no desktop) Current SDK releases run without a desktop session in Docker, CI, and SSH-only environments. No alternate SDK package or additional system-service setup is required. ## Build the Provided Samples The SDK package does not install the sample source code. To build the repository samples against the installed SDK, clone the SmartSpectra repository after installing `libsmartspectra-dev`. Several of the samples use OpenCV for video capture and display, and CMake configures every sample in the tree even when you build a single target, so install the OpenCV development package first. The `hello_vitals` quickstart earlier on this page is a standalone project and does not need it. ```bash sudo apt install libopencv-dev git clone https://github.com/Presage-Security/SmartSpectra.git cd SmartSpectra/cpp/samples cmake -S . -B build -DCMAKE_BUILD_TYPE=Release cmake --build build --target minimal_example ``` Run a sample with your API key: ```bash ./build/minimal_example/minimal_example --api_key=YOUR_API_KEY ``` To run headlessly against a recording instead of the default camera, add `--input_video_path=/path/to/video.mp4`. ## Advanced apt workflows Most users only need the stable `noble` repository above. Use these when you intentionally need release-candidate packages, version pinning, or repository removal. ### Pinning the installed SDK version To keep a working machine on the currently installed SDK version while you test or stage a rollout, hold the package: ```bash sudo apt-mark hold libsmartspectra-dev ``` Release the hold when you are ready to take SDK updates again: ```bash sudo apt-mark unhold libsmartspectra-dev ``` ### Release-candidate channel Release-candidate builds are published to the parallel `noble-rc` apt suite signed by the same Presage key: ```bash echo "deb [signed-by=/etc/apt/keyrings/presage-archive-keyring.gpg] https://packages.presagetech.com/apt/ubuntu noble-rc main" \ | sudo tee /etc/apt/sources.list.d/presage-technologies-rc.list sudo apt update && sudo apt -t noble-rc install libsmartspectra-dev ``` Keep the stable `noble` source configured alongside `noble-rc`; the RC channel does not republish stable releases. ### Returning from RC to stable ```bash sudo apt update sudo apt install --reinstall -t noble libsmartspectra-dev=$(apt-cache madison libsmartspectra-dev | awk '/noble\/main/ {print $3; exit}') sudo rm -f /etc/apt/sources.list.d/presage-technologies-rc.list sudo rm -f /etc/apt/preferences.d/presage-rc sudo apt update ``` ### Uninstalling the package ```bash sudo apt remove --purge libsmartspectra-dev sudo apt autoremove --purge sudo rm -f /etc/apt/sources.list.d/presage-technologies.list sudo rm -f /etc/apt/sources.list.d/presage-technologies-rc.list sudo rm -f /etc/apt/preferences.d/presage-rc sudo rm -f /etc/apt/keyrings/presage-archive-keyring.gpg sudo rm -f /etc/apt/trusted.gpg.d/presage-technologies.gpg sudo apt update ``` ## Next Steps - [Configure which metrics to compute](../metrics.md) - [Run headless without video output](../headless-mode.md) - [Migration Guide](../migration-guide.md) for upgrading from older SDK versions ## Documentation API reference available at [C++ API Reference](../api-reference.md). ## Troubleshooting If you are upgrading an older C++ integration, start with the [C++ Migration Guide](../migration-guide.md). ### Debian `Signed-By` conflict Older Debian instructions installed the Presage key in `/etc/apt/trusted.gpg.d/` and used a source line without `signed-by=`. If `apt update` reports `E: Conflicting values set for option Signed-By regarding source https://packages.presagetech.com/apt/ubuntu/ noble`, remove the legacy key copy and run `apt update` again: ```bash sudo rm -f /etc/apt/trusted.gpg.d/presage-technologies.gpg sudo apt update ``` For support: contact [support@presagetech.com](mailto:support@presagetech.com) or [submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues). ## LLM Insights on C++ Source: https://presagetech.com/docs/smartspectra-sdk/cpp/llm-insights --- title: LLM Insights on C++ description: Ask natural-language questions about a measurement and receive LLM Insights through the SmartSpectra C++ SDK, alongside the metrics stream. sidebarTitle: LLM Insights --- # C++ LLM Insights Platform-specific usage for the C++ SDK. For what LLM Insights are, the request/response model, required metrics, and the privacy notice, see the [LLM Insights overview](../../docs/llm-insights/index.md). ## Enable the required metrics Insights summarize the buffered vitals, so breathing (the default set) and cardio must both be active: ```cpp spectra::SmartSpectraConfig config; config.api_key = my_api_key; config.AddMetrics(spectra::SmartSpectraConfig::DefaultSupportedMetrics()); // breathing defaults config.AddMetrics(spectra::SmartSpectraConfig::CardioMetrics()); // pulse, HRV, arterial pressure trace ``` `CardioMetrics()` includes `ARTERIAL_PRESSURE_TRACE`, which drives the on-screen pulse waveform. ## Receive responses Register a callback before starting. It receives **both** the auto-fired periodic vitals insights and on-demand responses; correlate an on-demand reply by matching `Insight::request_id()` against the ID you got from `RequestInsight`. ```cpp using presage::smartspectra::Insight; std::mutex insight_mutex; smart_spectra.SetOnInsight([&](const Insight& insight) { std::lock_guard lock(insight_mutex); // callback runs on a background thread if (insight.has_analysis()) { // insight.analysis() — the LLM text; insight.request_id() correlates the reply } else if (insight.has_error()) { // insight.error() — failure message } }); ``` `OnInsightFn` is `std::function`. The callback is invoked on a **background thread**, so synchronize any state it shares with your application. ## Request an insight Call `RequestInsight` on a running session. The response arrives asynchronously through the callback above; correlate it via `Insight::request_id()`. ```cpp [[nodiscard]] SmartSpectraError RequestInsight( const std::string& text, int32_t* out_request_id = nullptr); ``` ```cpp int32_t request_id = 0; if (const auto err = smart_spectra.RequestInsight("Summarize my current vital signs and flag anything unusual.", &request_id); !err.ok()) { std::cerr << err.FullMessage() << '\n'; } ``` - `text` — the prompt. Combined with the latest buffered metrics when they exist, otherwise sent prompt-only. - `out_request_id` — if non-null, receives the request ID for correlation. - Returns `kInvalidState` (no active session) or `kProcessingFailed` (dispatch failed — e.g. no insight callback registered, or a server error). ## Reading the Insight Branch on `has_analysis()` / `has_error()` (exactly one is set), read the text with `analysis()` / `error()`, and correlate with `request_id()`. Every insight is currently delivered with `type()` == `INSIGHT_TYPE_VITALS` (`SPEECH` and `COMBINED` are reserved), so use `request_id()`, not `type()`, to distinguish on-demand replies from auto-fired vitals. Full field documentation is in [Data Types → Insight](../../docs/data-types.md#insight). The first auto-fired insight arrives once pulse rate has a stable reading, typically about 15 seconds after the session starts; further snapshots follow as breathing rate and HRV stabilise, then every 60 seconds. Allow at least 15 seconds of valid measurement before an on-demand request can be grounded in the user's physiology. ## Complete examples Each example below is a minimal, self-contained program covering the full flow — SDK init, metric config, a thread-safe `SetOnInsight` sink, an on-demand `RequestInsight`, and `analysis()`/`error()` handling. They are condensed for the docs; the linked sample apps are the full, buildable versions. ### Linux — CLI A console app: type a prompt, press Enter, and the response prints when it arrives. The insight callback runs on a **background thread**, so shared state is mutex-guarded. ```cpp // insights_cli.cc — minimal SmartSpectra LLM Insights example (Linux). #include #include #include #include #include #include #include namespace spectra = presage::smartspectra; int main(int argc, char** argv) { if (argc < 2) { std::cerr << "usage: insights_cli \n"; return 1; } // 1-2. Init + metrics: breathing (defaults) + cardio must both be active. spectra::SmartSpectraConfig config; config.api_key = argv[1]; config.AddMetrics(spectra::SmartSpectraConfig::DefaultSupportedMetrics()); config.AddMetrics(spectra::SmartSpectraConfig::CardioMetrics()); spectra::SmartSpectra smart_spectra(std::move(config)); // 3. Receive responses. The callback runs on a background thread; guard shared // state. Correlate on-demand replies via request_id(). std::mutex insight_mutex; smart_spectra.SetOnInsight([&](const spectra::Insight& insight) { std::lock_guard lock(insight_mutex); if (insight.has_analysis()) { std::cout << "\n[insight #" << insight.request_id() << "] " << insight.analysis() << "\n> " << std::flush; } else if (insight.has_error()) { std::cerr << "\n[insight error] " << insight.error() << '\n'; } }); smart_spectra.SetOnError([](const spectra::SmartSpectraError& err) { std::cerr << err.FullMessage() << '\n'; }); if (const auto err = smart_spectra .UseCamera(spectra::CameraSelection::Default()) .Build(); !err.ok()) { std::cerr << err.FullMessage() << '\n'; return 1; } if (const auto err = smart_spectra.Start(); !err.ok()) { std::cerr << err.FullMessage() << '\n'; return 1; } // 4-5. Type a prompt + Enter to request an insight; replies print above. std::cout << "Type a prompt and press Enter (empty line quits).\n> " << std::flush; std::string prompt; while (std::getline(std::cin, prompt) && !prompt.empty()) { int32_t request_id = 0; if (const auto err = smart_spectra.RequestInsight(prompt, &request_id); !err.ok()) { std::cerr << err.FullMessage() << '\n'; } } (void)smart_spectra.Stop(); return 0; } ``` Full runnable sample: [`cpp/samples/insights_example`](https://github.com/Presage-Security/SmartSpectra/tree/main/cpp/samples/insights_example). ### Windows — WinUI3 / C++WinRT `SetOnInsight` fires on a background thread; XAML must be touched only on the UI thread. Capture the UI `DispatcherQueue` up front and marshal the update with `TryEnqueue`. Members (declared in `MainWindow.xaml.h`): ```cpp std::unique_ptr m_spectra; winrt::Microsoft::UI::Dispatching::DispatcherQueue m_ui_queue{ nullptr }; std::thread m_start_thread; ``` ```cpp // MainWindow.xaml.cpp (excerpt) — WinUI 3 / C++WinRT. namespace spectra = presage::smartspectra; MainWindow::MainWindow() { InitializeComponent(); // Capture the UI-thread dispatcher so callbacks can marshal back to it. m_ui_queue = DispatcherQueue::GetForCurrentThread(); // 1-2. Init + metrics: breathing (defaults) + cardio. spectra::SmartSpectraConfig cfg; cfg.api_key = ApiKey(); // supply your key cfg.AddMetrics(spectra::SmartSpectraConfig::DefaultSupportedMetrics()); cfg.AddMetrics(spectra::SmartSpectraConfig::CardioMetrics()); m_spectra = std::make_unique(std::move(cfg)); (void)m_spectra->UseCamera(spectra::CameraSelection::Default()).Build(); // 3. Receive responses. Hop to the UI thread with TryEnqueue before touching // XAML. Hold a weak window ref so teardown can release it. auto weak = get_weak(); m_spectra->SetOnInsight([weak](spectra::Insight const& insight) { winrt::hstring text; if (insight.has_analysis()) text = winrt::to_hstring(insight.analysis()); else if (insight.has_error()) text = L"Error: " + winrt::to_hstring(insight.error()); else return; if (auto self = weak.get()) { self->m_ui_queue.TryEnqueue([weak, text] { if (auto self = weak.get()) self->InsightText().Text(text); }); } }); // Start() blocks on authentication and model loading, so keep it off the UI // thread — the window would not paint until it returned. m_start_thread = std::thread([this] { (void)m_spectra->Start(); }); } MainWindow::~MainWindow() { if (m_start_thread.joinable()) m_start_thread.join(); (void)m_spectra->Stop(); } // 4-5. Button handler — request an insight; the reply arrives via SetOnInsight. void MainWindow::OnInsightClick(IInspectable const&, RoutedEventArgs const&) { int32_t request_id = 0; if (auto err = m_spectra->RequestInsight( "Summarize my current vital signs and flag anything unusual.", &request_id); !err.ok()) { InsightText().Text(L"Error: " + winrt::to_hstring(err.FullMessage())); } } ``` Full runnable sample: [`cpp/samples/winui3_example`](https://github.com/Presage-Security/SmartSpectra/tree/main/cpp/samples/winui3_example). ### macOS — SwiftUI A SwiftUI app consumes the C++ SDK through an Objective-C++ bridge. The bridge registers the insight sink, marshals each response onto the main queue, and forwards it to a delegate; the SwiftUI model publishes it. Bridge interface (`SmartSpectraRunner.h`): ```objc @protocol SmartSpectraRunnerDelegate // ...existing callbacks... - (void)smartSpectraRunnerDidUpdateInsight:(NSString *)analysis requestId:(int32_t)requestId; - (void)smartSpectraRunnerDidFailInsight:(NSString *)message; @end @interface SmartSpectraRunner : NSObject // ...existing start/stop... - (int32_t)requestInsight:(NSString *)prompt; // returns request id, or -1 on failure @end ``` Bridge implementation (`SmartSpectraRunner.mm`, `#include `): ```objc namespace ss = presage::smartspectra; // 1-2. In config setup: breathing (defaults) + cardio. config.requested_metrics = ss::SmartSpectraConfig::DefaultSupportedMetrics(); config.AddMetrics(ss::SmartSpectraConfig::CardioMetrics()); // 3. Insight sink — fires on a background thread, so marshal to the main queue // before forwarding to the delegate (which drives SwiftUI state). spectra->SetOnInsight([weakSelf](const ss::Insight& insight) { if (insight.has_analysis()) { NSString *analysis = [NSString stringWithUTF8String:insight.analysis().c_str()]; int32_t request_id = insight.request_id(); dispatch_async(dispatch_get_main_queue(), ^{ SmartSpectraRunner *runner = weakSelf; [runner.delegate smartSpectraRunnerDidUpdateInsight:analysis requestId:request_id]; }); } else if (insight.has_error()) { NSString *message = [NSString stringWithUTF8String:insight.error().c_str()]; dispatch_async(dispatch_get_main_queue(), ^{ SmartSpectraRunner *runner = weakSelf; [runner.delegate smartSpectraRunnerDidFailInsight:message]; }); } }); // 4. Request an insight (spectra_ is the runner's std::unique_ptr). - (int32_t)requestInsight:(NSString *)prompt { std::lock_guard lock(mutex_); if (!spectra_) return -1; int32_t request_id = -1; if (auto err = spectra_->RequestInsight(std::string(prompt.UTF8String), &request_id); !err.ok()) { return -1; } return request_id; } ``` SwiftUI model (`AppModel.swift`, an `ObservableObject` conforming to `SmartSpectraRunnerDelegate`): ```swift @Published var insight = "Ask AI to analyze your vitals." func requestInsight() { let requestId = runner.requestInsight( "Summarize my current vital signs and flag anything unusual.") insight = requestId < 0 ? "Insight request failed." : "Analyzing… (request #\(requestId))" } // 5. Delegate callbacks — already marshaled to the main queue by the bridge. func smartSpectraRunnerDidUpdateInsight(_ analysis: String, requestId: Int32) { insight = analysis } func smartSpectraRunnerDidFailInsight(_ message: String) { insight = "Error: \(message)" } ``` View (`ContentView.swift`, with `@ObservedObject var model: AppModel`): ```swift Button("Ask AI") { model.requestInsight() } .disabled(!model.isRunning) Text(model.insight) .textSelection(.enabled) ``` Full sample app (metrics/preview; extend with the insight wiring above): [`cpp/samples/macos_swiftui_example`](https://github.com/Presage-Security/SmartSpectra/tree/main/cpp/samples/macos_swiftui_example). ## See also - [LLM Insights overview](../../docs/llm-insights/index.md) - [C++ API reference](api-reference.md) - [Data Types](../../docs/data-types.md) ## C++ on macOS Source: https://presagetech.com/docs/smartspectra-sdk/cpp/macos --- title: C++ on macOS description: Install the SmartSpectra C++ SDK and run the SwiftUI macOS sample app. sidebarTitle: macOS --- # SmartSpectra C++ Quickstart — macOS ## Supported Platforms | Platform | Status | Notes | | -------- | ------ | ----- | | macOS Apple Silicon (14.0+) | Supported | Homebrew package available | | macOS Intel | Not supported | — | For platforms not listed above, contact [support@presagetech.com](mailto:support@presagetech.com) if you have a specific need. ## Installation ### Prerequisites - **Xcode** with Command Line Tools (provides a C++20 toolchain and the Swift compiler) - **Homebrew** - **API key** from [physiology.presagetech.com](https://physiology.presagetech.com/auth/login) - **Apple Development signing identity** — required for SDK startup on macOS Install Xcode Command Line Tools if you do not have them: ```bash xcode-select --install ``` ### Add the SDK The Homebrew formula installs the SmartSpectra SDK and exposes its CMake package metadata. ```bash brew tap presage/smartspectra https://github.com/Presage-Security/homebrew-smartspectra brew install presage/smartspectra/smartspectra ``` For the release-candidate channel, install `smartspectra-rc` instead: ```bash brew unlink smartspectra brew install presage/smartspectra/smartspectra-rc ``` ### Verify the install After Homebrew finishes, confirm the SDK is wired up before opening Xcode: ```bash pkg-config --modversion SmartSpectra ls /opt/homebrew/lib/cmake/SmartSpectra ``` `pkg-config` should print the installed SDK version. The `ls` should list the `SmartSpectra` CMake config directory the formula installed (this is what `find_package(SmartSpectra CONFIG REQUIRED)` in your own CMake project will pick up — see "Installed Paths" below). If either fails, re-run `brew install` and check `brew doctor`. ### Permissions SmartSpectra's default macOS builds require a signed host app with the keychain entitlements needed for the SDK. This applies to SDK startup in general, including file-based processing. The SwiftUI sample below ships with the required entitlements already configured. For your own integrations, mirror the sample's `smartspectra_swift_ui.entitlements` file ([stable](https://github.com/Presage-Security/SmartSpectra/blob/main/cpp/samples/macos_swiftui_example/smartspectra_swift_ui.entitlements) | [rc](https://github.com/Presage-Security/SmartSpectra/blob/rc/cpp/samples/macos_swiftui_example/smartspectra_swift_ui.entitlements)). ## Example The recommended starting point is the **SmartSpectra SwiftUI macOS sample**, a native SwiftUI app that opens directly in Xcode and links against the Homebrew-installed SDK. It demonstrates camera capture, validation status, breathing/cardio metrics, and trend traces end to end. Source: - [Stable sample source](https://github.com/Presage-Security/SmartSpectra/tree/main/cpp/samples/macos_swiftui_example) - [RC sample source](https://github.com/Presage-Security/SmartSpectra/tree/rc/cpp/samples/macos_swiftui_example) ### Result you should get At the end, the sample app should show: - live camera preview - validation and processing status - breathing and cardio metric cards - waveform or trend traces for supported metrics - a start/stop control for processing ![SmartSpectra C++ macOS quickstart demo](images/macos-quickstart.gif) ### Clone and verify the environment Match the branch to the Homebrew formula you installed: `main` for the stable `smartspectra` formula, `rc` for the `smartspectra-rc` formula. Stable: ```bash git clone --branch main https://github.com/Presage-Security/SmartSpectra.git cd SmartSpectra/cpp/samples/macos_swiftui_example ./scripts/check-requirements.sh ``` Release candidate: ```bash git clone --branch rc https://github.com/Presage-Security/SmartSpectra.git cd SmartSpectra/cpp/samples/macos_swiftui_example ./scripts/check-requirements.sh ``` The script checks the SDK installation and code-signing setup. Follow any instructions it prints before opening Xcode. ### Open in Xcode From `cpp/samples/macos_swiftui_example`, open the Xcode project with: ```bash open smartspectra_swift_ui.xcodeproj ``` Then select the `smartspectra_swift_ui` scheme and configure signing on the app target under **Signing & Capabilities**: ```text Team = your Apple Development team Bundle Identifier = a unique identifier for your machine or organization ``` After selecting the app target, this is the Xcode area you should use: ![Signing & Capabilities view showing where Xcode exposes the Bundle Identifier for the Application ID and the signing identity used to look up the Organization ID](images/macos-xcode-signing-source.png) To find your Team ID, open `Xcode > Settings > Accounts`, select your Apple ID and team, and read the `Team ID` value. From the terminal: ```bash security find-identity -v -p codesigning ``` The 10-character ID in parentheses at the end of an `Apple Development` identity is your Team ID. Press **Run**. Enter your `SMARTSPECTRA_API_KEY` in the app and press **Start**. On first launch, allow camera access when macOS prompts. ## What success looks like When your program is running, you should see all of these: - the sample launches from Xcode without missing SDK or model-file errors - macOS prompts for camera access on first launch - the camera preview appears after you allow camera access - processing starts after you enter a valid API key and press **Start** - metric cards and traces update while you remain centered and well-lit ## Expected API key check The first measurement should start after camera permission is granted and the API key is submitted. If startup fails with an authentication error, verify that the API key is valid and authorized for this app. ## Common manual mistakes If the screen does not match the target state, check these first: - the Homebrew formula channel does not match the sample branch - `HOMEBREW_PREFIX` or `SMARTSPECTRA_SDK_ROOT` points at the wrong SDK install - the app target is not signed with an Apple Development identity - camera permission was denied in macOS System Settings - the API key was mistyped or copied with extra whitespace - the app is still running an older build from Xcode ### `brew install presage/smartspectra/smartspectra-rc` fails because `smartspectra` is already linked If you installed the stable formula first, Homebrew may require the stable formula to be unlinked before it links the RC formula: ```bash brew unlink smartspectra brew install presage/smartspectra/smartspectra-rc ``` ### Configuration If your Homebrew prefix is not `/opt/homebrew`, open the project's Build Settings and set: ```text HOMEBREW_PREFIX = output of `brew --prefix` SMARTSPECTRA_SDK_ROOT = $(HOMEBREW_PREFIX) ``` The project derives include and library paths from those two variables. The `Validate Setup` build phase checks the SDK header, library, and interface headers before compilation, so a wrong prefix surfaces early. ## Additional Details ### Portable `hello_vitals` example This example also lives in `smartspectra/cpp/samples/hello_vitals/`. **`hello_vitals.cpp`**: ```cpp #include #include #include #include #include #include #include #include #include namespace spectra = presage::smartspectra; namespace { volatile std::sig_atomic_t g_stop_requested = 0; void HandleSignal(int) { g_stop_requested = 1; } std::string ResolveApiKey(int argc, char** argv) { if (argc > 1) { return argv[1]; } if (const char* key = std::getenv("SMARTSPECTRA_API_KEY")) { return key; } return {}; } } // namespace int main(int argc, char** argv) { std::signal(SIGINT, HandleSignal); const std::string api_key = ResolveApiKey(argc, argv); if (api_key.empty()) { #if defined(_WIN32) std::cerr << "Usage: .\\hello_vitals.exe YOUR_API_KEY\n" << "or set SMARTSPECTRA_API_KEY=YOUR_API_KEY\n"; #else std::cerr << "Usage: ./hello_vitals YOUR_API_KEY\n" << "or export SMARTSPECTRA_API_KEY=YOUR_API_KEY\n"; #endif return 1; } spectra::SmartSpectraConfig config; config.api_key = api_key; config.requested_metrics = spectra::SmartSpectraConfig::BreathingMetrics(); config.AddMetrics(spectra::SmartSpectraConfig::CardioMetrics()); spectra::SmartSpectra sdk(config); sdk.SetOnMetrics([](const spectra::Metrics& metrics, int64_t) { if (metrics.has_cardio()) { std::cerr << "Cardio metrics: " << metrics.cardio().ShortDebugString() << "\n"; } if (metrics.has_breathing()) { std::cerr << "Breathing metrics: " << metrics.breathing().ShortDebugString() << "\n"; } }); sdk.SetOnValidationStatusChanged( [have_last_status = false, last_code = spectra::ValidationCode::kOk, last_hint = std::string{}](const spectra::ValidationStatus& status, int64_t) mutable { if (have_last_status && status.code == last_code && status.hint == last_hint) { return; } have_last_status = true; last_code = status.code; last_hint = status.hint; std::cerr << "Validation [" << status.code << "]: " << status.hint << "\n"; }); sdk.SetOnError([](const spectra::SmartSpectraError& error) { std::cerr << "Error [" << static_cast(error.code) << "]: " << error.message << "\n"; }); const auto source_error = sdk.UseCamera(spectra::CameraSelection::Default()) .SetResolution(1280, 720).SetFps(30).Build(); if (!source_error.ok()) { std::cerr << "Failed to create camera source: " << source_error.message << "\n"; return 1; } if (const auto err = sdk.Start(); !err.ok()) { std::cerr << "Failed to start: " << err.message << "\n"; return 1; } std::cout << "Processing... Press Ctrl+C to stop.\n"; while (!g_stop_requested) { std::this_thread::sleep_for(std::chrono::milliseconds(200)); } if (const auto err = sdk.Stop(); !err.ok()) { std::cerr << "Stop failed: " << err.message << "\n"; } return 0; } ``` The accompanying standalone project files are in that folder. To run the binary end to end on macOS, wrap and sign it using the app-style flow in `smartspectra/cpp/samples/README.md#macos-signing`, or use the SwiftUI sample above. ### Installed Paths On Apple Silicon Homebrew installs, the default paths are: - **Headers**: `/opt/homebrew/include/smartspectra/` - **Libraries**: `/opt/homebrew/lib/` - **CMake config**: `/opt/homebrew/lib/cmake/SmartSpectra/` - **pkg-config**: `/opt/homebrew/lib/pkgconfig/SmartSpectra.pc` Consumer code includes SmartSpectra headers as: ```cpp #include #include #include ``` When linking from your own CMake project: ```cmake set(CMAKE_CXX_STANDARD 20) set(CMAKE_CXX_STANDARD_REQUIRED ON) find_package(SmartSpectra CONFIG REQUIRED) target_link_libraries(my_app PRIVATE SmartSpectra::SDK) ``` ### Release-Candidate Builds Release-candidate builds are published as a separate Homebrew formula, `smartspectra-rc`, served by the same tap as the stable formula. Stable users see no change; RC opt-in is additive. ```bash brew install presage/smartspectra/smartspectra-rc ``` Release candidates do not automatically migrate to the stable formula. After a stable release ships, uninstall `smartspectra-rc` and install `smartspectra` to return to the stable channel: ```bash brew uninstall presage/smartspectra/smartspectra-rc brew install presage/smartspectra/smartspectra ``` ## Distributing an app that embeds the SDK (signing & notarization) If you ship a macOS app that bundles the SmartSpectra SDK to other Macs, [Apple notarization](https://developer.apple.com/documentation/security/notarizing-macos-software-before-distribution) will reject the submission unless every embedded Mach-O — `libsmartspectra.dylib` **and** each bundled FFmpeg dylib (see below) — is signed with **your** Developer ID, has the [**hardened runtime**](https://developer.apple.com/documentation/security/hardened-runtime) enabled, and carries a **secure timestamp**. The Homebrew-installed SDK does **not** satisfy this on its own. The formula ships `libsmartspectra.dylib` **ad-hoc signed** (no Developer ID, no team identifier, no secure timestamp), because Homebrew rewrites the library's install names on install and re-signs it ad-hoc. So you must (re-)sign the copy you embed in your app bundle — you cannot rely on the signature it has when Homebrew installs it. ### Embed the bundled FFmpeg libraries too The SDK isn't a single dylib: it also ships FFmpeg libraries in `/lib/smartspectra/` (`libavcodec`, `libavformat`, `libavutil`, `libswscale`, `libswresample`) for video-file/`.mkv` decode, which `libsmartspectra.dylib` loads via an `@loader_path/smartspectra` runpath. Copy the whole `smartspectra/` folder next to `libsmartspectra.dylib` in `Contents/Frameworks/` — preserve the subfolder or it fails to load — and sign each of those dylibs just like `libsmartspectra.dylib`. They already use `@rpath`/`@loader_path`, so no `install_name_tool` rewriting is needed. > **LGPL.** The FFmpeg libraries are LGPL v2.1+ (`--disable-gpl > --disable-nonfree`). Redistribute the `COPYING.LGPLv2.1`, `COPYING.LGPLv3`, and > `THIRD_PARTY_FFMPEG.md` files shipped in `smartspectra/`, and keep them > dynamically linked/replaceable (LGPL §6). ### Re-sign the embedded libraries with your own identity The embedded copy must be signed with the **same** identity as your app, for two reasons: notarization requires a Developer ID signature with a secure timestamp, and macOS **library validation** (in force once the hardened runtime is enabled) refuses to load a dylib whose Team ID differs from the host app's. Re-signing the copy you bundle satisfies both. (If you must keep a differently-signed copy instead, the host app needs the [Disable Library Validation entitlement](https://developer.apple.com/documentation/BundleResources/Entitlements/com.apple.security.cs.disable-library-validation), which weakens its security posture and is not recommended.) **Xcode (recommended).** Add `libsmartspectra.dylib` to your target under **Frameworks, Libraries, and Embedded Content** and set it to **Embed & Sign** (not "Embed Without Signing"). Xcode then re-signs the embedded copy with your team's identity, hardened runtime, and a secure timestamp on each build. Add the `smartspectra/` dylibs via a **Copy Files** phase (to *Frameworks*, preserving the subpath); the CLI snippet below signs them all. **Command-line / non-Xcode builds.** Re-sign every embedded dylib yourself, as the last step before signing the app (any `install_name_tool` edit invalidates a signature, so sign last): ```bash FW=".app/Contents/Frameworks" # libsmartspectra.dylib + each bundled FFmpeg dylib under smartspectra/ for dylib in "$FW/libsmartspectra.dylib" "$FW"/smartspectra/*.dylib; do codesign --force --options runtime --timestamp \ --sign "Developer ID Application: ()" \ "$dylib" done ``` Then sign the app itself with the same identity and `--options runtime`, submit with `xcrun notarytool submit --wait`, and `xcrun stapler staple` the result. See Apple's [Customizing the notarization workflow](https://developer.apple.com/documentation/security/customizing-the-notarization-workflow) for the full `notarytool`/`stapler` flow. > Note: the brew-installed library is not subject to Gatekeeper itself — > Homebrew downloads are not quarantined, so `brew install` users need no > signing action. The requirements above apply only when you redistribute the > library inside your own app bundle. Apple references: - [Notarizing macOS software before distribution](https://developer.apple.com/documentation/security/notarizing-macos-software-before-distribution) - [Customizing the notarization workflow](https://developer.apple.com/documentation/security/customizing-the-notarization-workflow) (`notarytool` / `stapler`) - [Hardened Runtime](https://developer.apple.com/documentation/security/hardened-runtime) - [Disable Library Validation Entitlement](https://developer.apple.com/documentation/BundleResources/Entitlements/com.apple.security.cs.disable-library-validation) ## Next Steps - [Configure which metrics to compute](metrics.md) - [Run headless without video output](headless-mode.md) - [Migration Guide](migration-guide.md) for upgrading from older SDK versions ## Documentation API reference available at [C++ API Reference](api-reference.md). ## Troubleshooting ### Missing SDK files If the sample reports missing SDK files, run the diagnostic script: ```bash ./scripts/check-requirements.sh ``` If required SDK files are missing, reinstall the formula you selected: `brew reinstall presage/smartspectra/smartspectra` for stable or `brew reinstall presage/smartspectra/smartspectra-rc` for release candidates. ### Metrics do not appear immediately Keep the subject still and centered in the camera preview until validation reports that recording is OK. For support: contact [support@presagetech.com](mailto:support@presagetech.com) or [submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues). ## Configuring Metrics on C++ Source: https://presagetech.com/docs/smartspectra-sdk/cpp/metrics --- title: Configuring Metrics on C++ description: Request pulse and breathing rates, HRV and EDA metrics from the SmartSpectra C++ SDK, and read callback results. sidebarTitle: Configuring Metrics --- # Configuring C++ Metrics By default, C++ measurements request the breathing metric set. Add pulse rate when your app needs a basic cardio value. ## Breathing and Pulse ### Request Metrics Request the breathing metrics plus `MetricType::PULSE_RATE` before constructing `SmartSpectra`: ```cpp #include #include namespace spectra = presage::smartspectra; using presage::smartspectra::MetricType; spectra::SmartSpectraConfig config; config.api_key = "YOUR_API_KEY"; config.requested_metrics = spectra::SmartSpectraConfig::BreathingMetrics(); config.AddMetrics({MetricType::PULSE_RATE}); ``` ### Read Metrics Read the latest breathing and pulse samples from `SetOnMetrics`. This complete example includes the SDK construction and the pulse accessor: ```cpp #include #include #include #include #include namespace spectra = presage::smartspectra; spectra::SmartSpectraConfig config; config.api_key = "YOUR_API_KEY"; config.requested_metrics = spectra::SmartSpectraConfig::BreathingMetrics(); config.AddMetrics(spectra::SmartSpectraConfig::CardioMetrics()); spectra::SmartSpectra spectra(config); spectra.SetOnMetrics([](const spectra::Metrics& metrics, int64_t) { if (metrics.has_breathing() && metrics.breathing().rate_size() > 0) { const auto& rate = metrics.breathing().rate(metrics.breathing().rate_size() - 1); LOG(INFO) << "Breathing rate: " << rate.value(); } if (metrics.has_cardio() && metrics.cardio().pulse_rate_size() > 0) { const auto& pulse = metrics.cardio().pulse_rate(metrics.cardio().pulse_rate_size() - 1); LOG(INFO) << "Pulse rate: " << pulse.value(); } }); ``` If `requested_metrics` is empty, the SDK uses `DefaultSupportedMetrics()`, which returns `BreathingMetrics()`. Use `BreathingMetrics()` when you are explicitly composing a breathing request. Cardio fields are empty unless you request a cardio metric such as `PULSE_RATE`. ## Metric Update Patterns `SetOnMetrics` receives the latest SDK metrics payload. Each payload contains the samples that became available since the previous metrics callback; it is not guaranteed to contain every requested field every time. | Metric category | Examples | Expected cadence | Empty behavior | | --- | --- | --- | --- | | Peak/event-driven rate metrics | `breathing().rate()`, `cardio().pulse_rate()`, `cardio().hrv()` | Updated when a new physiological event, cycle, or analysis window produces a value | Repeated fields may be empty between valid updates during active capture | | Frame-driven metrics | `face().expression()`, `face().landmarks()`, `face().blinking()`, `face().talking()`, breathing traces | Updated near device frame cadence, with SDK callbacks rate-limited to about 30 Hz | Usually present more continuously when the metric is enabled and the input signal is valid | For example, `metrics.cardio().pulse_rate_size()` and `metrics.breathing().rate_size()` may be `0` between valid updates. This is expected and does not mean capture stopped or the metric was disabled. By contrast, face expression samples are frame-driven, so `metrics.face().expression_size()` can be non-zero on most callbacks while face metrics are enabled and the face signal is valid. Recommended UI handling: - Keep the last valid rate sample in app state and update it only when the repeated field contains a new sample. - Show an initial loading or placeholder state until the first valid sample arrives. - Do not overwrite a displayed pulse rate or breathing rate just because one metrics payload has no new sample. - Clear retained values when a capture session starts, stops, or when your app intentionally changes the requested metric set. - Prefer sample timestamps, and `stable()` when present, to decide whether a retained value is fresh enough for your UI. ```cpp #include std::optional last_pulse_rate; spectra.SetOnMetrics([&last_pulse_rate](const presage::smartspectra::Metrics& metrics, int64_t) { if (metrics.has_cardio() && metrics.cardio().pulse_rate_size() > 0) { const auto& pulse = metrics.cardio().pulse_rate(metrics.cardio().pulse_rate_size() - 1); if (pulse.timestamp() > 0) { last_pulse_rate = pulse.value(); LOG(INFO) << "Pulse rate: " << *last_pulse_rate; } } else if (!last_pulse_rate.has_value()) { LOG(INFO) << "Pulse rate pending"; } }); ``` ## Advanced Request additional metrics only when your app needs them: ```cpp config.requested_metrics = spectra::SmartSpectraConfig::BreathingMetrics(); config.AddMetrics({ MetricType::PULSE_RATE, MetricType::ARTERIAL_PRESSURE_TRACE, MetricType::HRV, MetricType::EDA_TRACE, MetricType::FACE_LANDMARKS, MetricType::BLINKING, MetricType::TALKING, MetricType::EXPRESSIONS, }); ``` Read the advanced fields from the same metrics callback: ```cpp spectra.SetOnMetrics([](const presage::smartspectra::Metrics& metrics, int64_t) { if (metrics.has_cardio() && metrics.cardio().arterial_pressure_trace_size() > 0) { const auto& pressure = metrics.cardio().arterial_pressure_trace( metrics.cardio().arterial_pressure_trace_size() - 1); LOG(INFO) << "Arterial pressure trace: " << pressure.value(); } if (metrics.has_cardio() && metrics.cardio().hrv_size() > 0) { const auto& hrv = metrics.cardio().hrv(metrics.cardio().hrv_size() - 1); LOG(INFO) << "HRV RMSSD: " << hrv.rmssd(); } if (metrics.has_eda() && metrics.eda().trace_size() > 0) { const auto& eda = metrics.eda().trace(metrics.eda().trace_size() - 1); LOG(INFO) << "EDA Proxy trace: " << eda.value(); } if (metrics.has_face() && metrics.face().landmarks_size() > 0) { const auto& landmarks = metrics.face().landmarks(metrics.face().landmarks_size() - 1); LOG(INFO) << "Face landmark count: " << landmarks.value_size(); } if (metrics.has_face() && metrics.face().blinking_size() > 0) { const auto& blinking = metrics.face().blinking(metrics.face().blinking_size() - 1); LOG(INFO) << "Blinking: " << blinking.detected(); } if (metrics.has_face() && metrics.face().talking_size() > 0) { const auto& talking = metrics.face().talking(metrics.face().talking_size() - 1); LOG(INFO) << "Talking: " << talking.detected(); } if (metrics.has_face() && metrics.face().expression_size() > 0) { const auto& expression = metrics.face().expression(metrics.face().expression_size() - 1); LOG(INFO) << "Expression score count: " << expression.scores_size(); } }); ``` ### Advanced Payload Classes C++ uses the generated protobuf classes. Requested advanced metrics populate these fields: ```cpp presage::smartspectra::Metrics { Breathing breathing; Eda eda; Face face; Cardio cardio; } Cardio { repeated MeasurementWithConfidence pulse_rate; repeated MeasurementWithConfidence arterial_pressure_trace; repeated Hrv hrv; } Hrv { double rmssd; double mean_nn; double sdnn; double baevsky; int64 timestamp; float confidence; bool stable; } Eda { repeated Measurement trace; } Face { repeated Landmarks landmarks; repeated DetectionStatus blinking; repeated DetectionStatus talking; repeated Expression expression; } ``` EDA Proxy may take longer to produce its first sample than breathing or cardio outputs. See [Data Types](../../docs/data-types.md) for the complete protobuf schema. ## Timing and Stability All measurement samples use `timestamp` values in microseconds. Trace metrics are produced at frame cadence when the underlying signal is available; lower rate outputs such as EDA Proxy may arrive less frequently. Measurement types expose a `stable()` flag. Check it before using a sample for critical decisions or user-facing summaries: ```cpp if (metrics.has_breathing() && metrics.breathing().rate_size() > 0) { const auto& rate = metrics.breathing().rate(metrics.breathing().rate_size() - 1); if (rate.stable()) { LOG(INFO) << "Stable breathing rate: " << rate.value(); } } ``` Face landmark samples also expose `reset()`, which indicates that landmark tracking was reinitialized: ```cpp if (metrics.has_face() && metrics.face().landmarks_size() > 0) { const auto& landmarks = metrics.face().landmarks(metrics.face().landmarks_size() - 1); if (landmarks.reset()) { LOG(INFO) << "Face landmark tracking reset"; } } ``` ## Serialization Metrics are protobuf messages. Serialize them directly when you need to persist or transmit the exact SDK payload: ```cpp std::string binary; if (metrics.SerializeToString(&binary)) { writeMetrics(binary); } ``` For diagnostics, convert to JSON with protobuf utilities: ```cpp #include std::string json; google::protobuf::util::JsonPrintOptions options; options.preserve_proto_field_names = true; auto status = google::protobuf::util::MessageToJsonString(metrics, &json, options); if (status.ok()) { LOG(INFO) << json; } ``` ## C++ Migration Guide Source: https://presagetech.com/docs/smartspectra-sdk/cpp/migration-guide --- title: C++ Migration Guide description: "Migrate the SmartSpectra C++ SDK to C++20, typed camera selection, specific usage errors, and other release-by-release API changes." sidebarTitle: Migration Guide --- # SmartSpectra C++ SDK Migration Guide > Applies to SmartSpectra C++ SDK v3.x. ## C++ SDK v3.4.0 Migration ### Build with C++20 SmartSpectra C++ SDK v3.4.0 requires a C++20 compiler. Update projects that still compile as C++17 before upgrading: ```cmake set(CMAKE_CXX_STANDARD 20) set(CMAKE_CXX_STANDARD_REQUIRED ON) ``` Use GCC, Clang, or Visual Studio 2022 with C++20 support enabled. This change applies to application code that includes SmartSpectra headers as well as code that links the prebuilt SDK. ### Explicit camera selection Camera discovery and typed selection are available on Linux, Windows, macOS, and in the native C++ API on iOS. Existing callers keep index-based behavior: omitting source configuration or calling `UseCamera()` still selects index zero. The deprecated `UseCamera(int device_index = 0)` overload remains available, including its original camera inventory and index ordering. Opt in to the new selection policy with `UseCamera(CameraSelection::Default())`. To require a specific camera, use its discovered ID: ```cpp using namespace presage::smartspectra; std::vector cameras; if (auto error = SmartSpectra::AvailableCameras(cameras); !error.ok()) { // Handle discovery failure. return; } if (cameras.empty()) { // No cameras are currently visible to the application. return; } // Present cameras to the user and keep the selected entry's ID. const auto& selected = cameras.front(); if (auto error = spectra.UseCamera(CameraSelection::ById(selected.id)) .SetResolution(1280, 720) .SetFps(30) .Build(); !error.ok()) { // Handle configuration failure. return; } if (auto error = spectra.Start(); !error.ok()) { // Handle authentication, camera access, or other startup failures. } ``` `AvailableCameras()` is static: no SDK instance, API key, permission prompt, or capture session is needed. It replaces the output vector. An empty successful result means no cameras are visible; visibility can depend on permissions. Each `CameraInfo` contains an opaque `id`, an optional display `name`, and a `facing` of `kFront`, `kBack`, or `kUnknown`. Treat IDs as platform-local values and rediscover before reusing saved choices. Discovery does not reserve a camera or guarantee it can be opened later. | Selection | Behavior | | --- | --- | | `CameraSelection::Default()` | iOS prefers front and otherwise uses the first discovered camera. macOS uses discovery order: built-in, physical external, then virtual or unknown devices, with IDs breaking ties. Windows uses the first enumerated camera. Linux uses the first compatible V4L2 capture node in numeric device order. Other native backends retain their existing default camera behavior. | | `CameraSelection::Front()` | Requires a camera reported as front-facing. | | `CameraSelection::Back()` | Requires a camera reported as back-facing. | | `CameraSelection::ById(id)` | Requires that exact discovered ID. | When multiple cameras match a facing, the first in discovery order wins. A webcam can report `kUnknown`; use its ID to select it. The Apple inventory includes wide-angle built-in cameras and external camera endpoints. On iOS it also includes physical ultra-wide and telephoto cameras, when present, with wide-angle cameras ordered before the other built-in lenses and external cameras. Virtual dual/triple-camera combinations are not listed separately: select an individual lens by ID to avoid automatic switching between constituent cameras. The additional iOS lenses are available only through typed selection; they do not change the indices used by existing applications. Each `CameraInfo` includes `lens_type` (`CameraLensType::kWideAngle`, `kUltraWide`, `kTelephoto`, or `kUnknown`). Classification is best effort; unknown metadata and cameras combining multiple lenses report `kUnknown`. Use this field for picker labels and continue selecting by the camera's `id`. On Windows, discovery includes the cameras visible to the application, with optional display names and opaque IDs. Facing and lens type currently report `kUnknown`; select a camera by ID. `Front()` and `Back()` therefore fail at `Start()` with `kInputUnavailable`. Existing integer selections retain their original inventory and ordering. On Linux, discovery lists compatible single-planar V4L2 streaming capture nodes in numeric `/dev/videoN` order and uses the device's card name when available. IDs prefer `/dev/v4l/by-id` and `/dev/v4l/by-path` aliases, falling back to the device node; treat every form as opaque. Facing is populated from the standard V4L2 camera-orientation control when a driver provides it. External cameras and cameras without orientation metadata report `kUnknown`, and lens type currently reports `kUnknown`. Existing integer selections continue to map directly to `/dev/videoN`, so they are unaffected by discovery filtering or ordering. `Build()` validates the request without opening a camera. An empty ID returns `kConfigurationFailed` and preserves the previous input source. Explicit front/back/ID requests and discovery currently return `kConfigurationFailed` on other native backends. Those backends still support the integer camera API and `Default()` where native capture is available. `Start()` resolves the request against the current inventory. A missing or inaccessible camera returns `kInputUnavailable`; explicit requests never fall back to another ID or facing. Only `Default()` permits a facing fallback. Camera permission and application configuration requirements still apply. Choose a source while uninitialized or idle. To change cameras, complete `Stop()`, build the new camera source, and call `Start()` again. A request is preserved across stop/start/reset until another source is successfully built. Swift, Android, and Node.js also provide typed camera discovery and selection; follow the migration guide for each platform when updating a cross-platform app. The `smart_spectra_example` sample accepts `--list_cameras` to print camera IDs, names, and facing without an API key. Use `--camera_id="ID"` with your API key to select one. This flag takes precedence over `--camera_device_index` and cannot be combined with `--input_video_path`. ### Usage failures now report specific errors Usage entitlement checking continues throughout a measurement as before. What changes in v3.4.0 is how a terminal usage failure is reported. Handle the specific errors in `SetOnError()`: - `kAuthenticationFailed` for rejected credentials - `kCreditExhausted` for an explicit quota denial - `kNetworkError` when entitlement cannot be refreshed in time The SDK also includes short measurements and the final partial interval before `Stop()` in usage reporting. No public API changes are required. ## C++ SDK v3.3.0 Migration ### Default log verbosity is now warnings and errors only The SDK previously emitted informational log chatter by default. From v3.3.0 the default level is `SmartSpectraLogLevel::kWarning`. If you relied on the informational output, restore it via `SmartSpectraConfig::log_level`: ```cpp SmartSpectraConfig config; config.log_level = presage::smartspectra::SmartSpectraLogLevel::kInfo; ``` Setting the `GLOG_minloglevel` environment variable still works and is never overridden by the SDK default. ## C++ SDK v3.1.0-rc.4 Migration ### `MetricsToJsonSoA` return type `presage::smartspectra::MetricsToJsonSoA` previously returned `nlohmann::json` and required customers to have `nlohmann/json.hpp` reachable on their compile path through the bundled SDK headers. It now returns `absl::StatusOr` — the SDK no longer ships `nlohmann/json.hpp` and the public header (`smartspectra/messages/metrics.h`) no longer references it. ```cpp // Before (v3.1.0-rc.3 and earlier): #include nlohmann::json j = presage::smartspectra::MetricsToJsonSoA(metrics); double pr = j["series"]["cardio.pulse_rate"]["values"][0]; // After (v3.1.0-rc.4+): #include #include auto json = presage::smartspectra::MetricsToJsonSoA(metrics); if (!json.ok()) { // handle absl::InvalidArgumentError (serialization failure) } // Write the serialized JSON to a file / network / log, or parse it back // into a JSON object using the JSON library of your choice (nlohmann/json, // rapidjson, simdjson, etc.) — the SDK no longer opinionates this. std::string serialized = *json; ``` The on-the-wire JSON layout (`{"series": {"": {"timestamps": [...], "values": [...]}, ...}}`) is unchanged. ## C++ SDK v3.0 Migration Migrating from SDK v1.x to v3.0 (C++). ### Operation Mode Removal The SDK now operates in continuous mode by default. The `OperationMode` enum and related container type aliases have been removed. Configure the SDK with `SmartSpectraConfig` and use `SmartSpectra` directly. ```cpp // Before: Settings settings; CpuContinuousRestForegroundContainer container(settings); // After: presage::smartspectra::SmartSpectraConfig config; config.api_key = "..."; config.requested_metrics = presage::smartspectra::SmartSpectraConfig::BreathingMetrics(); presage::smartspectra::SmartSpectra spectra(config); ``` For timed sessions that previously depended on spot-style container behavior: ```cpp config.enable_accumulated_output = true; spectra.UseFile("video.mp4").SetMaxDuration(30000).Build(); // For live camera sessions, keep timing in application code and call Stop(). ``` ### Container Removal Use `SmartSpectra` directly instead of `Container` abstractions. ```cpp // Before: #include container::CpuRestForegroundContainer container(settings); container.Initialize(); container.Run(); ``` ```cpp // After: #include #include presage::smartspectra::SmartSpectraConfig config; config.api_key = "..."; presage::smartspectra::SmartSpectra spectra(config); const auto source_error = spectra.UseCamera().Build(); if (!source_error.ok()) { LOG(ERROR) << source_error.FullMessage(); return; } if (const auto err = spectra.Start(); !err.ok()) { LOG(ERROR) << err.FullMessage(); return; } ``` ### Public Headers Use the SmartSpectra public headers and protobuf metric headers directly: ```cpp #include #include #include ``` ### Input Source Builders Must Be Materialized The current API separates source configuration from source creation. After calling `UseCamera()`, `UseFile()`, or `UseCustomInput()`, call `Build()` and check the returned `SmartSpectraError` before `Start()`. ```cpp // Before: spectra.UseCamera(); spectra.Start(); ``` ```cpp // After: const auto source_error = spectra.UseCamera().SetResolution(1280, 720).SetFps(30).Build(); if (!source_error.ok()) { LOG(ERROR) << source_error.FullMessage(); return; } if (const auto err = spectra.Start(); !err.ok()) { LOG(ERROR) << err.FullMessage(); return; } ``` If you omit source configuration entirely, `Start()` uses `CameraSelection::Default()` (see the camera selection notes above). ### Metric Defaults Are Narrower If an older integration assumed pulse-related output was part of the default metric set, update it explicitly. `BreathingMetrics()` is the named breathing bundle. `DefaultSupportedMetrics()` is the fallback used when `requested_metrics` is empty and currently delegates to `BreathingMetrics()`. ```cpp presage::smartspectra::SmartSpectraConfig config; config.api_key = "..."; config.requested_metrics = presage::smartspectra::SmartSpectraConfig::BreathingMetrics(); config.AddMetrics( presage::smartspectra::SmartSpectraConfig::CardioMetrics()); ``` ### CMake Target Change Older container-based integrations linked `SmartSpectra::Container`. The installed v3 package exposes the public C++ SDK as `SmartSpectra::SDK`. ```cmake # Before: target_link_libraries(my_app SmartSpectra::Container) # After: target_link_libraries(my_app SmartSpectra::SDK) ``` ### Single Package SDK Older source-build integrations may have linked the Edge target directly as `Physiology::Edge`. That target is not a customer-facing installed package target in v3. Use the installed SmartSpectra package and link `SmartSpectra::SDK`. ```bash sudo apt install libsmartspectra-dev ``` ```cmake # Before: target_link_libraries(my_app Physiology::Edge) # After: target_link_libraries(my_app SmartSpectra::SDK) ``` ### Protobuf Namespace: `presage::physiology` → `presage::smartspectra` The generated protobuf message namespace has been renamed from `presage::physiology` to `presage::smartspectra`. Older container classes already lived under `presage::smartspectra::container`; current public SDK classes and generated `*.pb.h` messages now share the top-level `presage::smartspectra` namespace. ```cpp // Before: presage::physiology::MetricsBuffer metrics_buffer; presage::physiology::Metrics edge_metrics; // After: presage::smartspectra::SmartSpectraConfig config; presage::smartspectra::Metrics metrics; ``` The shared protobuf message schemas (`insights`, `metric_types`, `metrics`, `point_types`, `status`) also move from `package presage.physiology` to `package presage.smartspectra`. The on-the-wire byte format of serialized messages is unchanged (field numbers are untouched). #### `Any.type_url` and JSON `@type` non-interop `google.protobuf.Any.type_url` strings, JSON `@type` fields, and `DescriptorPool` lookups all key off the fully-qualified message name, which moves from `presage.physiology.` to `presage.smartspectra.`. A new SDK build cannot decode `Any`-wrapped or JSON-keyed payloads written by an older build of the SDK, and vice versa. Cross-version interop between old and new builds is not supported as part of this migration. If your application persists `Any`-wrapped blobs or stores JSON with `@type` fields keyed by the old type name, regenerate those artifacts with the new SDK before deploying. Re-emitting the data is the only supported path. #### Related identity surfaces This migration covers the C++ public surface and the proto `package` declarations. Related platform surfaces are documented separately: - **Android Java/Kotlin (`com.presage.physiology.proto.*` → `com.presagetech.smartspectra.proto.*`)**: consumers building against the new Android SDK release should update their generated-proto imports as part of the Android upgrade. See the Android Migration Guide for the platform-specific import examples. - **Python proto wheel (`SmartSpectra-Messages`)**: the internal Python wheel now publishes the generated protobuf modules under `smartspectra.messages.*` to match the SDK package identity. See the "Python Proto Wheel" section below for the import examples. The Android JNI-bound classes `com.presage.physiology.Messages` and `com.presage.physiology.emd.security.AndroidKeyStoreHelper` are pinned by native `.so` JNI symbol names and are not renamed by this migration or by the Java protobuf migration; their cleanup is tracked as a further follow-up. ### Container and Physiology Module Headers → SmartSpectra SDK Headers The old public surface combined `smartspectra/container/...` headers with `physiology/modules/...` generated protobuf headers. Current consumers include the SmartSpectra SDK headers and protobuf metric headers directly. ```cpp // Before: #include #include #include // After: #include #include #include ``` The SDK now exports a single `-I` root (`/include`); the previous secondary `-I/include/physiology` propagation is gone. ### Public CMake Surface Uses `SMARTSPECTRA_*` Installed CMake package variables now use the one-word `SMARTSPECTRA_*` prefix. There are no compatibility aliases for the old names, so update consumer `CMakeLists.txt` files and CI flags in one pass. | Old | New | | --- | --- | | `SMART_SPECTRA_VERSION` | `SMARTSPECTRA_VERSION` | | `SMART_SPECTRA_VERSION_MAJOR` | `SMARTSPECTRA_VERSION_MAJOR` | | `SMART_SPECTRA_VERSION_MINOR` | `SMARTSPECTRA_VERSION_MINOR` | | `SMART_SPECTRA_VERSION_PATCH` | `SMARTSPECTRA_VERSION_PATCH` | | `SMART_SPECTRA_VERSION_PLAIN` | `SMARTSPECTRA_VERSION_PLAIN` | | `SMART_SPECTRA_INSTALL_INCLUDE_DIR` | `SMARTSPECTRA_INSTALL_INCLUDE_DIR` | | `SMART_SPECTRA_INSTALL_LIB_DIR` | `SMARTSPECTRA_INSTALL_LIB_DIR` | | `SMART_SPECTRA_INSTALL_CMAKE_DIR` | `SMARTSPECTRA_INSTALL_CMAKE_DIR` | | `PHYSIOLOGY_EDGE_ARCHITECTURE` | `SMARTSPECTRA_ARCHITECTURE` | | `PHYSIOLOGY_EDGE_MODEL_DIRECTORY` | `SMARTSPECTRA_MODEL_DIRECTORY` | | `PHYSIOLOGY_EDGE_GRAPH_DIRECTORY` | `SMARTSPECTRA_GRAPH_DIRECTORY` | | `PHYSIOLOGY_EDGE_MLX_ENABLED` | `SMARTSPECTRA_MLX_ENABLED` | | `PHYSIOLOGY_EDGE_MLX_METALLIB_PATH` | `SMARTSPECTRA_MLX_METALLIB_PATH` | | `PHYSIOLOGY_EDGE_AUTO_FINALIZE` | `SMARTSPECTRA_AUTO_FINALIZE` | The user-facing CMake option for disabling remote model delivery was also renamed: ```diff - cmake -DDISABLE_REMOTE_MODEL_DELIVERY=ON ... + cmake -DSMARTSPECTRA_DISABLE_REMOTE_MODEL_DELIVERY=ON ... ``` ### MLX CMake Helper Rename The installed MLX helper module was renamed from `PhysiologyEdge_mlx.cmake` to `SmartSpectra_mlx.cmake`. `SmartSpectraConfig.cmake` includes this helper automatically on Apple/MLX builds, so only consumers that include the helper directly need to change anything. ```diff - include("${CMAKE_CURRENT_LIST_DIR}/PhysiologyEdge_mlx.cmake") + include("${CMAKE_CURRENT_LIST_DIR}/SmartSpectra_mlx.cmake") ``` In this release the helper's public CMake API was also renamed from the `PhysiologyEdge_` / `_PhysiologyEdge_` / `PHYSIOLOGY_EDGE_` / `physiology_edge_` prefixes to `SmartSpectra_` / `_SmartSpectra_` / `SMARTSPECTRA_` / `_SMARTSPECTRA_` / `_smartspectra_`. This is a **hard rename** — no compatibility aliases or deprecation shims are provided, so consumers calling the old names will see "Unknown CMake command" errors at configure time. | Old name | New name | | ----------------------------------------------------- | --------------------------------------------------- | | `PhysiologyEdge_finalize_target` | `SmartSpectra_finalize_target` | | `PhysiologyEdge_copy_mlx_metallib_to_target` | `SmartSpectra_copy_mlx_metallib_to_target` | | target property `_PHYSIOLOGY_EDGE_FINALIZED` | target property `_SMARTSPECTRA_FINALIZED` | | global property `_PHYSIOLOGY_EDGE_TARGETS_TO_FINALIZE`| global property `_SMARTSPECTRA_TARGETS_TO_FINALIZE` | | custom target `_physiology_edge_deploy_metallib` | custom target `_smartspectra_deploy_metallib` | | `message(STATUS "PhysiologyEdge: …")` | `message(STATUS "SmartSpectra: …")` | The installed SmartSpectra package exposes only the `SmartSpectra::*` targets (`SmartSpectra::SDK` for application consumers). `Physiology::Edge` is an internal source-build target and is not part of the installed package — see "Single Package SDK" above. To apply all renames mechanically across a downstream consumer's CMake sources, run from the repo root: ```bash git ls-files -z '*.cmake' '*.cmake.in' 'CMakeLists.txt' '**/CMakeLists.txt' \ | xargs -0 sed -i \ -e 's/PhysiologyEdge_finalize_target/SmartSpectra_finalize_target/g' \ -e 's/PhysiologyEdge_copy_mlx_metallib_to_target/SmartSpectra_copy_mlx_metallib_to_target/g' \ -e 's/_PHYSIOLOGY_EDGE_FINALIZED/_SMARTSPECTRA_FINALIZED/g' \ -e 's/_PHYSIOLOGY_EDGE_TARGETS_TO_FINALIZE/_SMARTSPECTRA_TARGETS_TO_FINALIZE/g' \ -e 's/_physiology_edge_deploy_metallib/_smartspectra_deploy_metallib/g' \ -e 's/"PhysiologyEdge: /"SmartSpectra: /g' ``` (On macOS, replace `sed -i` with `sed -i ''`.) ### `messages/` is now first-class Protobuf metric headers used to live under `physiology/modules/messages/`. The `modules/` nesting level has been dropped — `messages/` now sits directly under the new include base. ```cpp // Before: #include #include #include // After: #include #include #include ``` ### Pruned Public Headers Several headers — and the raw proto sources — that were previously installed publicly are no longer shipped: - `` — generated build-feature `#define`s, never a stable public API. No replacement. - `` — internal filesystem helper. No replacement. - `` — internal enum-reflection helper (`AllEnumValues`); reachable only via full Protobuf reflection and unused by the public API. No replacement. - `` — the `CameraControls` interface is not reachable through the public SDK (camera setup goes through `UseCamera`, which owns the source internally). No replacement. - The raw `messages/*.proto` sources — build against the generated `messages/*.pb.h` (still shipped) instead. If your code transitively included any of these, vendor your own equivalent or open an issue describing the use case. ### Version Header The SDK now exposes a single version header at ``. The previous `` and the transitional `` are retired; both have been collapsed into ``. ```cpp // Before: #include // After: #include ``` The version macros are now namespaced as `SMART_SPECTRA_VERSION_MAJOR`, `SMART_SPECTRA_VERSION_MINOR`, `SMART_SPECTRA_VERSION_PATCH`, `SMART_SPECTRA_VERSION_STRING`, and `SMART_SPECTRA_VERSION_PLAIN`. The constexpr accessors under `presage::smartspectra::GetVersionMajor()` / `GetVersionMinor()` / `GetVersionPatch()` / `GetVersionString()` / `GetVersionPlain()` are unchanged in name and signature. ### Container Headers → `smartspectra*` SDK Headers Public header filenames in `/include/smartspectra/` now expose the direct SDK surface instead of the old container surface: ```cpp // Before: // After: ``` ### Python Proto Wheel The internal Python protobuf wheel now publishes the generated modules under the SmartSpectra package identity: ```python # Before: import physiology.modules.messages.metrics_pb2 as metrics from physiology.modules.messages import status_pb2 # After: import smartspectra.messages.metrics_pb2 as metrics from smartspectra.messages import status_pb2 ``` ## C++ on Windows Source: https://presagetech.com/docs/smartspectra-sdk/cpp/windows --- title: C++ on Windows description: Install the SmartSpectra C++ SDK on Windows with NuGet or CMake, then build and run the sample app in Visual Studio. sidebarTitle: Overview --- # SmartSpectra C++ Quickstart — Windows > **Warning — Experimental platform:** Windows support for the SmartSpectra > C++ SDK is experimental. If you have any issues running SmartSpectra, > [contact Presage support](mailto:support@presagetech.com) for assistance. ## Supported Platforms | Platform | Status | Notes | | -------- | ------ | ----- | | Windows 10 / 11 (x64) | Experimental | ZIP distribution available | For platforms not listed above, contact [support@presagetech.com](mailto:support@presagetech.com) if you have a specific need. ## Installation ### Prerequisites Install [Visual Studio Build Tools 2022](https://visualstudio.microsoft.com/visual-cpp-build-tools/) or later — or a full Visual Studio 2022 or newer install (Community edition works) — with the **Desktop development with C++** workload. During installation, select the **Desktop development with C++** workload and make sure **C++ CMake tools for Windows** is selected. The workload installs the MSVC compiler, Windows SDK, CMake, and the developer command prompt needed for the quickstart below. You also need an **API key** from [physiology.presagetech.com](https://physiology.presagetech.com/auth/login). ### Add the SDK Download `smartspectra-sdk--windows-x64.zip` from [GitHub Releases](https://github.com/Presage-Security/SmartSpectra/releases) and extract it to a permanent location, for example `C:\SmartSpectra`. Keep the extracted layout intact — CMake config files, runtime DLLs, and bundled resources must stay in the locations expected by the package. ### Permissions No SDK-specific OS permission setup is required on Windows. ## Example: CMake Project This walkthrough sets up a minimal CMake project that reads from a camera and prints vitals to the console. ### Result you should get At the end, the app should show console output with: - a successful CMake configure and build - `Processing... Press Ctrl+C to stop.` - `Cardio metrics:` log lines when cardio metrics are available - `Breathing metrics:` log lines when breathing metrics are available - a clean exit after you stop the sample ![SmartSpectra C++ Windows quickstart demo](../images/win-quickstart.gif) ### 1. Open the developer command prompt Open the Windows **Start** menu and search for `x64 Developer Command Prompt` (a full Visual Studio install names it `x64 Native Tools Command Prompt for VS `). Choose the x64 developer command prompt installed by Visual Studio (Build Tools or a full install). For a default Build Tools installation, that shortcut launches: ```bat C:\Windows\System32\cmd.exe /k ""C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\Tools\VsDevCmd.bat" -arch=x64 -host_arch=x64" ``` If you installed Build Tools somewhere else — or use a full Visual Studio install — update the `VsDevCmd.bat` path to match. For example, Visual Studio 2026 Community installs it at `C:\Program Files\Microsoft Visual Studio\18\Community\Common7\Tools\VsDevCmd.bat`. ### 2. Create the project files Create a folder, for example `C:\Projects\HelloVitals`, and add these two files inside it. **`CMakeLists.txt`**: ```cmake cmake_minimum_required(VERSION 3.22.1) project(HelloVitals CXX) set(CMAKE_CXX_STANDARD 20) set(CMAKE_CXX_STANDARD_REQUIRED ON) set(SMARTSPECTRA_SDK_DIR "$ENV{SMARTSPECTRA_SDK_PATH}") if (SMARTSPECTRA_SDK_DIR STREQUAL "") message(FATAL_ERROR "SMARTSPECTRA_SDK_PATH is not set.") endif () list(APPEND CMAKE_PREFIX_PATH "${SMARTSPECTRA_SDK_DIR}") find_package(SmartSpectra CONFIG REQUIRED) add_executable(hello_vitals hello_vitals.cpp) target_link_libraries(hello_vitals SmartSpectra::SDK) ``` **`hello_vitals.cpp`**: This example also lives in `smartspectra/cpp/samples/hello_vitals/`. ```cpp #include #include #include #include #include #include #include #include #include namespace spectra = presage::smartspectra; namespace { volatile std::sig_atomic_t g_stop_requested = 0; void HandleSignal(int) { g_stop_requested = 1; } std::string ResolveApiKey(int argc, char** argv) { if (argc > 1) { return argv[1]; } if (const char* key = std::getenv("SMARTSPECTRA_API_KEY")) { return key; } return {}; } } // namespace int main(int argc, char** argv) { std::signal(SIGINT, HandleSignal); const std::string api_key = ResolveApiKey(argc, argv); if (api_key.empty()) { #if defined(_WIN32) std::cerr << "Usage: .\\hello_vitals.exe YOUR_API_KEY\n" << "or set SMARTSPECTRA_API_KEY=YOUR_API_KEY\n"; #else std::cerr << "Usage: ./hello_vitals YOUR_API_KEY\n" << "or export SMARTSPECTRA_API_KEY=YOUR_API_KEY\n"; #endif return 1; } spectra::SmartSpectraConfig config; config.api_key = api_key; config.requested_metrics = spectra::SmartSpectraConfig::BreathingMetrics(); config.AddMetrics(spectra::SmartSpectraConfig::CardioMetrics()); spectra::SmartSpectra sdk(config); sdk.SetOnMetrics([](const spectra::Metrics& metrics, int64_t) { if (metrics.has_cardio()) { std::cerr << "Cardio metrics: " << metrics.cardio().ShortDebugString() << "\n"; } if (metrics.has_breathing()) { std::cerr << "Breathing metrics: " << metrics.breathing().ShortDebugString() << "\n"; } }); sdk.SetOnValidationStatusChanged( [have_last_status = false, last_code = spectra::ValidationCode::kOk, last_hint = std::string{}](const spectra::ValidationStatus& status, int64_t) mutable { if (have_last_status && status.code == last_code && status.hint == last_hint) { return; } have_last_status = true; last_code = status.code; last_hint = status.hint; std::cerr << "Validation [" << status.code << "]: " << status.hint << "\n"; }); sdk.SetOnError([](const spectra::SmartSpectraError& error) { std::cerr << "Error [" << static_cast(error.code) << "]: " << error.message << "\n"; }); const auto source_error = sdk.UseCamera(spectra::CameraSelection::Default()) .SetResolution(1280, 720).SetFps(30).Build(); if (!source_error.ok()) { std::cerr << "Failed to create camera source: " << source_error.message << "\n"; return 1; } if (const auto err = sdk.Start(); !err.ok()) { std::cerr << "Failed to start: " << err.message << "\n"; return 1; } std::cout << "Processing... Press Ctrl+C to stop.\n"; while (!g_stop_requested) { std::this_thread::sleep_for(std::chrono::milliseconds(200)); } if (const auto err = sdk.Stop(); !err.ok()) { std::cerr << "Stop failed: " << err.message << "\n"; } return 0; } ``` ### 3. Build Navigate to your project folder with the two files, for example: ```bat cd /d C:\Projects\HelloVitals ``` Configure and build with CMake. Update `C:\SmartSpectra` if you extracted the SDK somewhere else. ```bat set "SMARTSPECTRA_SDK_PATH=C:\SmartSpectra" && cmake -S . -B build -G "NMake Makefiles" -DCMAKE_BUILD_TYPE=Release && cmake --build build ``` ### 4. Run The executable needs the SDK runtime DLLs (`smartspectra.dll`, `opencv_world*.dll`, `vulkan-1.dll`, …) on its load path. The simplest way is to put the SmartSpectra SDK `bin/` directory on `PATH` for the session — Windows then loads the DLLs from there, and the SDK finds its bundled resources relative to `smartspectra.dll`. No copying and no resource configuration are required: ```bat set "PATH=%SMARTSPECTRA_SDK_PATH%\bin;%PATH%" .\build\hello_vitals.exe YOUR_API_KEY ``` Or set the key once in the shell as well: ```bat set "SMARTSPECTRA_API_KEY=YOUR_API_KEY" && .\build\hello_vitals.exe ``` You should see breathing and cardio metrics printed to the console within a few seconds of the camera starting. Press `Ctrl+C` to stop the sample. ## What success looks like When your program is running, you should see all of these: - `Processing... Press Ctrl+C to stop.` prints after launch - the camera starts without a source creation error - `Cardio metrics:` or `Breathing metrics:` logs print while you sit centered and well-lit - the process exits after `Ctrl+C` without a `Stop failed` message ## Expected API key check The first measurement should start after the executable launches with a valid API key argument or `SMARTSPECTRA_API_KEY` environment variable. If startup fails with an authentication error, verify that the key is authorized for this app and that the shell value did not include extra quotes or whitespace. ## Common manual mistakes If the console output does not match the target state, check these first: - the ZIP was extracted into a different folder than `SMARTSPECTRA_SDK_PATH` - the project was built outside the x64 developer command prompt - the API key argument or `SMARTSPECTRA_API_KEY` environment variable is missing - the SDK `bin/` directory is not on `PATH`, so the runtime DLLs can't be found - the runtime DLLs were copied away from the SDK `bin/`, separating them from the sibling `share/smartspectra/` resource tree - another app is already using the camera - the executable is an older build from before the latest source change ## Additional Details ### Metric selection Adjust the metric bundles in `hello_vitals.cpp` before building: ```cpp config.requested_metrics = spectra::SmartSpectraConfig::BreathingMetrics(); config.AddMetrics(spectra::SmartSpectraConfig::CardioMetrics()); ``` Available bundles: `BreathingMetrics()`, `CardioMetrics()`, `FaceMetrics()`. ### ZIP layout reference ```text include/ smartspectra/ # C++ SDK headers and protobuf metric headers smartspectra/interface/ # Bundled third-party headers smartspectra_capi.h # C ABI shim for FFI consumers lib/ smartspectra.lib # C++ SDK import library (MSVC) smartspectra_capi.lib # C ABI shim import library SmartSpectra_MessageProtos_*.lib # Message proto static libs cmake/SmartSpectra/SmartSpectraConfig.cmake # CMake package bin/ smartspectra.dll # C++ SDK runtime DLL — must ship with your app smartspectra_capi.dll # C ABI shim runtime DLL — required for FFI consumers vulkan-1.dll # Vulkan loader used by the default inference backend smartspectra_manifest.json # lets smartspectra.dll locate ../share/smartspectra opencv_world*.dll # OpenCV runtime dependency — must ship with your app opencv_videoio_ffmpeg*.dll # OpenCV FFmpeg video-I/O backend (needed for video-file input) share/smartspectra/ # Bundled graph and model resources ``` When you ship your app to other machines, keep the `bin/` and `share/` directories together as a unit — the SDK locates its resources at `bin/../share/smartspectra` relative to `smartspectra.dll`, so the relationship between the two must be preserved. The recommended layout is to place your executable in `bin/` (or copy the whole `bin/ + share/` tree into your install directory); the DLLs then load without any `PATH` setup and resources resolve automatically. Do **not** copy the DLLs out on their own — separating `smartspectra.dll` from its sibling `share/` tree is what breaks resource resolution. ## Next Steps - [Configure which metrics to compute](../metrics.md) - [Run headless without video output](../headless-mode.md) - [Migration Guide](../migration-guide.md) for upgrading from older SDK versions ## Troubleshooting If the app starts but can't find DLLs, verify that the SDK `bin/` directory is on `PATH` (or that your executable sits in `bin/` alongside `smartspectra.dll`, `opencv_world*.dll`, and the other SDK DLLs). If the app loads but fails to find its models/graph resources, the runtime DLLs were likely separated from the SDK's `share/smartspectra/` tree. Keep `bin/` and `share/` together — the SDK resolves resources at `bin/../share/smartspectra` relative to `smartspectra.dll`. If you are upgrading an older C++ integration, see the [C++ Migration Guide](../migration-guide.md). For support: contact [support@presagetech.com](mailto:support@presagetech.com) or [submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues). ## Issues and Limitations Source: https://presagetech.com/docs/smartspectra-sdk/cpp/windows/issues-and-limitations --- title: Issues and Limitations description: Known issues and platform limitations for the SmartSpectra C++ SDK on Windows, and the workarounds available for each of them. --- # Issues and Limitations — Windows > **Warning — Experimental platform:** Windows support for the SmartSpectra > C++ SDK is experimental. The items below are known limitations we are actively > tracking. If you hit something that is not listed here, > [contact Presage support](mailto:support@presagetech.com) for assistance. ## Camera support Built-in / integrated cameras (for example, the internal webcam on a laptop) do **not** currently work with the SmartSpectra C++ SDK on Windows. Use an external USB webcam instead. The following camera has been tested and works with the Windows SDK: | Camera | Status | Notes | | ------ | ------ | ----- | | Logitech C920 | Tested — working | External USB webcam | | Built-in / integrated webcams | Not working | Use an external USB webcam | ## Reporting an issue If you run into a limitation that is not listed here, let us know so we can track it: contact [support@presagetech.com](mailto:support@presagetech.com) or [submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues). ## Using the AI agent skill Source: https://presagetech.com/docs/smartspectra-sdk/agent-skill --- title: Using the AI agent skill description: Install the using-smartspectra agent skill in Claude Code or Codex, and build pulse and breathing apps with an AI coding assistant. --- > **Important:** SDK metrics are offered for general wellness and informational purposes only. SDK metrics have not been cleared by the FDA and may not be used for medical diagnosis or treatment. `using-smartspectra` is an [Agent Skill](https://agentskills.io) — a small guide that an AI coding assistant loads on demand to build apps with the SmartSpectra SDK. It teaches the API model (getting a key, the config → start/stop lifecycle, choosing metrics) and points the assistant at these docs for exact, current per-platform detail. It is one portable `SKILL.md` in the standard Agent Skills format, so the **same skill works in both Claude Code and OpenAI Codex**. Both are hosted in the public repo [`Presage-Security/SmartSpectra`](https://github.com/Presage-Security/SmartSpectra). ## Install in Claude Code Add the repo as a plugin marketplace, then install the plugin: ```text /plugin marketplace add Presage-Security/SmartSpectra /plugin install smartspectra-sdk@smartspectra ``` Run `/plugin` to confirm `smartspectra-sdk` is enabled. The skill then loads automatically when a task matches it; there is nothing else to configure. ## Install in Codex Install the skill straight from the repo with the Codex skill installer: ```text $skill-installer Presage-Security/SmartSpectra/.agents/skills/using-smartspectra ``` Codex detects newly installed skills automatically; restart Codex if it does not appear. When you are working inside a clone of the repo, Codex also discovers the skill automatically from `.agents/skills/`. ## Install anywhere else For any other assistant, the same skill is served straight from this site: ```text https://smartspectra.presagetech.com/skill.md ``` Point your tool at that URL, or paste the file into the assistant's instructions. It is the identical `SKILL.md` the plugin installs, so there is nothing to keep in sync. ## Reading these docs as markdown The skill has your assistant read this site for exact API detail, and every page here is available as markdown — append `.md` to any documentation URL: ```text https://smartspectra.presagetech.com/docs/swift/metrics.md ``` [llms.txt](https://smartspectra.presagetech.com/llms.txt) indexes every page with a one-line description, and [llms-full.txt](https://smartspectra.presagetech.com/llms-full.txt) is the whole corpus in one file. Assistants generally find these on their own; the URLs are here for when you want to hand one over directly. ## Use it Describe what you want to build and let the assistant drive: ```text Using the SmartSpectra SDK, build a minimal app that measures and displays my pulse. ``` The skill guides the assistant to the correct API for your platform — C++, Swift/iOS, Kotlin/Android, or Node — and to the matching pages in these docs for exact signatures, install steps, and sample apps. You review and run the code it writes, the same as any other assistant output. ## What you need - An **API key** from the [Developer Admin Portal](https://physiology.presagetech.com/auth/login) — the assistant will use it to build a working app. The skill itself does not fetch a key for you, but if you also connect the [SmartSpectra MCP Server](mcp-server.md) the assistant can retrieve it from your account. Otherwise set it as described on your platform's setup page. - **Network access** for the assistant while it works: the skill has the assistant read the live docs at `smartspectra.presagetech.com` for exact API detail. Without web access it can still apply the general model but cannot look up precise signatures. ## Getting Help - Email: [support@presagetech.com](mailto:support@presagetech.com) - [Submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues) - [Docs and FAQ](https://smartspectra.presagetech.com) - [Developer Admin Portal](https://physiology.presagetech.com/auth/login) ## SmartSpectra Metrics Payload Data Types Source: https://presagetech.com/docs/smartspectra-sdk/data-types --- title: SmartSpectra Metrics Payload Data Types description: The language-agnostic SmartSpectra metrics payload — every message, enum, and field the SDK emits. sidebarTitle: Data Types --- ## InsightType Origin of an Insight. Only INSIGHT_TYPE_VITALS is emitted on delivered Insights today; SPEECH and COMBINED are reserved for a future release. Correlate an on-demand reply with its prompt via Insight.request_id, not this field. ```proto enum InsightType { INSIGHT_TYPE_VITALS = 0; INSIGHT_TYPE_SPEECH = 1; INSIGHT_TYPE_COMBINED = 2; } ``` - `INSIGHT_TYPE_VITALS` -- Auto-fired vitals snapshot dispatched periodically by InsightSession with the accumulated metrics buffer and no caller prompt. The only value set on delivered Insights today. - `INSIGHT_TYPE_SPEECH` -- Reserved, not emitted. Describes a request from RequestInsight with a prompt but no buffered metrics at dispatch time (prompt-only nudge). - `INSIGHT_TYPE_COMBINED` -- Reserved, not emitted. Describes a request from RequestInsight carrying both the caller's prompt and a non-empty metrics buffer. ## Insight LLM-generated analysis or error returned by the insights endpoint. ### Properties ```proto message Insight { int32 request_id = 1; string processed_at = 2; oneof result { string analysis = 3; string error = 4; } InsightType type = 5; } ``` - `int32` `request_id` -- Client-generated request ID. Populated client-side by the async callback closure (not returned by the server). Lets callers correlate responses to requests. - `string` `processed_at` -- ISO timestamp of when the analysis was produced - `string` `analysis` -- The LLM analysis text, set on success. - `string` `error` -- Error message, set on failure. - [`InsightType`](#insighttype) `type` -- Origin of this insight. Always INSIGHT_TYPE_VITALS today — SPEECH and COMBINED are reserved and never set — so route on request_id rather than on this field to tell auto-fired snapshots from on-demand replies. ## FeatureType FeatureType defines high-level physiological measurement categories. Each feature represents a super-metric that may encompass multiple individual metrics. ```proto enum FeatureType { BREATHING = 0; EDA = 2; FACE = 3; CARDIO = 4; } ``` > Some wire values are intentionally omitted from this view; the gaps in the numeric sequence preserve compatibility with the underlying proto. Use the listed names — do not renumber. - `BREATHING` -- Breathing measurements (chest and abdomen) - `EDA` -- EDA Proxy (electrodermal activity) measurement - `FACE` -- Facial tracking and analysis - `CARDIO` -- Cardiovascular metrics (pulse, HRV, relative arterial pressure waveform) ## MetricType MetricType defines individual physiological measurements that can be requested and produced during video-based vital signs analysis. ```proto enum MetricType { CHEST_BREATHING = 0; ABDOMEN_BREATHING = 1; BREATHING_RATE = 2; BREATHING_AMPLITUDE = 3; APNEA = 4; RESPIRATORY_LINE_LENGTH = 5; BASELINE = 6; INHALE_EXHALE_RATIO = 7; EDA_TRACE = 10; FACE_LANDMARKS = 11; BLINKING = 12; TALKING = 13; EXPRESSIONS = 14; PULSE_RATE = 15; ARTERIAL_PRESSURE_TRACE = 16; HRV = 17; } ``` > Some wire values are intentionally omitted from this view; the gaps in the numeric sequence preserve compatibility with the underlying proto. Use the listed names — do not renumber. - `CHEST_BREATHING` -- Breathing upper (chest) metrics - `ABDOMEN_BREATHING` -- Breathing lower (abdomen) metrics - `BREATHING_RATE` -- Breathing aggregate metrics - `EDA_TRACE` -- EDA Proxy metrics - `FACE_LANDMARKS` -- Face metrics - `PULSE_RATE` -- Cardio metrics ## RequestedMetrics Wrapper message for passing a list of requested metrics through MediaPipe packets ### Properties ```proto message RequestedMetrics { repeated MetricType metrics = 1; } ``` ## Measurement Represents a single measurement with timestamp and stability information. Used for various physiological measurements throughout the system. ### Properties ```proto message Measurement { float value = 1; bool stable = 2; int64 timestamp = 3; } ``` - `float` `value` -- The measured or estimated value - `bool` `stable` -- Whether the measurement is considered stable/reliable - `int64` `timestamp` -- Absolute timestamp at which the measurement was taken, in microseconds, since Linux epoch ## DetectionStatus Represents detection status with timestamp information. Used to track whether a particular physiological feature or state is detected. ### Properties ```proto message DetectionStatus { bool detected = 1; bool stable = 2; int64 timestamp = 3; } ``` - `bool` `detected` -- Whether the feature/state was detected - `bool` `stable` -- Whether the detection is considered stable/reliable - `int64` `timestamp` -- Absolute timestamp at which the detection status was updated, in microseconds, since Linux epoch ## MeasurementWithConfidence Represents a measurement with an associated confidence score. Extends basic measurement with confidence information for quality assessment. ### Properties ```proto message MeasurementWithConfidence { float value = 1; bool stable = 2; float confidence = 3; int64 timestamp = 4; } ``` - `float` `value` -- The measured value - `bool` `stable` -- Whether the measurement is considered stable/reliable - `float` `confidence` -- Confidence score for the measurement, expressed as a percentage in the range [0.0, 100.0] - `int64` `timestamp` -- Absolute timestamp at which the measurement was taken, in microseconds, since Linux epoch ## ExpressionType Enumerates the supported facial expression types output by the model. ```proto enum ExpressionType { UNSPECIFIED = 0; ANGRY = 1; CONTEMPT = 2; DISGUST = 3; FEAR = 4; HAPPY = 5; NEUTRAL = 6; SAD = 7; SURPRISE = 8; } ``` - `UNSPECIFIED` -- Expression is unspecified or unknown - `ANGRY` -- Angry expression - `CONTEMPT` -- Contempt expression - `DISGUST` -- Disgust expression - `FEAR` -- Fear expression - `HAPPY` -- Happy expression - `NEUTRAL` -- Neutral expression - `SAD` -- Sad expression - `SURPRISE` -- Surprise expression ## ExpressionScore Associates an expression type with its confidence score. ### Properties ```proto message ExpressionScore { ExpressionType type = 1; float confidence = 2; } ``` - [`ExpressionType`](#expressiontype) `type` -- Expression type identifier - `float` `confidence` -- Confidence score for the expression, expressed as a percentage in the range [0.0, 100.0] ## Expression Represents a detected expression with metadata. Used for facial expression analysis and emotion detection. ### Properties ```proto message Expression { bool stable = 1; int64 timestamp = 2; repeated ExpressionScore scores = 3; } ``` - `bool` `stable` -- Whether the detection is considered stable/reliable - `int64` `timestamp` -- Absolute timestamp at which the expression was detected, in microseconds, since Linux epoch - `repeated` [`ExpressionScore`](#expressionscore) `scores` -- Confidence distribution across all expression types ## Hrv Represents a single HRV measurement computed over some period of pulse data. ### Properties ```proto message Hrv { double rmssd = 1; double mean_nn = 2; double sdnn = 3; double baevsky = 4; int64 timestamp = 5; float confidence = 6; bool stable = 7; } ``` - `double` `rmssd` -- root mean square of successive differences between normal heartbeats - `double` `mean_nn` -- mean normal-to-normal (NN) interval length - `double` `sdnn` -- Standard Deviation of normal-to-normal (NN) Intervals - `double` `baevsky` -- Baevsky's Stress Index: a measure of autonomic balance derived from the NN interval distribution, computed as `AMo / (2 * Mo * MxDMn)`. Reported without a unit, matching the HRV model card. - `int64` `timestamp` -- Absolute timestamp at which the HRV measurement was taken, in microseconds, since Linux epoch - `float` `confidence` -- Confidence score for the HRV measurement, expressed as a percentage in the range [0.0, 100.0] - `bool` `stable` -- Whether the HRV measurement is considered stable/reliable ## Pulse Comprehensive pulse-related measurements and derived metrics. Contains heart rate, pulse trace, and respiratory coupling information. ### Properties ```proto message Pulse { repeated MeasurementWithConfidence rate = 1; repeated Measurement trace = 2; repeated Measurement pulse_respiration_quotient = 3; } ``` - `repeated` [`MeasurementWithConfidence`](#measurementwithconfidence) `rate` -- Heart rate measurements with confidence scores - `repeated` [`Measurement`](#measurement) `trace` -- Raw pulse trace measurements - `repeated` [`Measurement`](#measurement) `pulse_respiration_quotient` -- Pulse-respiration quotient measurements indicating cardio-respiratory coupling ## Breathing Comprehensive breathing/respiratory measurements and derived metrics. Contains respiratory rate, traces, and various breathing pattern indicators. ### Properties ```proto message Breathing { repeated MeasurementWithConfidence rate = 1; repeated Measurement upper_trace = 2; repeated Measurement lower_trace = 3; repeated Measurement amplitude = 4; repeated DetectionStatus apnea = 5; repeated Measurement respiratory_line_length = 6; repeated Measurement baseline = 7; repeated Measurement inhale_exhale_ratio = 8; } ``` - `repeated` [`MeasurementWithConfidence`](#measurementwithconfidence) `rate` -- Respiratory rate measurements with confidence scores. `stable` marks confidence at or above the minimum accepted accuracy standard for breathing rate (+/-1 br/min), which corresponds to confidence >= 45; see the [breathing model card](https://vandv.presagetech.com/breathing_model_card.html) for the confidence-to-error mapping. - `repeated` [`Measurement`](#measurement) `upper_trace` -- Chest breathing movement trace measurements. The trace has no confidence of its own; `stable` is inherited from the breathing rate verdict (>= 45); see the [breathing model card](https://vandv.presagetech.com/breathing_model_card.html). - `repeated` [`Measurement`](#measurement) `lower_trace` -- Abdominal breathing movement trace measurements. The trace has no confidence of its own; `stable` is inherited from the breathing rate verdict (>= 45); see the [breathing model card](https://vandv.presagetech.com/breathing_model_card.html). - `repeated` [`Measurement`](#measurement) `amplitude` -- Breathing amplitude measurements - `repeated` [`DetectionStatus`](#detectionstatus) `apnea` -- Apnea (breathing cessation) detection status - `repeated` [`Measurement`](#measurement) `respiratory_line_length` -- Respiratory line length measurements for breathing pattern analysis - `repeated` [`Measurement`](#measurement) `baseline` -- Baseline breathing measurements - `repeated` [`Measurement`](#measurement) `inhale_exhale_ratio` -- Inhale to exhale duration ratio measurements ## Landmarks Facial landmark coordinates with temporal and stability information. Used for face tracking and facial feature analysis. ### Properties ```proto message Landmarks { repeated Point2dFloat value = 1; bool stable = 2; bool reset = 3; int64 timestamp = 4; } ``` - `repeated` [`Point2dFloat`](#point2dfloat) `value` -- Array of 2D coordinate points representing facial landmarks - `bool` `stable` -- Whether the landmark detection is considered stable/reliable - `bool` `reset` -- Indicates whether the landmark set was reset (cannot be directly associated with previous set) - `int64` `timestamp` -- Absolute timestamp at which the landmarks were detected, in microseconds, since Linux epoch ## Face Comprehensive facial analysis measurements and detections. Contains blinking, talking detection, landmarks, and expressions. ### Properties ```proto message Face { repeated DetectionStatus blinking = 1; repeated DetectionStatus talking = 2; repeated Landmarks landmarks = 3; repeated Expression expression = 4; } ``` - `repeated` [`DetectionStatus`](#detectionstatus) `blinking` -- Blinking detection status over time - `repeated` [`DetectionStatus`](#detectionstatus) `talking` -- Talking/speech detection status over time - `repeated` [`Landmarks`](#landmarks) `landmarks` -- Facial landmark coordinates over time - `repeated` [`Expression`](#expression) `expression` -- Detected expressions over time ## Eda EDA Proxy (electrodermal activity) measurements. Tracks skin conductance changes related to autonomic nervous system activity. ### Properties ```proto message Eda { repeated Measurement trace = 1; } ``` - `repeated` [`Measurement`](#measurement) `trace` -- EDA Proxy trace measurements over time ## Cardio ### Properties ```proto message Cardio { repeated MeasurementWithConfidence pulse_rate = 1; repeated MeasurementWithConfidence arterial_pressure_trace = 2; repeated Hrv hrv = 3; } ``` - `repeated` [`MeasurementWithConfidence`](#measurementwithconfidence) `pulse_rate` -- Heart rate measurements with confidence scores. `stable` marks confidence at or above the minimum accepted accuracy standard for heart rate (+/-3 bpm), which corresponds to confidence >= 40; see the [arterial pressure model card](https://vandv.presagetech.com/arterial_pressure_model_card.html) for the confidence-to-error mapping. Heart rate is derived from the same signal. - `repeated` [`MeasurementWithConfidence`](#measurementwithconfidence) `arterial_pressure_trace` -- Arterial pressure trace (uncalibrated, unitless) measurements with confidence scores. `stable` uses the same threshold as heart rate -- the minimum accepted accuracy standard (+/-3 bpm), confidence >= 40 -- since both derive from the arterial pressure signal; see the [arterial pressure model card](https://vandv.presagetech.com/arterial_pressure_model_card.html). - `repeated` [`Hrv`](#hrv) `hrv` -- Heart rate variability measurements with confidence scores. `stable` marks confidence at or above the minimum accepted accuracy standard for HRV (+/-5 ms), which corresponds to confidence >= 50; see the [HRV model card](https://vandv.presagetech.com/hrv_model_card.html) for the confidence-to-error mapping. ## Metrics Comprehensive physiological metrics container. Contains all available physiological measurements and analysis results. ### Properties ```proto message Metrics { Breathing breathing = 1; Eda eda = 3; Face face = 4; Cardio cardio = 5; } ``` - [`Breathing`](#breathing) `breathing` -- Breathing and respiratory analysis results - [`Eda`](#eda) `eda` -- EDA Proxy (electrodermal activity) measurements. Note: processing needs to run for over 35 seconds to generate the first EDA Proxy result. - [`Face`](#face) `face` -- Facial analysis results - [`Cardio`](#cardio) `cardio` -- Cardiovascular measurements (pulse rate, arterial pressure, HRV) ## Point2dInt32 Represents a 2D point with integer coordinates. Used for pixel-based coordinate systems and discrete positioning. ### Properties ```proto message Point2dInt32 { int32 x = 1; int32 y = 2; } ``` - `int32` `x` -- X coordinate as 32-bit signed integer - `int32` `y` -- Y coordinate as 32-bit signed integer ## Point2dFloat Represents a 2D point with floating-point coordinates. Used for precise positioning, normalized coordinates, and sub-pixel accuracy. ### Properties ```proto message Point2dFloat { float x = 1; float y = 2; } ``` - `float` `x` -- X coordinate as floating-point value - `float` `y` -- Y coordinate as floating-point value ## Point3dFloat Represents a 3D point with floating-point coordinates. Used for spatial positioning, depth information, and 3D landmark representation. ### Properties ```proto message Point3dFloat { float x = 1; float y = 2; float z = 3; } ``` - `float` `x` -- X coordinate as floating-point value - `float` `y` -- Y coordinate as floating-point value - `float` `z` -- Z coordinate (depth) as floating-point value ## Headless Testing in CI Overview Source: https://presagetech.com/docs/smartspectra-sdk/headless-testing-in-ci --- title: Headless Testing in CI Overview description: Run a SmartSpectra sample headlessly in a CI pipeline — feed a recorded video for automated measurement, or smoke-test that the SDK builds and starts. --- # Headless Testing in CI The SmartSpectra SDKs are **headless** — they ship no built-in UI, so your application (or a test harness) drives the SDK and reads results from callbacks. That same property lets you exercise the SDK **unattended in a CI pipeline**: on every commit, prove that your integration still builds, starts, and produces readings, without a person holding a phone in front of a camera. This page is the cross-platform overview. Each SDK has its own guide with the exact sample, commands, and flags — see [Per-platform guides](#per-platform-guides). ## Two ways to test headlessly What you can automate depends on how the platform accepts input: | Platform | Mode | Input in CI | | --- | --- | --- | | C++ (Linux/macOS/Windows) | Video-fed measurement | A recorded video file you supply | | Node.js | Video-fed measurement | A recorded video file you supply | | Android | Video-fed measurement (public custom-input API) | Decoded frames from a recorded video you supply | | iOS | Video-fed measurement (public custom input or testing-only file API) | Decoded frames or a recorded video file you supply | - **Video-fed measurement** — every SDK can run a full measurement from a recorded video in place of a live camera, so CI can assert that readings came out. No camera or display is needed. On desktop (C++, Node.js) the file input is regular public API. Android and iOS also accept caller-decoded frames through public `useCustomInput()`. The mobile testing helpers require explicit opt-in: `@_spi(Testing)` on iOS and `@OptIn(SmartSpectraTestingApi::class)` on Android. - **Build-integration smoke** — the lighter fallback on any platform when you don't have a recorded clip: prove the SDK builds, links, launches headless, initializes, and surfaces the expected permission/error states. ## What you'll need For any of the above: - **A SmartSpectra API key**, provided to the job as a CI **secret** (never hard-code it in the repo). Your platform sample accepts it at startup. - **Network access** from the CI runner to the SmartSpectra service. Measurement authenticates online — there is **no offline mode**, so an air-gapped runner can't complete a measurement. For video-fed measurement, additionally: - **A short recorded video you supply** — around 30–60 seconds of a well-lit, mostly still face, framed like a real measurement (long enough for rates to compute; a few seconds isn't). Keep the clip in your own test assets. See the platform guide for the container/codec each SDK expects. ## The video-fed model A video-fed headless test is the same shape on every platform: ```text recorded video -> SDK video input -> metric callbacks -> assertions (no camera, no display) ``` Feed the file, let the SDK run its normal pipeline, and assert on what the callbacks emit. Keep the assertion **smoke-level**: check that the SDK produced real readings (for example, a pulse rate and a breathing rate appeared), rather than checking exact values. A smoke test confirms the integration and the model pipeline are wired up end-to-end; it is not an accuracy benchmark. ## A CI pipeline, in general terms Whatever CI system you use, the shape is the same: 1. **Expose the API key** as a job secret. 2. **Install or build** the SmartSpectra SDK for the platform (follow the platform's install guide). 3. **Run the sample headlessly** — feed your recorded video through the platform's video input (a headless sample on desktop, an instrumented / XCTest run on mobile). 4. **Fail the job** if the run crashed or didn't produce the expected output. A minimal, provider-neutral sketch (GitHub Actions, video-fed) — adapt the install and run steps to your platform's guide: ```yaml name: smartspectra-headless-smoke on: [push] jobs: headless: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 # Install the SmartSpectra SDK for your platform (see the platform guide). - name: Install SmartSpectra run: echo "follow the platform install guide" # Run a headless sample against a recorded video you keep in the repo. # The sample exits non-zero (failing the job) if it can't measure. - name: Headless measurement smoke run: | ./your_smartspectra_sample \ --api_key="${{ secrets.SMARTSPECTRA_API_KEY }}" \ --input_video_path=./test-assets/face.mp4 ``` ## Per-platform guides Bring the model above; each guide fills in the concrete sample, flags, and CI steps for its platform (see the sidebar): - **[C++](../cpp/docs/headless-mode.md#running-headlessly-in-ci)** — headless sample with recorded-video input on Linux, macOS, and Windows. - **[Node.js](../nodejs/docs/headless-testing-in-ci.md)** — headless sample that plays a recorded video through the SDK. - **[Android](../android/docs/headless-testing-in-ci.md)** — video-fed measurement as an instrumented test on an emulator, via the public `CustomInput` frame handle. - **[iOS](../swift/docs/headless-testing-in-ci.md)** — video-fed measurement as an XCTest on the iOS Simulator, via public custom input or testing-only `@_spi(Testing)` file playback. ## Limitations - **No offline mode.** A measurement authenticates against the SmartSpectra service; the CI runner needs network access. - **iOS file playback is testing-only.** Use public custom input for production Android and iOS capture integrations. Keep mobile testing helpers out of production paths. ## LLM Insights Overview Source: https://presagetech.com/docs/smartspectra-sdk/llm-insights --- title: LLM Insights Overview sidebarTitle: Overview description: Conceptual overview, common API interface, required configuration, and privacy notice for the SmartSpectra LLM Insights feature. --- # LLM Insights LLM Insights turn the physiological metrics the SDK computes on-device (pulse, heart-rate variability, breathing) into natural-language analysis produced by a large language model. The SDK buffers a rolling window of metrics, sends that window — optionally with a prompt you supply — to the Presage 3.0 Analytics Gateway, and delivers the model's reply to your application as an `Insight`. This page is the primary reference for the feature: the concepts, the common API shape, configuration, and privacy. For exact signatures and runnable examples, follow the link to your platform's guide. ## How it works LLM Insights follow a request/response model. There are two ways a request is dispatched, and both deliver their response through the same sink: - **Auto-fired vitals** — once a session is running, the SDK automatically dispatches vitals snapshots of the accumulated metrics buffer: one the first time pulse rate has a stable reading, one when breathing rate does, one when HRV does, and then one every **60 seconds** for as long as the session runs. You supply no prompt for these; the SDK sends a fixed one of its own — *"Provide a concise vitals check-in based on the latest measurements."* The service may return no analysis for a given snapshot, in which case no insight is delivered for it. - **On-demand** — you request an insight with a prompt. If the metrics buffer already holds vitals at dispatch time, the request is **combined** (your prompt plus the latest metrics); if the buffer is still empty, it is **prompt-only**. The physiological metrics are what make the analysis specific to the user: the model receives the buffered metric series, not just your prompt text, so it can ground its response in the measured pulse, HRV, and breathing. Responses are asynchronous — there is no blocking call that returns an analysis inline. ## Common API interface Every platform exposes the same two operations, in a platform-idiomatic form: 1. **Request an insight** — submit a prompt on a running session. It returns a **request ID** that identifies the reply. The prompt is combined with the latest buffered metrics when they exist, otherwise sent prompt-only. 2. **Receive responses** — register a single sink that receives **all** insight responses (both auto-fired vitals and on-demand replies). Each response is an `Insight` carrying either `analysis` (success) or `error` (failure), plus a `request_id`. Correlate an on-demand reply with the request that produced it by matching its `request_id` (auto-fired vitals won't match a request you made). Every insight is currently delivered with type `INSIGHT_TYPE_VITALS` (`SPEECH`/`COMBINED` are reserved), so use `request_id`, not `type`, to tell them apart. The `Insight` type is documented in [Data Types → Insight](../data-types.md#insight). How each platform surfaces these: | Platform | Request | Receive responses | | --- | --- | --- | | [C++](../../cpp/docs/llm-insights.md) | `RequestInsight(text, &request_id)` | `SetOnInsight(callback)` | | [Android](../../android/docs/llm-insights.md) | `requestInsight(text): Int` | `insight: LiveData` | | [Swift](../../swift/docs/llm-insights.md) | `requestInsight(_:) throws -> Int32` | observable `insight` property | | [Node.js](../../nodejs/docs/llm-insights.md) | `requestInsight(text): number` | `on('insight', …)` event | See your platform's guide for exact signatures, setup, and examples: [C++](../../cpp/docs/llm-insights.md) · [Android](../../android/docs/llm-insights.md) · [Swift](../../swift/docs/llm-insights.md) · [Node.js](../../nodejs/docs/llm-insights.md). For guidance on writing prompts the model can act on, see [Writing Effective Prompts](writing-prompts.md). When a request fails to send or a response carries an error, see [LLM Insights: Error Handling & Troubleshooting](troubleshooting.md) for the two error surfaces, common failure modes, retry guidance, and UI patterns. ## Required metrics configuration Insights are only meaningful when the metrics they summarize are being computed: - **Breathing (defaults) and cardio must both be active.** The SDK buffers pulse rate, HRV, and breathing rate for the insight payload, so those metric groups must be enabled. Each platform guide shows the exact configuration call. - **`ARTERIAL_PRESSURE_TRACE` is available for visualization.** It drives the on-screen pulse waveform and is part of the cardio metrics. - **Allow warm-up time before insights are meaningful.** The metrics buffer starts empty and fills as valid measurement accumulates. The first auto-fired vitals snapshot goes out when pulse rate first has a stable reading, typically about **15 seconds** in; breathing rate and HRV join in their own snapshots as their analysis windows fill (about 30 and 60 seconds; see the warm-up section of the measurement-quality guide). An on-demand request made before the buffer has filled is sent prompt-only (no metrics). ## Privacy & data notice When an insight is dispatched, the SDK sends the following off-device to Presage, which forwards it to the LLM: - The **buffered metric values** — an allowlisted subset of the computed metrics (pulse rate, heart-rate variability, and breathing rate) as a columnar series. - The **prompt text** you supply, when present. - Request metadata (a session ID, request origin, and request ID). **Raw video and facial imagery are never transmitted** — only the derived numeric metric values and your prompt leave the device for this feature. > **Important:** LLM Insights are generated by a large language model and can be inaccurate, incomplete, or misleading — language models sometimes produce confident but false statements ("hallucinations"). Treat every insight as general wellness information, not medical advice, and confirm anything important before acting on it. The SDK metrics themselves are offered for general wellness and informational purposes only; they have not been cleared by the FDA and may not be used for medical diagnosis or treatment. ## Getting Help - Email: [support@presagetech.com](mailto:support@presagetech.com) - [Submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues) - [Docs and FAQ](https://smartspectra.presagetech.com) - [Developer Admin Portal](https://physiology.presagetech.com/auth/login) ## Troubleshooting LLM Insights Errors Source: https://presagetech.com/docs/smartspectra-sdk/llm-insights/troubleshooting --- title: Troubleshooting LLM Insights Errors description: The two LLM Insights error surfaces, common failure modes, retry guidance, and how to surface errors in your UI. --- # LLM Insights: Error Handling & Troubleshooting LLM Insights can fail in two distinct places, and telling them apart is the key to handling them correctly. This page covers the two error surfaces, the common failure modes and which surface each shows up on, how to retry, and how to surface errors in your UI. For the feature itself — concepts, the request/response model, configuration, and privacy — see the [LLM Insights overview](index.md). ## The two error surfaces | Surface | When | Meaning | Delivered as | | --- | --- | --- | --- | | **1. Request failed to send** | Synchronously, from the request call | The request never left the device | A non-OK status / thrown error from `RequestInsight` | | **2. Response carries an error** | Asynchronously, on the response sink | The request was sent, but the server failed | An `Insight` with `error` set instead of `analysis` | > **A successful request call does not mean the insight was delivered.** The > network request is asynchronous: `RequestInsight` returns as soon as the request > is *enqueued*, not when the server replies. Anything that goes wrong on the > wire — connectivity, timeouts, rate limits, server errors — comes back later > on **Surface 2**, not from the request call. ### Surface 1 — the request failed to send This surface is synchronous and local: the request was rejected before it was dispatched, so no response will ever arrive for it. In the C++ SDK the call returns a `SmartSpectraError`; the exact codes are: - **`kInvalidState`** — there is no active session. Call `Start()` first (or the session was already stopped). - **`kProcessingFailed`** — the request could not be dispatched. Causes: the prompt exceeds the **2048-byte** size limit (see [Writing prompts](writing-prompts.md#keep-in-mind)); no response sink was registered before requesting; the SDK could not reach the server to establish the session (this is the one case where a *network* failure surfaces synchronously); or the session is shutting down. How each platform reports a failed request: | Platform | Mechanism | | --- | --- | | [C++](../../cpp/docs/llm-insights.md) | `RequestInsight` returns `SmartSpectraError` — check `err.ok()`, read `err.FullMessage()` | | [Android](../../android/docs/llm-insights.md) | `requestInsight` throws — wrap in `runCatching { … }.onFailure { … }` | | [Swift](../../swift/docs/llm-insights.md) | `requestInsight(_:) throws` — use `do { try … } catch { … }` | | [Node.js](../../nodejs/docs/llm-insights.md) | `requestInsight` throws — use `try { … } catch { … }` | ### Surface 2 — the response carries an error This surface is asynchronous: the request was sent and the reply arrived on your response sink, but it carries an `error` message instead of `analysis`. The `analysis` and `error` fields are a `result` oneof — **exactly one is set** — so branch on which is present. The error string typically includes an HTTP status code (for example, `429` or `503`), or a transport error. How each platform reports a response-level error: | Platform | Mechanism | | --- | --- | | [C++](../../cpp/docs/llm-insights.md) | `insight.has_error()` → `insight.error()` | | [Android](../../android/docs/llm-insights.md) | `insight.hasError()` → `insight.error` | | [Swift](../../swift/docs/llm-insights.md) | `insight.result` → `.error(String)` (or `insight.error`) | | [Node.js](../../nodejs/docs/llm-insights.md) | decode the buffer → `insight.error` (the `result` oneof) | ### A third outcome: no response at all A request can also produce **no response** — the sink is simply never called. This is not an error; it is how the service signals "nothing to analyze" (for example, when a session has already produced its analysis and a further request adds nothing). Because there is no callback and no error, your UI must not wait forever — see [Retry patterns and best practices](#retry-patterns-and-best-practices). ## Response error status codes When a response carries an `error` (Surface 2), the message includes an HTTP status. Use it to decide whether to retry: | Status | Meaning | Retry? | | --- | --- | --- | | `400` | Malformed request payload | No — fix the request | | `401` | API key or OAuth setup is missing or invalid | No — fix your credentials | | `403` | Key is not authorized for this resource | No — check account access | | `404` / `410` | The insights session expired | Retry | | `408` | The server timed out handling the request | Yes — back off and retry | | `429` | Quota or credits exhausted, or rate limited | Back off; if out of credits, top up | | `500` / `502` / `503` / `504` | Internal server error | Yes — retry with backoff | `4xx` statuses mean the request or account needs attention — retrying unchanged won't help, except `404` / `410`, which are safe to retry. `5xx` statuses are server-side and transient. An out-of-credits `429` is fixed by topping up in the [Developer Admin Portal](https://physiology.presagetech.com/auth/login). Not every failure has an HTTP status: a transport-level failure (connection refused, host unreachable, client-side timeout), or an error reporting that the server returned no analysis, also arrives as an `error` — treat both as transient and retry with backoff. ## Common failure modes | Failure mode | Surface | What you observe | What to do | | --- | --- | --- | --- | | No active session | 1 | `kInvalidState` / thrown | `Start()` a session before requesting | | Prompt exceeds the size limit | 1 | `kProcessingFailed` / thrown | Shorten the prompt to **≤ 2048 bytes** (see [Writing prompts](writing-prompts.md#keep-in-mind)) | | No response sink registered | 1 | `kProcessingFailed` / thrown | Register the sink **before** the first request | | Can't reach server to establish the session | 1 | `kProcessingFailed` / thrown | Check connectivity and API key; safe to retry | | Network unavailable / server unreachable (after dispatch) | 2 | `error` with a transport message | Retry with backoff | | Request timed out | 2 | `error` (times out after ~30 s) | Retry with backoff | | Server returned an error status | 2 | `error` including an HTTP status | Look it up in [Response error status codes](#response-error-status-codes) | | Empty / insufficient metrics buffer | neither | On-demand request is sent prompt-only; auto-fired vitals wait for the first stable reading | Allow ~15 s of measurement to warm up the buffer | | Nothing to analyze | neither | Sink is never called | Use a UI timeout (below); don't block indefinitely | ## Retry patterns and best practices **The SDK does not retry insight dispatches** — there is no automatic retry or backoff. Retrying is your application's responsibility, and the right policy depends on the surface: - **Surface 1, `kInvalidState`** — do **not** retry blindly. Ensure a session is running (`Start()`) and only then request. Retrying against a stopped session will just fail again. - **Surface 1, `kProcessingFailed`** — usually transient (session or connectivity). Make sure a response sink is registered, then retry once or twice with a short delay. - **Surface 2, transient errors** (network unavailable, timeout, `5xx`, or an error reporting no analysis was returned) — retry with **exponential backoff**. - **Surface 2, `429`** — back off aggressively; if it means credits are exhausted rather than rate limiting, top up instead of retrying. - **Surface 2, permanent errors** (`400` bad request, `401`/`403` auth) — do **not** retry as-is. Fix the prompt, API key, or account first. Additional practices: - **Debounce user-triggered requests.** Disable the trigger (button, etc.) while a request is pending, since rapid back-to-back requests are all sent — and repeated requests within a single session may not each produce a reply, so extra requests can waste quota without adding analyses. - **Always set a UI timeout.** A request can produce no response at all (the "nothing to analyze" outcome above). A failed request resolves within ~30 seconds via Surface 2, but a "nothing to analyze" result never calls back — so add your own pending-state timeout and clear the spinner. - **Correlate with the request ID.** Match the `request_id` on each `Insight` against the ID returned by your request so a late or failed reply updates the right pending item (and so you can ignore auto-fired vitals, which won't match a request you made). ## Surfacing errors in your UI Two reference C++ samples show both surfaces end to end. Use them as templates: - **`smartspectra/cpp/samples/insights_example/main.cc`** — a minimal OpenCV app. It renders *something* for every outcome: `[request failed] ` when the request call fails (Surface 1), `[error] ` when a response carries an error (Surface 2), `[empty response]` when a reply has neither field set, and a `[waiting for response…]` placeholder while a request is in flight. Handling all outcomes — including the empty one — is the pattern to copy. - **`smartspectra/cpp/samples/winui3_example/`** (see `SmartSpectraWinUI/MainWindow.xaml.cpp`) — a production-shaped WinUI 3 app. It maps typed error codes to friendly strings (for example, authentication → "Authentication failed. Check your API key.", credits → "Account credits exhausted.", network → "Network issue — please try again."), disables the "Ask AI" button and shows "Analyzing…" while a request is pending, and re-enables it when a response or a synchronous failure arrives. Distilling both into a checklist: - **Map error codes/messages to friendly strings** for the user; keep the raw message in your logs. - **Show a pending state** and **disable the trigger** while a request is in flight. - **Re-enable and clear the pending state on *every* terminal outcome** — success, error, **and an empty/absent response.** If you only clear it on success and error, a "nothing to analyze" reply leaves your control stuck. - **Add a timeout** for the no-response case so the UI recovers on its own. - **Marshal UI updates onto the UI thread.** In C++ the insight callback runs on a **background thread**; Swift delivers on the main actor and Android on the main thread, but if you fan out to your own handlers, keep UI writes on the UI thread. ## Common symptoms and fixes ### `RequestInsight` returns `kInvalidState` (or the call throws immediately) The session isn't running. Call `Start()` and wait for the session to be active before requesting. --- ### `RequestInsight` returns `kProcessingFailed` (or the call throws immediately) Either no response sink was registered, or the SDK couldn't reach the server to establish the session. Register your insight sink **before** the first request, verify network connectivity and that your API key is valid, then retry. --- ### A response's `error` includes an HTTP status Look the status up in [Response error status codes](#response-error-status-codes). In short: `4xx` means the request or account needs fixing (retrying unchanged won't help, except `404` / `410`), while `5xx`, transport failures, and errors reporting the server returned no analysis are transient — retry with backoff. --- ### No insight ever arrives Three benign causes, in order of likelihood: (1) the metrics buffer hasn't warmed up yet — the first auto-fired insight waits for a stable pulse rate, typically ~15 seconds after `Start()`; (2) no response sink was registered, so replies have nowhere to go — register it before starting; (3) the server had nothing to analyze and returned no result, so the sink was deliberately not called. Because (3) is silent, always back a pending request with a UI timeout. > **Important:** LLM Insights are generated by a large language model and can be inaccurate, incomplete, or misleading — language models sometimes produce confident but false statements ("hallucinations"). Treat every insight as general wellness information, not medical advice, and confirm anything important before acting on it. The SDK metrics themselves are offered for general wellness and informational purposes only; they have not been cleared by the FDA and may not be used for medical diagnosis or treatment. ## Getting Help - Email: [support@presagetech.com](mailto:support@presagetech.com) - [Submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues) - [Docs and FAQ](https://smartspectra.presagetech.com) - [Developer Admin Portal](https://physiology.presagetech.com/auth/login) ## Writing Effective Prompts Source: https://presagetech.com/docs/smartspectra-sdk/llm-insights/writing-prompts --- title: Writing Effective Prompts description: What data an LLM Insights prompt can rely on, effective vs. ineffective prompt patterns, and the caveats that shape a good prompt. --- # LLM Insights: Writing Effective Prompts > **Important:** LLM Insights are generated by a large language model and can be inaccurate, incomplete, or misleading — language models sometimes produce confident but false statements ("hallucinations"). Treat every insight as general wellness information, not medical advice, and confirm anything important before acting on it. The SDK metrics themselves are offered for general wellness and informational purposes only; they have not been cleared by the FDA and may not be used for medical diagnosis or treatment. An on-demand insight request combines your **prompt** with the metrics the SDK has buffered — the user's measured **pulse rate**, **heart-rate variability (HRV)**, and **breathing rate**. Those three values are the only physiological data the model receives, so the most effective prompts ask specific questions about them. A prompt that asks the model to *measure* anything else has nothing in the payload to ground an answer — but the prompt is free text, so you can also supply extra context yourself (see [Add your own context](#add-your-own-context)). For the feature itself — concepts, the request/response model, configuration, and privacy — see the [LLM Insights overview](index.md). A good general-purpose starting point is a prompt that stays entirely within the measured data: > Summarize my current vital signs and flag anything unusual. Use it as-is, or make it more specific to focus the response on a single metric. ## What works, and what doesn't | Effective prompt | Why it works | | --- | --- | | ✅ "Summarize my current vital signs and flag anything unusual." | Asks only about the measured pulse, HRV, and breathing — exactly the data in the payload. | | ✅ "Give me a plain-language summary of my heart-rate variability." | HRV is one of the three metrics sent. | | ✅ "How do my pulse and breathing rate compare right now?" | Both metrics are in the payload, so the model can relate them. | | ✅ "Point out any of my vital signs that stand out." | Pulse, HRV, and breathing are all sent; the model can look across them. | | Ineffective prompt | Why it doesn't work | | --- | --- | | ❌ "What's my blood pressure, blood-oxygen, or temperature?" | The SDK never sends these — only pulse rate, HRV, and breathing rate leave the device. | | ❌ "How does today compare to last week?" | Only the current session's rolling window is sent; there's no *measured* history — unless you [put the past values in the prompt](#add-your-own-context) yourself. | | ❌ "How stressed or anxious am I?" | No emotional, mood, or activity signal is in the payload. | | ❌ "Analyze my face" / "What do I look like?" | Raw video and facial imagery never leave the device — only the numeric metrics do. | | ❌ "Do I have an arrhythmia? Should I see a doctor?" | Requests for medical diagnosis or advice are outside this feature's purpose — see the notice above. | ## Add your own context The SDK only *measures* three metrics, but the prompt is free text — so you can give the model context it has no other way to know, and it will factor that into its answer. Recent sleep, activity, symptoms, medications, or a longer-term baseline all work. For example: > I only got 4 hours of sleep last night. Given that, how do my current vitals > look? Or, to compare against a longer-term baseline: > My average resting pulse over the last 30 days has been about 110 bpm — is my > pulse right now high or low for me? The model treats whatever you add as given information: it can reason about it and relate it to the live measurements. It can't verify it, though, and it won't turn it into measured data — so keep the context accurate, and keep it health-related (see [Keep in mind](#keep-in-mind)). Like the rest of the prompt, any context you add is sent off-device to the Presage gateway — see the [privacy notice](index.md#privacy--data-notice). ## Keep in mind - **Wait for warm-up.** A request made before the metrics buffer has filled is sent **prompt-only**, with no physiology to ground it. Allow at least **15 seconds** of measurement (see [Required metrics configuration](index.md#required-metrics-configuration)) before expecting a data-grounded answer, and up to about **60 seconds** for HRV to be part of it. - **Keep prompts under the size limit.** A prompt is capped at **2048 bytes** (counted in bytes, so a non-ASCII character can use more than one). A prompt over the limit is rejected on-device, before the request is sent — shorten it, and trim any [context you add](#add-your-own-context) if it grows large. - **Only three metrics are available.** Pulse rate, HRV, and breathing rate are the entire physiological surface a prompt can rely on. - **Only the current session is in scope.** The buffered window covers the running session, not past ones — a prompt can't compare across days or separate measurements. - **The assistant stays on your health data.** A server-side guardrail keeps it on task: it answers from the vitals and the context you provide, won't invent readings it wasn't given, and steers an off-topic prompt (say, "write me a poem") back to your health data. Keep any context you supply accurate and on-topic. - **Not for medical use.** The model gives general-wellness commentary, not diagnosis or treatment advice; don't prompt for either. ## Getting Help - Email: [support@presagetech.com](mailto:support@presagetech.com) - [Submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues) - [Docs and FAQ](https://smartspectra.presagetech.com) - [Developer Admin Portal](https://physiology.presagetech.com/auth/login) ## SmartSpectra MCP Server Source: https://presagetech.com/docs/smartspectra-sdk/mcp-server --- title: SmartSpectra MCP Server description: "Connect an AI assistant to your SmartSpectra account with the hosted MCP server: manage API keys, register apps, check plan usage, and search these docs." --- # SmartSpectra MCP Server > **Important:** SDK metrics are offered for general wellness and informational purposes only. SDK metrics have not been cleared by the FDA and may not be used for medical diagnosis or treatment. Presage hosts a [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server at: ```text https://mcp.presagetech.com/mcp ``` Connect it to an MCP-capable AI assistant — Claude Code, claude.ai, Codex, Cursor, or any other compliant client — and the assistant can manage your SmartSpectra developer account for you: fetch or rotate your API key, check your plan and credits, register apps, download the OAuth config file for a registered app, and search these docs. It pairs with the [AI agent skill](agent-skill.md): the skill teaches an assistant how to build with the SmartSpectra SDK; the MCP server gives it authorized access to your account, so "set up a new Android app and wire in my API key" works end to end. ## What you need - A **SmartSpectra account** — the same login as the [Developer Admin Portal](https://physiology.presagetech.com/auth/login). Sign-in happens in your browser through OAuth; the assistant never sees your password. - An **MCP client that supports the streamable HTTP transport and OAuth**. The server uses standard OAuth 2.1 with dynamic client registration, so no client ID or manual token setup is required — just the URL. ## Connect from Claude Code Add the server, then authenticate: ```bash claude mcp add --transport http smartspectra https://mcp.presagetech.com/mcp ``` Run `/mcp` inside Claude Code, select `smartspectra`, and choose **Authenticate**. Your browser opens to the SmartSpectra login; approve the consent screen and the tools become available. ## Connect from claude.ai or Claude Desktop 1. Open **Settings → Connectors → Add custom connector**. 2. Enter `https://mcp.presagetech.com/mcp` as the server URL and save. 3. Click **Connect** and complete the browser login. ## Connect from Codex Add the server, then authenticate: ```bash codex mcp add smartspectra --url https://mcp.presagetech.com/mcp codex mcp login smartspectra ``` `codex mcp login` opens your browser for the SmartSpectra sign-in. Run `codex mcp list` to confirm the server shows as connected. ## Connect from other MCP clients Any client that speaks streamable HTTP with OAuth works. For clients configured with JSON (Cursor and similar), add: ```json { "mcpServers": { "smartspectra": { "url": "https://mcp.presagetech.com/mcp" } } } ``` The client discovers the OAuth endpoints automatically and opens a browser to sign you in. ## Available tools | Tool | What it does | | --- | --- | | `api_keys.get` | Return your current SmartSpectra API key. | | `api_keys.rotate` | Generate a new API key and invalidate the old one. | | `usage.get_plan` | Report your plan tier, remaining credits, and next renewal date. Credits are counted per account, not per registered app. | | `apps.list` | List your registered apps — platform, app ID, sandbox mode, and the registration ID used by the other app tools. | | `apps.register` | Register (or update) an Apple or Android app ID, including sandbox mode and the descriptive fields the portal asks for. | | `apps.get_config` | Fetch the OAuth config file for a registered app — `PresageService-Info.plist` on Apple, `presage_services.xml` on Android. | | `apps.delete` | Delete an app registration, identified by the registration ID from `apps.list`. | | `docs.map` | List the pages of this documentation site. | | `docs.search` | Search this documentation. | | `docs.read` | Read a documentation page: `/`, `/docs`, any `/docs/...` path, `/llms.txt`, or `/llms-full.txt`. | The server also exposes a `using-smartspectra` prompt that loads the [agent skill](agent-skill.md) content directly, for clients without skill support. ## Setting up OAuth through an assistant On iOS and Android, OAuth setup normally means registering your app in the portal by hand and downloading a config file. With this server connected, an assistant does both steps for you: `apps.register` with your bundle ID and Apple team ID (or package name and signing certificate SHA-256 fingerprint), then `apps.get_config` to fetch the config file and write it into your project at the right path. - **iOS** — [Option 2: OAuth](../swift/docs/option-2-oauth.md) covers where `PresageService-Info.plist` goes and how to confirm OAuth is wired correctly. - **Android** — [Option 2: OAuth](../android/docs/option-2-oauth.md) covers `presage_services.xml` and the signing-certificate fingerprint the registration needs. Sandbox mode is part of the registration. On iOS, ask for it when you want to test locally from Xcode builds pushed to your phone; without sandbox, the registration allows App Store and TestFlight builds only. ## Confirmations for destructive actions Tools that change your account — `api_keys.rotate`, `apps.register`, and `apps.delete` — never act on the first call. The first call returns a `confirmation_required` response describing exactly what will happen, plus a confirmation token; the assistant must call the tool again with that token to proceed. A token lasts five minutes, works once, and is bound to the exact arguments it was issued for, so a stale or altered confirmation cannot be replayed. Assistants surface this as an explicit "confirm this action" step — if yours does not, decline and run the action from the [Developer Admin Portal](https://physiology.presagetech.com/auth/login) instead. > **Important:** `api_keys.rotate` invalidates the current key immediately — every app using it > stops authenticating until you deploy the new key. Deleting your account is not available through the MCP server. Use the [Developer Admin Portal](https://physiology.presagetech.com/auth/login). ## Security notes - `api_keys.get` places your API key in the assistant's conversation context. Treat that conversation as sensitive, and rotate the key if you believe it leaked. - To revoke the assistant's access, remove or disconnect the server in your MCP client's settings. Sessions also expire on their own and require re-authenticating. ## Getting Help - Email: [support@presagetech.com](mailto:support@presagetech.com) - [Submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra/issues) - [Docs and FAQ](https://smartspectra.presagetech.com) - [Developer Admin Portal](https://physiology.presagetech.com/auth/login) ## Getting a Good Measurement Source: https://presagetech.com/docs/smartspectra-sdk/measurement-quality --- title: Getting a Good Measurement description: Camera setup, lighting and positioning for accurate SmartSpectra readings, plus how to surface the SDK's live validation feedback to users. --- > **Important:** SDK metrics are offered for general wellness and informational purposes only. SDK metrics have not been cleared by the FDA and may not be used for medical diagnosis or treatment. SmartSpectra estimates pulse, breathing, and related vitals from subtle, per-frame changes in the camera image: small color shifts in skin caused by blood volume pulse, and small movements of the chest and abdomen caused by breathing. Because the signal itself is subtle, capture conditions matter as much as the model — a well-lit, stable, centered shot with an unobstructed face (and chest, for breathing) will consistently produce high-confidence results, while poor lighting, motion, or framing will not. This page is the practical setup guide: what to tell your users, what the SDK tells you back in real time, and what a good vs. a bad capture looks like. For the precise per-metric operating ranges and known limitations, see [Model Cards and Limitations](model-cards-and-limitations.md). ## Quick checklist Point end users at this before their first measurement: | Condition | Requirement | | --- | --- | | **Lighting** | Even, diffuse light on the face. Not too dark, not too bright, no harsh shadows or flicker. | | **Framing** | One face, centered, roughly facing the camera. For breathing/chest metrics, the chest is also visible. | | **Distance** | Close enough that the face isn't tiny in frame, far enough that it isn't clipped or overexposed. | | **Stability** | Camera itself must stay still for breathing/chest metrics. Handheld is fine for pulse, HRV, and the relative arterial pressure waveform only. | | **Subject motion** | Stay still. Avoid talking, chewing gum, or large head/body movement during capture. | | **Clothing** | Avoid all-dark clothing or tight, high-contrast stripes if measuring breathing. | | **Patience** | Some metrics need a full window before they're trustworthy — see [Confidence and warm-up time](#confidence-and-warm-up-time) below. | ![A good SmartSpectra capture setup: centered, evenly lit, camera on a stable stand, face and chest in frame](hero-good-setup.svg) ## Set up the environment ### Lighting Good lighting is the single biggest lever for measurement quality, because the pulse signal is a very small change in skin color. **Do:** - Use even, diffuse light on the face — natural window light or soft indoor lighting both work well. - Light the face from the front or slightly off-axis, not from directly behind. - Keep the light source constant. Avoid sources that visibly flicker, such as some fluorescent lights, or a TV/monitor providing the primary light on the face. **Avoid:** - **Too dark.** Underexposed frames don't carry enough signal, and low light also encourages some cameras to automatically lower their frame rate to lengthen exposure — which can trigger a frame-rate warning even when the picture "looks" acceptable. - **Too bright / backlit.** A window or bright light directly behind the subject silhouettes the face and blows out the parts of the frame the model needs. Overhead-only lighting that casts hard shadows across half the face has the same effect. - **Uneven lighting.** One side of the face brightly lit and the other in shadow reduces signal quality even when the average exposure looks fine. ![Good lighting: evenly lit face with no shadows. Bad lighting: one-sided light casting a hard shadow across the face](lighting-good-vs-bad.svg) ### Framing and distance - Exactly **one** face should be in frame. Multiple faces, or no face at all, both block processing. - The face should be roughly orthogonal to the camera — a face turned far to the side or tilted away doesn't track reliably. - The face shouldn't be so close that it's clipped or fills the whole frame, or so far away that it's a small fraction of the image. - For breathing and chest-based metrics, the upper chest also needs to be visible and unobstructed; the lower abdominal breathing waveform additionally needs the waistline visible. ![Good distance: centered face and chest with comfortable margin. Too close: face fills and clips the frame. Too far: face is a small fraction of the frame](framing-good-vs-bad.svg) ### Camera stability Stability requirements differ by metric: - **Pulse rate, HRV, and the arterial pressure waveform** tolerate a handheld camera as long as it's relatively stable — small hand shake is fine. - **Breathing rate and the chest/abdomen waveforms** need the camera itself to be stationary (e.g., propped on a stand, table, or dock). Handheld capture defeats breathing measurement because chest-motion tracking can't distinguish camera motion from breathing motion. If your app supports both, default to a stable mount so the same session can measure everything requested. ### Subject stillness and behavior - The subject should sit still — large head, body, or camera motion produces unreliable readings with deceptively high confidence, so it's worth coaching users explicitly rather than relying on the SDK to catch every case. - Avoid talking during a breathing measurement; speech changes chest/abdomen motion in ways that don't reflect the breathing signal. - Avoid chewing gum during pulse, HRV, or blood-pressure-waveform measurement. ### Clothing (breathing) Dark clothing absorbs light needed to track subtle chest motion, and tight, high-contrast striped patterns can alias against the camera's pixel grid and get misread as motion. Prefer plain, lighter-colored clothing for breathing measurements. ### Frame rate The on-device pipeline needs a sustained camera frame rate — don't let the camera fall below roughly **25 fps**; target 30 fps for headroom. This is usually automatic on modern devices, but two things can undermine it: - Low light, which can cause the camera's auto-exposure to lengthen exposure time and drop frame rate to compensate — another reason to fix lighting first if you see intermittent frame-rate warnings. - Heavy on-device CPU/GPU contention from other work happening at the same time as capture. ## Confidence and warm-up time Every rate-based metric ships with a `confidence` value (0–100) and a `stable` flag alongside the value itself — treat both as part of the reading, not just the number: - **Confidence starts at 0** and only becomes meaningful once the metric's analysis window has filled: 30 seconds for breathing rate and the breathing waveforms, 60 seconds for HRV, 12 seconds for pulse rate. Don't surface or act on a reading before its window has elapsed — early "0 confidence" samples are expected, not a bug. - **`stable`** indicates whether the underlying detection (face, landmarks, etc.) is currently reliable frame-to-frame. Treat `stable: false` the same way you'd treat a validation warning: something in the capture (framing, motion, lighting) is currently degraded. See [Data Types](data-types.md) for the full payload schema. ## Reading the SDK's live feedback The SDK continuously evaluates the incoming frames and reports a `ValidationStatus` — a stable `code` plus a human-readable `hint` string — so you don't have to guess why a measurement looks bad. Surface the `hint` text directly to end users; it's designed to be shown as-is. | Code | Meaning | What to tell the user | | --- | --- | --- | | `OK` | Frame passes all checks. | Nothing — this is the "good" state. | | `NO_FACE_FOUND` | No face detected in frame. | Get a face into frame, facing the camera. | | `MULTIPLE_FACES_FOUND` | More than one face detected. | Make sure only one person is in frame. | | `FACE_NOT_CENTERED` | Face detected but off-center horizontally (left/right). | Center your face in the frame. | | `TOO_DARK` | Image is underexposed. | Add light, or move to a brighter area. | | `TOO_BRIGHT` | Image is overexposed. | Reduce light, or move away from a bright/backlit source. | | `CHEST_NOT_VISIBLE` | Chest not visible or too far away (breathing/chest metrics). | Reframe so the upper chest is visible. | | `CAMERA_TUNING` | Camera is still auto-adjusting exposure/focus. | Wait a moment before recording. | | `FRAME_RATE_TOO_LOW` | Sustained camera frame rate has dropped too low. | Improve lighting (see above) or reduce concurrent device load. | | `EXCESSIVE_MOTION` | Body or head motion exceeds the reliable-measurement threshold. | Hold still — motion may affect accuracy. | | `FACE_TOO_CLOSE` | Face fills too much of the frame. | Move away from the camera. | | `FACE_TOO_FAR` | Face is smaller than the minimum usable size. | Move closer to the camera. | | `FACE_TOO_HIGH` | Face is too high in the frame (off-center vertically). | Move down, or tilt the camera up. | | `FACE_TOO_LOW` | Face is too low in the frame (off-center vertically). | Move up, or tilt the camera down. | | `FACE_NOT_FORWARD` | Face has turned or tilted too far away from the camera. | Face the camera. | This is the same enum across platforms — Swift's `ValidationCode`, Kotlin's `ValidationCode`, C++'s `spectra::ValidationCode`, and the Node client's `ValidationCodeValue` all carry the same codes and wire values. The table above lists every code you should surface. Each enum also carries one further value, `FACE_SIZE_OUT_OF_RANGE` (wire value 4), which is **deprecated** — it is superseded by `FACE_TOO_CLOSE` and `FACE_TOO_FAR`, which say which direction to move. Handle it if you switch exhaustively, but don't build UI around it. **Swift:** ```swift // sdk.validationStatus is an @Observable property: (code, hint) guard let validationStatus = sdk.validationStatus else { return } statusLabel.text = validationStatus.hint ``` **Kotlin:** ```kotlin sdk.validationStatus.observe(this) { status -> validationLabel.text = status?.hint ?: "--" } ``` **C++:** ```cpp sdk.SetOnValidationStatusChanged( [](const spectra::ValidationStatus& status, int64_t timestamp_us) { std::cerr << "Validation: " << status.hint << "\n"; }); ``` **TypeScript:** ```typescript sdk.on('validationStatus', (code, timestampUs, hint) => { showBanner(hint); }); ``` > **Tip:** Debounce or coalesce repeated identical statuses before showing them — a stationary, > well-positioned subject can still flicker between `OK` and a borderline code frame-to-frame. > Update the UI only when the code actually changes. ## What good vs. bad looks like ![Good capture: a clean, periodic waveform with rising confidence. Bad capture: an irregular, motion-corrupted waveform with low confidence](trace-good-vs-bad.svg) A good pulse or breathing trace looks like a clean, regular, periodic waveform — smoothly repeating peaks roughly in line with a plausible heart or breathing rate, with confidence rising toward 100 as the analysis window fills. A bad trace is irregular and jagged, with no consistent periodicity, confidence that stays low or drops out, and `stable: false` or a non-`OK` `ValidationStatus` during the affected stretch. If a trace looks noisy, check the underlying cause first — camera motion, subject motion, poor lighting, or a partially occluded face/chest — rather than assuming the reading is a valid-but-unusual result. See [Model Cards and Limitations](model-cards-and-limitations.md) for the exact operating range and "works when / does not work when" conditions for each metric. ## Guidance to give your end users The setup guidance above translates directly into a short in-app tutorial. A concrete example (adapted from the SmartSpectra sample apps): 1. Place your device on a stable surface, like a table or stand. 2. Use a well-lit environment — natural daylight works best. 3. Make sure your face is evenly lit with no strong shadows. 4. Avoid bright light sources directly behind you, such as a window or overhead light. 5. Stay still and avoid talking during the measurement. 6. Watch for on-screen feedback — it tells you exactly what to fix if something's off. 7. Start recording when prompted, and follow any auto-restart prompts if feedback appears mid-measurement. ## Related reading - [Model Cards and Limitations](model-cards-and-limitations.md) — per-metric valid ranges, "works when" / "does not work when" conditions, and links to the full model cards. - [Data Types](data-types.md) — the `Measurement`, `MeasurementWithConfidence`, and `ValidationStatus` payload schema. ## Model Cards and Limitations Source: https://presagetech.com/docs/smartspectra-sdk/model-cards-and-limitations --- title: Model Cards and Limitations description: Model cards, operating ranges, and known limitations for SmartSpectra SDK metrics. --- # Model Cards and Limitations > **Important:** SDK metrics are offered for general wellness and informational purposes only. SDK metrics have not been cleared by the FDA and may not be used for medical diagnosis or treatment. The SDK exposes one language-agnostic metrics payload across platforms. This overview summarizes the documented capability surface and calls out validation gaps directly. See the [payload schema](data-types.md) for field names and wire types. For camera setup, lighting, and framing guidance — plus how to read the SDK's live validation feedback — see [Getting a Good Measurement](measurement-quality.md). ## Relative Arterial Pressure Waveform **Output:** Waveform of relative arterial pressure (amplitude in arbitrary units vs. time). Represents waveform shape only (e.g., heartbeat timing, shape and relative changes). Not a blood pressure measurement and cannot be used to estimate systolic or diastolic pressure. **Confidence:** See model card for confidence to error mapping. **Range:** Valid when pulse rate is 40-110 BPM. **Works when:** - Subject is stationary. - Camera is relatively stable; handheld is acceptable. - Face and upper chest are visible, unobstructed, and well illuminated. **Does not work when:** These conditions may produce inaccurate measurements with high confidence and should be avoided. - Large head, body, or camera motion. - Chewing gum. - Flickering illumination, such as TV screens. [Model card](https://vandv.presagetech.com/arterial_pressure_model_card.html) ## Heart Rate Variability **Output:** 60-second beat-to-beat (NN interval) statistics with confidence. Confidence is 0 until the full 60-second window is reached. **Confidence:** See model card for confidence to error mapping. **Metrics:** - Mean NN: Average NN interval in milliseconds. - RMSSD: Root mean square of successive NN differences in milliseconds. - SDNN: Standard deviation of NN intervals in milliseconds. - Baevsky Stress Index: A measure of autonomic balance derived from the NN interval distribution, computed as AMo / (2 × Mo × MxDMn). Interpret it relative to a subject's own baseline rather than as an absolute score. **Range:** Valid when pulse rate is 40-110 BPM. **Works when:** - Subject is stationary. - Camera is relatively stable; handheld is acceptable. - Face and upper chest are visible, unobstructed, and well illuminated. **Does not work when:** These conditions may produce inaccurate measurements with high confidence and should be avoided. - Large head, body, or camera motion. - Chewing gum. - Flickering illumination, such as TV screens. [Model card](https://vandv.presagetech.com/hrv_model_card.html) ## Pulse Rate **Output:** 12-second average pulse rate in beats per minute (BPM) with confidence. **Confidence:** See model card for confidence to error mapping. **Range:** 40-110 BPM. **Works when:** - Subject is stationary. - Camera is relatively stable; handheld is acceptable. - Face and upper chest are visible, unobstructed, and well illuminated. **Does not work when:** These conditions may produce inaccurate measurements with high confidence and should be avoided. - Large head, body, or camera motion. - Chewing gum. - Flickering illumination, such as TV screens. [Model card](https://vandv.presagetech.com/arterial_pressure_model_card.html) ## Face Analysis **Output:** Includes: **Details:** - 478 facial landmark coordinates as x/y pixel locations. - Eye blink detection (binary). - Talking detection (binary). - Facial expression classification with probabilities across anger, contempt, disgust, fear, happiness, sadness, surprise, and neutral. **Works when:** - Face is visible, unobstructed, and well illuminated. **Does not work when:** - Face is significantly rotated and not approximately orthogonal to the camera. [Face landmark numbering reference](https://storage.googleapis.com/mediapipe-assets/documentation/mediapipe_face_landmark_fullsize.png) ## Breathing Rate **Output:** 30-second average breathing rate in breaths per minute with confidence. Confidence is 0 until the full 30-second window is reached. **Confidence:** See model card for confidence to error mapping. **Range:** 5-40 breaths per minute. **Works when:** - Subject is stationary. - Camera is stable and not handheld. - Face and chest are visible and well illuminated. **Does not work when:** - Talking. - Large body motion. - Unstable camera, including handheld capture. - Dark clothing. - Tightly striped patterns on clothing causing image aliasing. [Model card](https://vandv.presagetech.com/breathing_model_card.html) ## Upper and Lower Breathing Waveforms **Output:** Upper chest and lower abdominal breathing waveforms (amplitude in arbitrary units vs. time) with confidence. Confidence is derived from breathing rate confidence and remains 0 until 30 seconds is reached. **Confidence:** See model card for confidence to error mapping. **Range:** Valid for breathing rates of 5-40 breaths per minute. **Works when:** - Subject is stationary. - Camera is stable. - Face and chest are visible and well illuminated. - Lower abdominal waveform requires waistline visibility. **Does not work when:** - Talking. - Large body motion. - Unstable camera, including handheld capture. - Dark clothing. - Tightly striped patterns on clothing causing image aliasing. [Model card](https://vandv.presagetech.com/breathing_model_card.html) ## Redistribute on Linux Source: https://presagetech.com/docs/smartspectra-sdk/redistribute-smartspectra-on-linux --- title: Redistribute on Linux description: Bundle the SmartSpectra Linux C++ SDK tarball into your own .deb so end users install your app on stock Ubuntu without a Presage apt source. --- # Redistribute SmartSpectra on Linux Validated path for a developer to take the published SmartSpectra Linux C++ SDK release tarball, build their own application against it, and ship a self-contained `.deb` that installs on a stock Ubuntu / Mint / Debian host without configuring a Presage apt source. This is the **only** Linux redistribution shape currently supported. Other shapes are not currently validated. ## When to use this path You're a developer who: - Ships a desktop app to enterprise Linux end-users - Wants the end-user install to be one `sudo apt install ./your-app.deb` step - Cannot ask end-users to configure a Presage apt source - Targets Ubuntu 22.04 / 24.04 (Mint 21 / 22). Those are the supported distributions; nothing else is tested. arm64 redistribution remains a follow-up If your end-users run a non-Debian-family distro (Fedora, RHEL, openSUSE, Arch), you're in territory we don't currently validate. ## The example app The public GitHub mirror ships the example as [`cpp/samples/debian-app-example/README.md`](https://github.com/Presage-Security/SmartSpectra/blob/main/cpp/samples/debian-app-example/README.md) with its packaging logic in [`cpp/samples/debian-app-example/CMakeLists.txt`](https://github.com/Presage-Security/SmartSpectra/blob/main/cpp/samples/debian-app-example/CMakeLists.txt). It demonstrates every choice this page documents and is a suitable starting point to copy into your own repository when integrating the SDK into a redistributable `.deb`. Build it standalone against an extracted SDK release tarball: ```bash # 1. Extract the published SDK release tarball # Use the codename-qualified asset name: # - Ubuntu 22.04 / Mint 21 / Debian 11+: linux-jammy-amd64 # - Ubuntu 24.04 / Mint 22: linux-noble-amd64 mkdir -p /tmp/sdk tar -xzf smartspectra-sdk--linux--amd64.tar.gz -C /tmp/sdk # 2. Choose the example source path for your checkout: # - Public GitHub mirror: cpp/samples/debian-app-example cmake -S -B /tmp/dae-build \ -DCMAKE_PREFIX_PATH=/tmp/sdk \ -DCMAKE_BUILD_TYPE=Release cmake --build /tmp/dae-build -j$(nproc) # 3. Produce the .deb (per-codename revision suffix) cd /tmp/dae-build cpack -G DEB # Produces e.g. debian-app-example_0.1.0-noble1_amd64.deb on Ubuntu 24.04 # or debian-app-example_0.1.0-jammy1_amd64.deb on Ubuntu 22.04 — the # revision suffix tracks the build-host codename so the same .deb stream # never serves a mismatched stock-archive Depends set. # 4. Install on a clean Debian-family container sudo apt install -y ./debian-app-example_*.deb ``` Read the example [`CMakeLists.txt`](https://github.com/Presage-Security/SmartSpectra/blob/main/cpp/samples/debian-app-example/CMakeLists.txt) end-to-end — it's the canonical source for the install layout, CPack DEB config, Depends manifest reading, and maintainer-script wiring. ## Install layout The `.deb` installs entirely under `/opt/debian-app-example/` (an app-private prefix per [FHS 3.0 §3.12](https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s12.html)). Replace `debian-app-example` with your own app name when you adapt the example. ```text /opt// ├── bin/ │ └── (the executable) ├── lib/ │ ├── libsmartspectra.so (bundled from SDK tarball) │ └── smartspectra_manifest.json (written at install time) └── share/ └── smartspectra/ └── graph/ ├── models/ (bundled from SDK tarball) └── ... ``` Plus one file outside the app's prefix, written by `postinst`: ```text # /etc/ld.so.conf.d/.conf /opt//lib ``` `postrm` removes that fragment on `remove` / `purge`. ## RPATH and per-app ldconfig Two mechanisms place the bundled `libsmartspectra.so` on the loader's path when the end-user launches the binary. **RPATH on the binary.** The example binary is linked with `INSTALL_RPATH=$ORIGIN/../lib` and `INSTALL_RPATH_USE_LINK_PATH=FALSE` so the resulting `DT_RUNPATH` is exactly `$ORIGIN/../lib`. From `/opt//bin/` this walks to `/opt//lib/`, where `libsmartspectra.so` lives. **ldconfig fragment for `libsmartspectra.so`.** The SDK's shared library has no embedded `DT_RUNPATH` / `DT_RPATH`, and `/opt//lib` is not on the loader's default search path, so `postinst` also registers it with the system loader cache: ```text # /etc/ld.so.conf.d/.conf /opt//lib ``` and runs `ldconfig`, so `libsmartspectra.so` resolves at load time. `postrm` removes the ldconfig fragment on `remove` / `purge` so the cache no longer maps libraries to the now-removed prefix. ## Model-path resolution At runtime the SDK locates its own shared-library directory and reads `smartspectra_manifest.json` from that directory. The manifest contains a single absolute path: ```json { "resource_root_dir": "/opt//share/smartspectra" } ``` The SDK joins `resource_root_dir` with the relative model paths it ships internally, so model lookups land under `/opt//share/smartspectra/...` at runtime. The example's CMake writes this manifest at install time (not build time) so the recorded prefix matches the actual install location, not the build prefix. ## Depends: from SmartSpectraPackageManifest.json The example does **not** hard-code its `Depends:` list. Drift between the SDK's own apt-resolved dependencies and the example's list would silently produce a `.deb` that installs but fails to load the SDK on a stock container. Instead, the example reads `/share/smartspectra/package/SmartSpectraPackageManifest.json` at CMake-configure time and feeds its `debian_depends` field straight into `CPACK_DEBIAN_PACKAGE_DEPENDS`. The manifest is per-codename, so each codename produces its own Depends set from the same example source. A `FATAL_ERROR` guard in the example's CMake refuses to package if the manifest is missing or its `debian_depends` field is empty. The end-user therefore needs no Presage apt source — every transitive dependency resolves from the codename's stock archive at `apt install ./.deb` time on both **jammy** and **noble**. arm64 remains a follow-up. ## See also - Example README: [`cpp/samples/debian-app-example/README.md`](https://github.com/Presage-Security/SmartSpectra/blob/main/cpp/samples/debian-app-example/README.md) - Example packaging logic: [`cpp/samples/debian-app-example/CMakeLists.txt`](https://github.com/Presage-Security/SmartSpectra/blob/main/cpp/samples/debian-app-example/CMakeLists.txt) ## SDK Telemetry & Privacy Source: https://presagetech.com/docs/smartspectra-sdk/telemetry-and-privacy --- title: SDK Telemetry & Privacy description: What the SmartSpectra SDK's aggregate, opt-out session telemetry collects, how to disable it, and the privacy guarantees behind it. --- # SDK Telemetry & Privacy The SmartSpectra SDK can report a small, aggregate diagnostic summary once per measurement session. It exists so we can measure release quality across the devices and conditions the SDK actually runs on — which platforms work well, where sessions fail, and under what conditions. It is designed to be lightweight and privacy-preserving. ## What is collected A single aggregate summary per session, containing only: - **SDK build** — SDK version (with a short commit appended for non-release builds) and package origin. - **Runtime** — platform, OS version, device model, CPU architecture, SDK binding. - **Session shape** — source kind (camera / video file / custom input), outcome (completed / stopped / reset / error), and the *number* of metrics requested. - **Performance** — frame throughput counts, input frame-rate statistics, startup-latency milestones, flow-limiter throughput, and produced-metric counts. - **Quality** — counts of on-device validation states (e.g. lighting or motion warnings) and, for a failed session, a coarse error code. - A wall-clock timestamp for the session. ## What is NOT collected The telemetry is aggregate-only. It never includes: - Raw video or image frames. - Measured metric values of any kind — pulse rate, breathing rate, HRV, the relative arterial pressure waveform, and every other metric the SDK computes. - File paths, prompts, or free-text error messages. - User identifiers or stable device identifiers. Because it carries no stable identifiers and no measurement values, it is not linked to a user's identity and is not used for tracking. For Apple App Store privacy labels this maps to the *Performance Data* and *Product Interaction* types; for Google Play Data Safety it corresponds to *App info and performance* and *App activity* — in both cases not linked to identity and not used for tracking. ## Turning it off Telemetry is opt-out through configuration on supported SDKs. It defaults to on; when disabled, the SDK starts no telemetry session and transmits nothing. Set `enableTelemetry = false` on the SDK config: `SmartSpectraConfig::enable_telemetry` in C++, `enableTelemetry` in Swift/iOS, Kotlin/Android and Node, and `SmartSpectraConfig.EnableTelemetry` in the [.NET wrapper for Windows](https://smartspectra.presagetech.com/docs/cpp/windows/windows-dotnet). Telemetry is sent over the same authenticated, TLS-pinned channel the SDK uses for its other server requests, and only while the SDK has an active server session — never to a separate or unauthenticated destination. Delivery is best-effort: a summary that cannot be sent is dropped and never affects the measurement session. ## Your responsibilities If you distribute an application that embeds the SDK, disclose this collection to your users and complete your app-store data-collection declarations accordingly, consistent with Presage's [Privacy Policy](https://api.physiology.presagetech.com/privacypolicy). ## Node.js and Electron SDK Source: https://presagetech.com/docs/smartspectra-sdk/nodejs --- title: Node.js and Electron SDK sidebarTitle: Overview description: Build Node.js and Electron apps that measure pulse and breathing from a camera on Linux, macOS, and Windows, using a prebuilt native runtime from npm. --- # @smartspectra/node-sdk Pure-FFI Node.js (Electron) binding for SmartSpectra vitals measurement. [koffi](https://koffi.dev) loads the SmartSpectra C ABI shim at runtime — no native addon, no `binding.gyp`, no `electron-rebuild`, no `node-gyp`. ## Start here - Electron sample: [electron-quickstart](https://github.com/Presage-Security/SmartSpectra/tree/main/nodejs/samples/electron-quickstart) - API reference: - Metrics guide: ## Architecture ```text ┌──────────────────────────┐ │ your Node / Electron app │ └───────────┬──────────────┘ │ require('@smartspectra/node-sdk') ┌───────────▼──────────────┐ │ js/index.js │ public surface │ js/smartspectra.js │ SmartSpectraSDK class │ js/ffi.js │ koffi types + Session │ js/resolve-native.js │ shim path resolver └───────────┬──────────────┘ │ koffi.load() ┌────────────────────────────────────────┐ │ @smartspectra/node-sdk--/ │ self-contained native runtime, │ libsmartspectra_capi.* │ shipped as a dependency package; │ …bundled runtime libraries… │ load paths pre-relocated to │ │ @loader_path / $ORIGIN / └────────────────────────────────────────┘ adjacent-DLL search ``` The native runtime ships in per-platform packages (`@smartspectra/node-sdk--`) pulled in as dependencies of the main package. There is **no install script** in the published tarball and no postinstall download; the binding loads the package matching your machine at runtime. SDK-thread callbacks are marshalled onto the V8 event loop by koffi thread-safe trampolines. ## Supported Platforms | Platform | Status | Notes | | --- | --- | --- | | macOS Apple Silicon (`darwin-arm64`) | Supported | Electron and headless Node workflows supported | | Linux x64 (`linux-x64`) | Supported | Requires glibc 2.35+ | | Linux ARM64 (`linux-arm64`) | Supported | Requires glibc 2.35+ | | Windows x64 (`win32-x64`) | Supported | Electron and headless Node workflows supported | ## Common Prerequisites | Requirement | Version | | --- | --- | | Node.js | >= 20 (Node 24 LTS recommended) | | Electron (optional) | >= 28 | You also need an API key from [physiology.presagetech.com](https://physiology.presagetech.com/auth/login). An AI assistant connected to the [SmartSpectra MCP Server](../docs/mcp-server.md) can fetch it from your account for you. The native runtime ships in per-platform npm packages pulled in automatically as dependencies — no install script and no system libraries to install. Supported platforms: `darwin-arm64`, `linux-x64`, `linux-arm64`, `win32-x64`. On Linux the native runtime is built on Ubuntu 22.04 (Jammy), so it requires glibc >= 2.35 — Ubuntu 22.04+, Debian 12+, or any distribution at least that new. glibc and libstdc++ are provided by your system, not bundled. ## Installation ```bash npm install @smartspectra/node-sdk ``` Early adopters can track the latest release candidate with the `rc` dist-tag: ```bash npm install @smartspectra/node-sdk@rc ``` The native packages (`@smartspectra/node-sdk--`) come in as dependencies and the binding loads the one matching your `platform`/`arch` at runtime. Each is self-contained — it bundles everything the binding loads at runtime, with paths already rewritten at publish time, so nothing needs to be on PATH and no system libraries are required. There is no install script. > **Note** — `npm install` downloads the native runtime for every supported > platform (a few hundred MB total), not just your host's. This is expected; > only the package matching your machine is loaded at runtime. If your platform is unsupported (no `@smartspectra/node-sdk--` is published for it), the first `require()` fails with an actionable error naming the missing package. ## Pick Your Integration Path - Electron desktop app: use `@smartspectra/node-sdk/main`, `@smartspectra/node-sdk/preload`, and `@smartspectra/node-sdk/renderer`. - Headless Node process with a local camera: use `@smartspectra/node-sdk` directly with `useCamera(CameraSelection.default)`. - Headless or server-side Node process with host-provided frames: use `useCustomInput()` / `sendFrame()`. - Runnable reference app: [electron-quickstart](https://github.com/Presage-Security/SmartSpectra/tree/main/nodejs/samples/electron-quickstart) ## Camera Quickstart > The snippets below are ES modules (`import` + top-level `await`) — save them as `.mjs`, or > set `"type": "module"` in your `package.json`. In a CommonJS project, use `require()` and > wrap the `await` calls in an `async` function. ```ts import { SmartSpectraSDK, CameraSelection, breathingMetrics, cardioMetrics, decodeMetrics } from '@smartspectra/node-sdk'; const sdk = new SmartSpectraSDK({ apiKey: 'YOUR_API_KEY', requestedMetrics: [...breathingMetrics, ...cardioMetrics], }); sdk.on('processingStatus', (status) => console.log('Processing status:', status)); sdk.on('validationStatus', (code, ts, hint) => console.log('Validation:', code, hint, 'at', ts, 'µs')); sdk.on('metrics', (buf, ts) => { console.log('Metrics at', ts, 'µs:', decodeMetrics(buf)); }); sdk.on('error', (code, message, retryable) => console.error('SmartSpectra error', code, message, 'retryable=', retryable)); sdk.useCamera(CameraSelection.default); sdk.start(); console.log('Measuring from the default camera. Press Ctrl+C to stop.'); // On shutdown: process.on('SIGINT', async () => { await sdk.stopAsync(); await sdk.destroy(); process.exit(0); }); ``` `useCamera(CameraSelection.default)` selects the default camera while stopped; capture begins at `start()`. You can discover cameras without creating an SDK instance: ```ts import { CameraSelection, SmartSpectraSDK } from '@smartspectra/node-sdk'; const cameras = SmartSpectraSDK.availableCameras(); if (cameras.length > 0) { sdk.useCamera(CameraSelection.byId(cameras[0].id), { width: 1280, height: 720 }); } ``` `CameraSelection.default`, `.front`, `.back`, and `.byId(id)` share the camera selection contract across SDKs. Explicit requests never fall back. Stop/start/reset retain the chosen input. Native Node.js supports camera discovery and typed selection on macOS, Linux, and Windows. Linux and Windows cameras may report unknown facing or lens type, so select those cameras by discovered ID. On Windows, `.front` and `.back` fail with `kInputUnavailable` because facing is not reported. See the [migration guide](docs/migration-guide.md) and [API reference](docs/api-reference.md#cameraoptions). Electron renderer capture uses browser-owned streams and `useMediaStream()`; browser device IDs cannot be used for native camera selection. Use the custom-input path below when your app already owns frame capture. ## Custom Input Quickstart To supply frames from another source, replace the camera setup with: ```ts import { FrameTransform, PixelFormat } from '@smartspectra/node-sdk'; sdk.useCustomInput(FrameTransform.kNone); sdk.start(); sdk.sendFrame(rgbBuf, width, height, width * 3, PixelFormat.kRGB, captureTsUs); ``` ## API reference The full API — `SmartSpectraSDK` constructor options, methods, events, error codes, and enums — lives in the [API reference](docs/api-reference.md), generated from the SDK's bundled TypeScript declarations so it tracks the published package. For which metrics to request and how to read the decoded payloads, see the [metrics guide](docs/metrics.md). For natural-language analysis of the measured vitals, see the [LLM Insights guide](docs/llm-insights.md). ## Electron integration Runnable sample at [electron-quickstart](https://github.com/Presage-Security/SmartSpectra/tree/main/nodejs/samples/electron-quickstart). See its [README](https://github.com/Presage-Security/SmartSpectra/blob/main/nodejs/samples/electron-quickstart/README.md). The package ships these entry points: ```text @smartspectra/node-sdk → SmartSpectraSDK (low-level, main process) @smartspectra/node-sdk/main → bindSmartSpectraIpc(window) @smartspectra/node-sdk/preload → preload bridge (contextBridge) @smartspectra/node-sdk/renderer → SmartSpectraSDK (renderer-side, MediaStream input) @smartspectra/node-sdk/messages → decodeMetrics(buf) + the generated Metrics class ``` ### Main process ```ts import { app, BrowserWindow } from 'electron'; import { bindSmartSpectraIpc } from '@smartspectra/node-sdk/main'; app.whenReady().then(() => { const win = new BrowserWindow({ webPreferences: { preload: require.resolve('@smartspectra/node-sdk/preload'), contextIsolation: true, sandbox: true, }, }); bindSmartSpectraIpc(win); win.loadFile('index.html'); }); ``` `bindSmartSpectraIpc` listens on a single private IPC channel, accepts the MessagePort the preload ships from the renderer, and owns one `SmartSpectraSDK` per renderer connection. SDK teardown happens automatically on window close. ### Renderer ```ts import { SmartSpectraSDK } from '@smartspectra/node-sdk/renderer'; import { breathingMetrics, cardioMetrics } from '@smartspectra/node-sdk'; const sdk = new SmartSpectraSDK({ apiKey: 'YOUR_KEY', requestedMetrics: [...breathingMetrics, ...cardioMetrics], }); sdk.on('streamAvailable', (stream) => { videoEl.srcObject = stream; }); sdk.on('metrics', (buf, ts) => { /* render dashboard */ }); sdk.on('validationStatus', (code, ts, hint) => { /* show user hint */ }); sdk.on('error', (code, msg, retryable) => { /* surface in UI */ }); await sdk.start(); // SDK acquires the front camera + emits 'streamAvailable' await sdk.requestInsight('How is my breathing?'); await sdk.stop(); // SDK releases the camera; next start() re-acquires sdk.destroy(); ``` Defaults: front-facing camera at 1280x720 / 30 fps, AE/AWB/focus locked once the graph reports `Running`. Call `sdk.useMediaStream(stream)` before `sdk.start()` to override with a virtual camera, `.captureStream()`, `desktopCapturer`, etc. Host-supplied streams are managed by the host; SDK-acquired streams are released on `stop()` / `reset()` / `destroy()`. The renderer uses `MediaStreamTrackProcessor` + `OffscreenCanvas` to extract RGBA pixels and ships each frame to the main process via the MessagePort. Graph callbacks flow back through the same port. Each control call (`start` / `stop` / `reset` / `requestInsight`) waits for an ack from the main process. If the main process crashes or stalls, the call rejects after `sendTimeoutMs` (default `30000`) with an `Error` whose `code` is `'SMARTSPECTRA_IPC_TIMEOUT'`, so the UI surfaces "main process unresponsive" instead of hanging forever. Pass `sendTimeoutMs` to the constructor to tune it, or `0` to disable. ### Preload If the app already has a preload script, require this module from it: ```js require('@smartspectra/node-sdk/preload'); ``` Otherwise point `webPreferences.preload` directly: ```ts preload: require.resolve('@smartspectra/node-sdk/preload') ``` ### Content Security Policy The renderer runs a Web Worker from a `blob:` URL. If your app sets a CSP, allow `blob:` in `worker-src`: ```html ``` A `Refused to create a worker from 'blob:…'` console error at `start()` time means the policy is blocking the worker. ### Permission flow Camera permission flows through Electron's native handler — the SDK calls `navigator.mediaDevices.getUserMedia()` (or the host's stream, if supplied via `useMediaStream()`) and Electron surfaces the OS prompt + indicator LED. To grant programmatically: ```ts import { session } from 'electron'; session.defaultSession.setPermissionRequestHandler((wc, permission, callback, details) => { // Grant only the camera, and only to your own bundled page — deny everything // else so a window that later loads remote content can't auto-grant the camera. const url = (details && details.requestingUrl) || (wc && wc.getURL()) || ''; callback(permission === 'media' && url.startsWith('file://')); }); ``` On macOS, add `NSCameraUsageDescription` to `Info.plist`. Windows and Linux need no additional entitlements. ## Electron desktop packaging The installed platform package `node_modules/@smartspectra/node-sdk--/` ships the full native closure — every library `libsmartspectra_capi` needs at runtime, with install_names / RPATHs pre-relocated to `@loader_path` (macOS), `$ORIGIN` (Linux), or adjacent-directory search (Windows). Include that directory as an extra resource and you're done. ### electron-builder Use one block per target OS. electron-builder's `${platform}` macro expands to the **build host's** platform, not the build target — so a single `${platform}` entry ships the wrong (or no) closure on a cross-build (e.g. `electron-builder --win` on a Mac). The per-OS `mac`/`win`/`linux` blocks are applied only to their matching target and are cross-build-safe: ```jsonc // package.json { "build": { "mac": { "extraResources": [{ "from": "node_modules/@smartspectra/node-sdk-darwin-${arch}/", "to": "smartspectra/" }] }, "win": { "extraResources": [{ "from": "node_modules/@smartspectra/node-sdk-win32-${arch}/", "to": "smartspectra/" }] }, "linux": { "extraResources": [{ "from": "node_modules/@smartspectra/node-sdk-linux-${arch}/", "to": "smartspectra/" }] } } } ``` ### electron-forge ```js // forge.config.js module.exports = { packagerConfig: { extraResource: [ `node_modules/@smartspectra/node-sdk-${process.platform}-${process.arch}/`, ], }, }; ``` > **Cross-builds:** `process.platform`/`process.arch` resolve at config-load > time to the **host**, not the build target — so this single line is correct > only for native (per-OS-CI) builds. To cross-build, switch on the target > (`--platform`/`--arch` passed to `electron-forge package`) and emit the > matching `node_modules/@smartspectra/node-sdk-/` path. ### macOS code signing The bundled `.dylib` files are ad-hoc signed so dyld will load them; for distribution you'll want to re-sign with your own identity. With electron-builder: ```jsonc { "build": { "mac": { "hardenedRuntime": true, "entitlements": "build/entitlements.mac.plist", "extendInfo": { "NSCameraUsageDescription": "Vitals measurement" } } } } ``` In `entitlements.mac.plist`, allow loading the bundled dylibs: ```xml com.apple.security.cs.disable-library-validation ``` Alternative: re-sign every `.dylib` under `node_modules/@smartspectra/node-sdk-darwin-arm64/` with your own identity before packaging (preferable for notarization-strict deployments). ### Windows code signing Authenticode-sign each `.dll` under `node_modules/@smartspectra/node-sdk-win32-x64/` alongside your app's main executable. Windows resolves adjacent DLLs first, so placing them in the same directory as `.exe` is the simplest layout. ### Linux No signing required. The bundled `.so` files have `$ORIGIN` RPATHs so they resolve siblings without any `LD_LIBRARY_PATH` plumbing. **glibc floor — Ubuntu 22.04 (Jammy, `GLIBC_2.35`).** The runtime closure is built on a Jammy base, so a packaged app (e.g. an AppImage) runs on Ubuntu 22.04 and any newer distribution. glibc and the C++ runtime come from the host system — they are not vendored in the platform package. ## Logging SDK log verbosity defaults to warnings and errors only. To change it, pass `logLevel` in the SDK options: ```js const { SmartSpectraSDK, SmartSpectraLogLevel } = require('@smartspectra/node-sdk'); const sdk = new SmartSpectraSDK({ apiKey: '...', logLevel: SmartSpectraLogLevel.kInfo, }); ``` Levels are cumulative — `kDebug`, `kInfo`, `kWarning` (default), `kError`, `kNone`. `kDebug` cannot restore debug-only statements compiled out of the release engine binary. ## Performance notes - koffi adds ~100 ns per FFI call — negligible for `sendFrame()` at 30 fps. - SDK callbacks arrive on worker threads; koffi marshals them onto the V8 event loop, so listeners run on the main thread. - In Electron, the renderer-side SDK (`@smartspectra/node-sdk/renderer`) captures frames in the renderer and ships them to the main process — simplest to wire and fine for typical use. For the lowest latency at high resolution, capture in the main process instead (`desktopCapturer` or an off-screen renderer), since renderer→main IPC adds frame-time latency that scales with resolution. ## Troubleshooting | Symptom | Likely cause | Fix | | --- | --- | --- | | `the native runtime package "@smartspectra/node-sdk--" is not installed` on require | platform unsupported, or a partial/offline install dropped the dependency | Confirm your platform is supported, then re-run `npm install` | | `Cannot open shared object file` / `LoadLibrary failed` on require | platform package present but its bundled closure is incomplete or corrupt | Reinstall the platform package (`npm install`) | | `kAuthenticationFailed` | API key issue | Check the key + the keychain entitlement on macOS | | `kNonMonotonicTimestamp` | Wall-clock timestamps | Use `process.hrtime.bigint() / 1000n` (monotonic) | ## Support - Docs site: - GitHub issues: - Email: ## Node.js API Reference Source: https://presagetech.com/docs/smartspectra-sdk/nodejs/api-reference --- title: Node.js API Reference description: API reference for the SmartSpectra Node.js SDK — the SmartSpectraSDK class, events, metrics decoding, and error codes. sidebarTitle: API Reference --- ## SmartSpectraSDK Vitals measurement entry point. Owns one processing pipeline and surfaces lifecycle events via `on()`. Lifecycle methods throw a JS `Error` on failure with numeric `code` (a `SmartSpectraErrorCode` value), `message`, and boolean `retryable` properties. ```ts import { SmartSpectraSDK, PixelFormat, decodeMetrics } from '@smartspectra/node-sdk'; const sdk = new SmartSpectraSDK({ apiKey: 'YOUR_KEY' }); sdk.on('metrics', (buf, ts) => console.log(decodeMetrics(buf))); sdk.useCustomInput(); sdk.start(); sdk.sendFrame(rgbBuf, width, height, stride, PixelFormat.kRGB, timestampUs); await sdk.destroy(); ``` ### Methods - ```typescript static availableCameras(): CameraInfo[] ``` No SDK instance, authentication, or capture required. Throws when discovery is unsupported. - ```typescript constructor(options?: SmartSpectraOptions) ``` - ```typescript start(): void ``` Initialize and begin the input configured with useCamera, useCustomInput, or useFile. - ```typescript stop(): void ``` Request the session to stop. Idempotent. - ```typescript stopAsync(): Promise ``` Async variant of stop() — the native stop blocks until the pipeline drains; prefer this in event-loop-sensitive contexts. - ```typescript reset(): void ``` Rebuild after kError. Clears the source unless useCamera(selection, options) opted into retention. After typed selection, throws kInvalidState while stopAsync() is pending; await it before resetting. - ```typescript waitUntilComplete(timeoutMs?: number): boolean ``` Block until the session reaches a terminal state (idle or error), or the timeout elapses. Returns true if the session settled, false on timeout. `timeoutMs <= 0` (the default) waits indefinitely. Use after `useFile()` + `start()` to block until end-of-file. - ```typescript destroy(): Promise ``` Tear down the session. Idempotent. Await before constructing a replacement session when teardown ordering matters, since native SDK state is process-global. - ```typescript requestInsight(text: string): number ``` Dispatch an on-demand insight prompt; returns the request id. The matching Insight response arrives asynchronously through the `'insight'` event with the same id. - ```typescript useCustomInput(frameTransform?: FrameTransformValue): this ``` Select the custom frame-push input source (push frames via `sendFrame()` after `start()`). Returns `this` for chaining; call before `start()`. - ```typescript useCamera(selection: CameraSelection, options?: Omit): this ``` Select a live camera as the input source. The SDK opens the camera and pumps frames internally on `start()` — no `sendFrame()` needed. Returns `this` for chaining. Captures in THIS process — for Electron, prefer the renderer SDK's `useMediaStream()`. Select while stopped; explicit requests never fall back. Selection is resolved at startup and retained across stop/start/reset. Invalid or busy requests preserve the previous source. Pass CameraSelection.default for SDK selection. - ```typescript useCamera(options?: CameraOptions): this ``` Select a camera by legacy index (default 0). Reset clears the source. @deprecated Pass an explicit CameraSelection and separate capture options instead. - ```typescript useFile(videoPath: string, options?: VideoFileOptions): this ``` Select a pre-recorded video file as the input source. `start()` begins playback (non-blocking, on SDK worker threads); use `waitUntilComplete()` or watch `'processingStatus'` for the idle transition to detect end-of-file. Returns `this` for chaining. - ```typescript sendFrame( buffer: Uint8Array | Buffer, width: number, height: number, stride: number, pixelFormat: PixelFormatValue, timestampUs: number, ): boolean ``` Submits a raw video frame. Requires `useCustomInput()` + `start()` first. - ```typescript on(event: 'processingStatus', callback: (status: ProcessingStatusValue) => void): this ``` Register a callback for a named event. Replaces any previously registered callback for the same event. Returns `this` for chaining. - ```typescript on(event: 'validationStatus', callback: (code: ValidationCodeValue, timestampUs: number, hint: string) => void): this ``` - ```typescript on(event: 'metrics', callback: (buf: Buffer, timestampUs: number) => void): this ``` - ```typescript on(event: 'accumulatedMetrics', callback: (buf: Buffer, timestampUs: number) => void): this ``` - ```typescript on(event: 'insight', callback: (buf: Buffer, requestId: number) => void): this ``` - ```typescript on(event: 'error', callback: (code: SmartSpectraErrorCodeValue, message: string, retryable: boolean) => void): this ``` - ```typescript on(event: 'frameSentThrough', callback: (sent: boolean, timestampUs: number) => void): this ``` - ```typescript on(event: 'videoOutput', callback: (buf: Buffer, width: number, height: number, stride: number, pixelFormat: PixelFormatValue, timestampUs: number) => void): this ``` ### Properties - ```typescript static readonly version: string ``` SDK package version. - ```typescript readonly processingStatus: ProcessingStatusValue ``` Current ProcessingStatus integer value. ## SmartSpectraOptions Options passed to the SmartSpectra constructor. ```typescript apiKey?: string ``` API key for server-validated auth. ```typescript requestedMetrics?: number[] ``` MetricType integer codes. Defaults to `breathingMetrics` when omitted. ```typescript enableAccumulatedOutput?: boolean ``` Also emit an accumulated metrics packet at the end of each session. ```typescript logLevel?: SmartSpectraLogLevelValue ``` Verbosity of SDK logging, applied when the session initializes. Defaults to SmartSpectraLogLevel.kWarning (warnings and errors only). ```typescript enableTelemetry?: boolean ``` Aggregate SDK telemetry. Defaults to `true`; set `false` to opt out. ## VideoFileOptions Playback options for useFile(). ```typescript timestampsPath?: string | null ``` Path to a per-frame timestamps file (one timestamp per line). ```typescript interframeDelayMs?: number ``` Throttle between frames in ms; omit/0 = as fast as possible. ```typescript startOffsetMs?: number ``` Seek this far into the file before playback (ms); omit/0 = start. ```typescript maxDurationMs?: number ``` Stop after this much content (ms); omit/0 = no limit. ```typescript frameTransform?: FrameTransformValue ``` Spatial transform applied to every frame. ## CameraOptions Capture settings, separate from camera selection. ```typescript deviceIndex?: number ``` @deprecated Use useCamera(CameraSelection.byId(id), options) instead. ```typescript width?: number ``` Capture width in px; omit/0 = SDK default. ```typescript height?: number ``` Capture height in px; omit/0 = SDK default. ```typescript fps?: number ``` Capture frame rate; omit/0 = SDK default. ```typescript frameTransform?: FrameTransformValue ``` Spatial transform applied to every frame. ## FrameTransform Frame transform applied by the SDK to every pushed frame. - `readonly kNone: 0` - `readonly kRotate90CW: 1` - `readonly kRotate90CCW: 2` - `readonly kRotate180: 3` - `readonly kMirrorHorizontal: 4` - `readonly kMirrorVertical: 5` ## PixelFormat Pixel format of a raw frame buffer passed to sendFrame(). - `readonly kRGB: 0` - `readonly kBGR: 1` - `readonly kRGBA: 2` - `readonly kBGRA: 3` - `readonly kNV12: 4` - `readonly kNV21: 5` - `readonly kYUYV: 6` ## ProcessingStatus Processing lifecycle status. Integer values are stable across SDK versions. - `readonly kUninitialized: 0` - `readonly kIdle: 1` - `readonly kStarting: 2` - `readonly kRunning: 3` - `readonly kStopping: 4` - `readonly kError: 5` ## ValidationCode Measurement-readiness codes delivered via the 'validationStatus' event. - `readonly kOk: 0` - `readonly kNoFaceFound: 1` - `readonly kMultipleFacesFound: 2` - `readonly kFaceNotCentered: 3` - `readonly kFaceSizeOutOfRange: 4` - `readonly kTooDark: 5` - `readonly kTooBright: 6` - `readonly kChestNotVisible: 7` - `readonly kCameraTuning: 10` - `readonly kFrameRateTooLow: 11` - `readonly kExcessiveMotion: 12` - `readonly kFaceTooClose: 13` - `readonly kFaceTooFar: 14` - `readonly kFaceTooHigh: 15` - `readonly kFaceTooLow: 16` - `readonly kFaceNotForward: 17` ## SmartSpectraErrorCode Error codes on errors thrown by lifecycle methods and delivered via the 'error' event. - `readonly kOk: 0` - `readonly kInvalidState: 1` - `readonly kAuthenticationFailed: 2` - `readonly kConfigurationFailed: 3` - `readonly kCreditExhausted: 4` - `readonly kNetworkError: 5` - `readonly kServerError: 6` - `readonly kInputUnavailable: 7` - `readonly kProcessingFailed: 8` - `readonly kFrameConversionFailed: 9` - `readonly kNonMonotonicTimestamp: 10` - `readonly kTimestampGap: 11` ## decodeMetrics() Deserialize a protobuf Metrics buffer from the 'metrics' or 'accumulatedMetrics' events. Returns the decoded message if a class has been registered via `setMetricsClass`, otherwise the raw Buffer. `export declare function decodeMetrics(buf: Buffer): unknown` ## setMetricsClass() Register a protobuf Metrics class exposing `deserializeBinary(buf)` (google-protobuf) or `decode(buf)` (protobufjs). `decodeMetrics()` will use it; otherwise the raw Buffer is returned. `export declare function setMetricsClass(cls: unknown): void` ## CameraInfo Discovery snapshot. IDs are opaque and platform-local; rediscover before reuse. ```typescript readonly id: string ``` ```typescript readonly name: string | null ``` ```typescript readonly facing: CameraFacing ``` ```typescript readonly lensType: CameraLensType ``` ## SmartSpectraLogLevel Verbosity of SDK logging, set via the SmartSpectraSDK `logLevel` option. Levels are cumulative: a level shows its own messages plus everything more severe. - `readonly kDebug: 0` - `readonly kInfo: 1` - `readonly kWarning: 2` - `readonly kError: 3` - `readonly kNone: 4` ## MetricType MetricType integer codes accepted by `requestedMetrics`. Runtime values are generated from protobuf. - `readonly CHEST_BREATHING: 0` - `readonly ABDOMEN_BREATHING: 1` - `readonly BREATHING_RATE: 2` - `readonly BREATHING_AMPLITUDE: 3` - `readonly APNEA: 4` - `readonly RESPIRATORY_LINE_LENGTH: 5` - `readonly BASELINE: 6` - `readonly INHALE_EXHALE_RATIO: 7` - `readonly EDA_TRACE: 10` - `readonly FACE_LANDMARKS: 11` - `readonly BLINKING: 12` - `readonly TALKING: 13` - `readonly EXPRESSIONS: 14` - `readonly PULSE_RATE: 15` - `readonly ARTERIAL_PRESSURE_TRACE: 16` - `readonly HRV: 17` ## CameraSelection - `readonly default: Readonly<{ kind: 'default' }>` - `readonly front: Readonly<{ kind: 'front' }>` - `readonly back: Readonly<{ kind: 'back' }>` - `readonly byId: (id: string) => Readonly<{ kind: 'byId'; id: string }>` ## ProcessingStatusValue ```typescript export type ProcessingStatusValue = typeof ProcessingStatus[keyof typeof ProcessingStatus]; ``` ## SmartSpectraLogLevelValue ```typescript export type SmartSpectraLogLevelValue = typeof SmartSpectraLogLevel[keyof typeof SmartSpectraLogLevel]; ``` ## PixelFormatValue ```typescript export type PixelFormatValue = typeof PixelFormat[keyof typeof PixelFormat]; ``` ## ValidationCodeValue ```typescript export type ValidationCodeValue = typeof ValidationCode[keyof typeof ValidationCode]; ``` ## SmartSpectraErrorCodeValue ```typescript export type SmartSpectraErrorCodeValue = typeof SmartSpectraErrorCode[keyof typeof SmartSpectraErrorCode]; ``` ## FrameTransformValue ```typescript export type FrameTransformValue = typeof FrameTransform[keyof typeof FrameTransform]; ``` ## MetricTypeValue ```typescript export type MetricTypeValue = typeof MetricType[keyof typeof MetricType]; ``` ## CameraFacing ```typescript export type CameraFacing = 'front' | 'back' | 'unknown'; ``` ## CameraLensType Best-effort classification. Missing metadata and multi-lens cameras are unknown; independent of digital zoom. ```typescript export type CameraLensType = 'unknown' | 'wideAngle' | 'ultraWide' | 'telephoto'; ``` ## Headless Testing in CI on Node.js Source: https://presagetech.com/docs/smartspectra-sdk/nodejs/headless-testing-in-ci --- title: Headless Testing in CI on Node.js description: Run a full SmartSpectra measurement in CI by feeding a recorded video through the Node.js SDK. sidebarTitle: Headless Testing in CI --- # Headless Testing in CI (Node.js) See [Headless Testing in CI](../../docs/headless-testing-in-ci.md) for the cross-platform overview of what's automatable and why. This page covers the Node.js specifics. ## What you can automate The Node.js SDK accepts a **recorded video file** in place of a live camera via `useFile()`, so CI can run a **full video-fed measurement**: the SDK plays the file through the same pipeline as a live camera, emits `'metrics'` events as readings arrive, and settles to idle at end-of-file. No camera or display is needed — a stock `ubuntu-latest`-style runner works. Keep the assertion **smoke-level**: check that real readings appeared (a pulse rate and a breathing rate), not their exact values. ## The headless script A complete runnable check — feed a video, wait for end-of-file, and exit non-zero if no readings came out: ```js // headless-smoke.mjs import { SmartSpectraSDK, ProcessingStatus, breathingMetrics, cardioMetrics, } from '@smartspectra/node-sdk'; // Import decodeMetrics from the `/messages` entry point: it always decodes // using the bundled Metrics class. The root export's decodeMetrics returns // the raw Buffer unless you first register a class with setMetricsClass(), // which would make every field read below undefined and fail the check. import { decodeMetrics } from '@smartspectra/node-sdk/messages'; const sdk = new SmartSpectraSDK({ apiKey: process.env.SMARTSPECTRA_API_KEY, requestedMetrics: [...breathingMetrics, ...cardioMetrics], }); let sawPulse = false; let sawBreathing = false; sdk.on('metrics', (buf) => { const metrics = decodeMetrics(buf); if (metrics.cardio?.pulseRate?.length) sawPulse = true; if (metrics.breathing?.rate?.length) sawBreathing = true; }); sdk.on('error', (code, message, retryable) => { console.error(`SmartSpectra error ${code}: ${message} (retryable=${retryable})`); process.exitCode = 1; }); // Resolves when the session settles: playback runs the file through the // pipeline and transitions back to idle at end-of-file (or to error). const settled = new Promise((resolve) => { let started = false; sdk.on('processingStatus', (status) => { if (status === ProcessingStatus.kRunning) started = true; if (started && (status === ProcessingStatus.kIdle || status === ProcessingStatus.kError)) { resolve(); } }); }); sdk.useFile(process.argv[2] ?? './test-assets/face.mp4'); sdk.start(); // non-blocking: playback runs on SDK worker threads await settled; await sdk.destroy(); if (!sawPulse || !sawBreathing) { console.error('FAIL: no pulse/breathing readings were produced'); process.exit(1); } console.log('OK: pulse and breathing readings appeared'); ``` Notes on the shape: - The script is fully event-driven: `start()` returns immediately, the Node event loop stays free while SDK worker threads play the file, and the `'processingStatus'` idle transition marks end-of-file. - The `'error'` listener turns an SDK-level failure (bad key, no network) into a non-zero exit instead of a silent zero-readings pass. ## The recorded video Supply your own short clip — around 30–60 seconds of a well-lit, mostly still face, framed like a real measurement (long enough for rates to compute; a few seconds isn't) — and keep it in your repo's test assets. Use a widely-supported container/codec such as MP4 (H.264); the SDK decodes it automatically. See the [metrics guide](metrics.md) for which metrics to request and how to read the decoded payloads. ## A CI pipeline, in general terms 1. **Expose the API key** as a job secret. 2. **Install the SDK** — `npm install @smartspectra/node-sdk` pulls in the per-platform native runtime; nothing else to install. 3. **Run the script** against your recorded video. 4. **Fail the job** on a non-zero exit. A minimal, provider-neutral sketch (GitHub Actions): ```yaml name: smartspectra-nodejs-headless-smoke on: [push] jobs: headless: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20 - name: Install SmartSpectra run: npm install @smartspectra/node-sdk - name: Headless measurement smoke env: SMARTSPECTRA_API_KEY: ${{ secrets.SMARTSPECTRA_API_KEY }} run: node headless-smoke.mjs ./test-assets/face.mp4 ``` ## Limitations - **No offline mode.** The measurement authenticates against the SmartSpectra service, so the CI runner needs network access. - **Linux runners need glibc 2.35+** (Ubuntu 22.04+, Debian 12+); the native runtime is built on Ubuntu 22.04. `ubuntu-latest` qualifies. - **Smoke, not accuracy.** A recorded-clip run confirms the integration and model pipeline end to end; it is not an accuracy benchmark. ## LLM Insights on Node.js Source: https://presagetech.com/docs/smartspectra-sdk/nodejs/llm-insights --- title: LLM Insights on Node.js description: Ask natural-language questions about a measurement and receive LLM Insights through the SmartSpectra Node.js SDK, alongside the metrics stream. sidebarTitle: LLM Insights --- # Node.js LLM Insights Platform-specific usage for the Node.js SDK (`@smartspectra/node-sdk`). For what LLM Insights are, the request/response model, required metrics, and the privacy notice, see the [LLM Insights overview](../../docs/llm-insights/index.md). The main-process (`@smartspectra/node-sdk`) and Electron renderer (`@smartspectra/node-sdk/renderer`) entry points expose the same API; only the call/return shapes differ (noted below). ## Enable the required metrics Insights summarize the buffered vitals, so breathing (the default set) and cardio must both be active. Select them via `requestedMetrics`: ```js import { SmartSpectraSDK, breathingMetrics, cardioMetrics } from '@smartspectra/node-sdk'; const sdk = new SmartSpectraSDK({ apiKey: 'YOUR_KEY', requestedMetrics: [...breathingMetrics, ...cardioMetrics], }); ``` `cardioMetrics` includes `ARTERIAL_PRESSURE_TRACE`, which drives the pulse waveform. Omitting `requestedMetrics` measures breathing only. ## Request an insight Call `requestInsight` on a configured session. It returns the request ID used to correlate the asynchronous response: ```js // main process: requestInsight(text: string): number // renderer process: requestInsight(text: string): Promise const requestId = sdk.requestInsight('Summarize my current vital signs and flag anything unusual.'); // await in the renderer ``` The prompt is combined with the latest buffered metrics when they exist, otherwise sent prompt-only. ## Receive responses Register a single `'insight'` callback. It receives **both** the auto-fired periodic vitals insights and on-demand responses: ```js sdk.on('insight', (buf, requestId) => { // buf is a serialized `presage.smartspectra.Insight` protobuf. // The Node SDK does not ship a decoder — decode it with your own protobuf // type built from the Insight schema (see Data Types below). const insight = Insight.decode(buf); // e.g. a protobufjs-generated type you supply const text = insight.analysis || insight.error; }); ``` - `buf` is the serialized `Insight` message — a `Buffer` in the main process, a `Uint8Array` in the renderer. - `requestId` is also passed as a plain number, so you can route responses without decoding. - `on('insight', …)` registers a single callback (a second call replaces it). ## Reading the Insight Decode `buf` using the [Insight schema](../../docs/data-types.md#insight), then read the fields. With a protobufjs type the fields are camelCase — `analysis` and `error` are a `result` oneof (exactly one is set), with `requestId`, `processedAt`, and `type` alongside. Every insight is currently delivered with `type` == `INSIGHT_TYPE_VITALS`, so correlate on-demand replies via `requestId`, not `type`. The first auto-fired insight arrives once pulse rate has a stable reading, typically about 15 seconds after processing starts; further snapshots follow as breathing rate and HRV stabilise, then every 60 seconds. Allow at least 15 seconds of valid measurement before an on-demand request can be grounded in the user's physiology. Note that the service may return no insight for a given request (for example when nothing new is worth surfacing), in which case the callback does not fire. ## See also - [LLM Insights overview](../../docs/llm-insights/index.md) - [Node.js API reference](api-reference.md) - [Data Types](../../docs/data-types.md) ## Configuring Metrics on Node.js Source: https://presagetech.com/docs/smartspectra-sdk/nodejs/metrics --- title: Configuring Metrics on Node.js description: Request pulse and breathing rates, HRV and EDA metrics from the SmartSpectra Node.js SDK, and read event results. sidebarTitle: Configuring Metrics --- The Node.js SDK passes requested metric codes through to the native SmartSpectra C++ SDK. Omitting `requestedMetrics` uses the default breathing bundle. ## Breathing and Pulse ### Request Metrics Request the breathing bundle plus the cardio bundle (pulse rate, arterial pressure trace, HRV): ```typescript import { SmartSpectraSDK, breathingMetrics, cardioMetrics } from '@smartspectra/node-sdk'; const sdk = new SmartSpectraSDK({ apiKey: 'YOUR_API_KEY', requestedMetrics: [...breathingMetrics, ...cardioMetrics], }); ``` The bundle exports (`breathingMetrics`, `cardioMetrics`, `faceMetrics`, `edaMetrics`) contain the `MetricType` integer codes defined in `metric_types.proto`. `defaultSupportedMetrics` is the bundle used when `requestedMetrics` is omitted (currently equal to `breathingMetrics`). ### Read Metrics Read the latest breathing and pulse samples from the `metrics` event: ```typescript import { decodeMetrics } from '@smartspectra/node-sdk'; sdk.on('metrics', (buf, timestampUs) => { const metrics = decodeMetrics(buf) as any; if (Buffer.isBuffer(metrics)) return; const breathingRate = metrics.breathing?.rate?.at(-1)?.value; const chestTrace = metrics.breathing?.upperTrace?.at(-1)?.value; const abdomenTrace = metrics.breathing?.lowerTrace?.at(-1)?.value; const pulseRate = metrics.cardio?.pulseRate?.at(-1)?.value; }); ``` Cardio fields are empty unless you request a cardio metric such as `PULSE_RATE`. ## Advanced Combine bundles or hand-pick `MetricType` codes for finer control: ```typescript import { SmartSpectraSDK, breathingMetrics, cardioMetrics, faceMetrics, edaMetrics, } from '@smartspectra/node-sdk'; const sdk = new SmartSpectraSDK({ apiKey: 'YOUR_API_KEY', requestedMetrics: [ ...breathingMetrics, ...cardioMetrics, ...faceMetrics, ...edaMetrics, ], }); ``` Read the advanced fields from the same decoded metrics object: ```typescript sdk.on('metrics', (buf, timestampUs) => { const metrics = decodeMetrics(buf) as any; if (Buffer.isBuffer(metrics)) return; const pressureTrace = metrics.cardio?.arterialPressureTrace?.at(-1)?.value; const hrvRmssd = metrics.cardio?.hrv?.at(-1)?.rmssd; const edaTrace = metrics.eda?.trace?.at(-1)?.value; const faceLandmarks = metrics.face?.landmarks?.at(-1)?.value; const blinking = metrics.face?.blinking?.at(-1)?.detected; const talking = metrics.face?.talking?.at(-1)?.detected; const expression = metrics.face?.expression?.at(-1); }); ``` ### Advanced Payload Shape `decodeMetrics` returns the generated protobuf payload as a JavaScript object. Requested advanced metrics populate these fields: ```typescript type Metrics = { breathing?: Breathing; eda?: Eda; face?: Face; cardio?: Cardio; }; type Cardio = { pulseRate?: MeasurementWithConfidence[]; arterialPressureTrace?: MeasurementWithConfidence[]; hrv?: Hrv[]; }; type Hrv = { rmssd: number; meanNn: number; sdnn: number; baevsky: number; timestamp: number; confidence: number; stable: boolean; }; type Eda = { trace?: Measurement[]; }; type Face = { landmarks?: Landmarks[]; blinking?: DetectionStatus[]; talking?: DetectionStatus[]; expression?: Expression[]; }; ``` EDA Proxy may take longer to produce its first sample than breathing or cardio outputs. See [Data Types](../../docs/data-types.md) for the complete protobuf schema. ## Node.js Migration Guide Source: https://presagetech.com/docs/smartspectra-sdk/nodejs/migration-guide --- title: Node.js Migration Guide description: "Migrate the SmartSpectra Node.js SDK to CameraSelection, cross-platform camera discovery, stopped-only source changes, and specific usage errors." sidebarTitle: Migration Guide --- # SmartSpectra Node.js SDK Migration Guide ## Node.js SDK v3.4.0 camera selection The existing `useCamera()` and `useCamera({ deviceIndex: ... })` calls remain available, with deprecation warnings in TypeScript. They retain index-based capture and the previous lifecycle behavior. Opt in to the new API by passing an explicit selection and separate capture options. Camera IDs are opaque and local to the native platform; do not convert old integer indices to strings or reuse browser device IDs. ```ts import { CameraSelection, SmartSpectraSDK } from '@smartspectra/node-sdk'; // Discovery requires no SDK instance, authentication, capture, or permission prompt. const cameras = SmartSpectraSDK.availableCameras(); const sdk = new SmartSpectraSDK({ apiKey: 'YOUR_API_KEY' }); sdk.useCamera(CameraSelection.default); sdk.useCamera(CameraSelection.front); sdk.useCamera(CameraSelection.back); if (cameras.length > 0) { sdk.useCamera(CameraSelection.byId(cameras[0].id), { width: 1280, height: 720, fps: 30, }); } sdk.start(); ``` Discovery returns `CameraInfo` records containing `id`, nullable `name`, `lensType`, and `facing` (`'front'`, `'back'`, or `'unknown'`). External webcams may report unknown facing. Visibility depends on platform permissions; a snapshot does not reserve a camera or guarantee that later capture will succeed. `lensType` is `'wideAngle'`, `'ultraWide'`, `'telephoto'`, or `'unknown'`. Classification is best effort; missing metadata and cameras combining multiple lenses report `'unknown'`. It does not describe digital zoom or change selection. Use it for picker labels and continue selecting by `id`. Native discovery and typed selection are supported on macOS, Linux, and Windows. Default selection follows each platform's native discovery order. Linux uses compatible V4L2 capture nodes in numeric device order. Windows uses the first enumerated camera. Explicit front/back/ID requests never fall back; an unavailable selection fails startup with `kInputUnavailable`. Linux populates facing only when the camera driver provides orientation metadata. Windows currently reports unknown facing and lens type, so `.front` and `.back` fail with `kInputUnavailable`; use `.byId(id)` for a specific Windows camera. Empty IDs, NUL-containing IDs, invalid selectors, and invalid capture options fail selection with `kConfigurationFailed`. The deprecated Node.js `deviceIndex` option and older index-based C ABI remain compatible. Do not pass `deviceIndex` in the new overload's capture options. ### Select sources while stopped After opting in with `useCamera(selection, options)`, `useCamera`, `useFile`, and `useCustomInput` reject changes during startup, processing, asynchronous stop, and shutdown with `kInvalidState`. Recover an error state with `stop()` or `reset()` before selecting again. A rejected request preserves the previously selected source. The SDK copies camera selection and capture settings, so mutating your original options afterward has no effect. ```ts await sdk.stopAsync(); sdk.useCamera(CameraSelection.back); sdk.start(); ``` Configure a source before calling `start()`, as before. For legacy callers, `reset()` clears the source, so select one again before restarting. After opting in with `useCamera(selection, options)`, `stop()` and `reset()` retain the selected source and its settings; call `start()` to use it again. Calling the deprecated `useCamera(options)` overload while stopped restores the legacy lifecycle. After typed selection, `reset()` throws `kInvalidState` while a `stopAsync()` call is pending. Await that stop before resetting. Legacy callers retain the ability to request a reset while an asynchronous stop is pending. ### Electron renderer The renderer API continues to use browser capture and `useMediaStream(stream)`. These native selectors apply to the package root API, not the renderer entry point. Keep browser camera IDs and native camera IDs within their respective capture paths. ### Usage failures now report specific errors Usage entitlement checking continues throughout a measurement as before. In v3.4.0, terminal usage failures delivered through the `error` event use these specific codes: - `SmartSpectraErrorCode.kAuthenticationFailed` for rejected credentials - `SmartSpectraErrorCode.kCreditExhausted` for an explicit quota denial - `SmartSpectraErrorCode.kNetworkError` when entitlement cannot be refreshed in time The SDK also includes short measurements and the final partial interval before `stop()` in usage reporting. No public API changes are required. ## Swift Quick Start Source: https://presagetech.com/docs/smartspectra-sdk/swift --- title: Swift Quick Start description: Measure pulse and breathing from the iPhone camera or custom AVFoundation, CVPixelBuffer, and CMSampleBuffer input with the SmartSpectra Swift SDK. Requires iOS 17 or later. sidebarTitle: Quick Start --- # SmartSpectra Swift Quickstart This repo contains two build guides that produce similar user end states: - [Option 1: API Key](docs/option-1-api-key.md) - [Option 2: OAuth](docs/option-2-oauth.md) The only difference in builds is that the API key build gets up and running very fast, but hard-codes your API key. The OAuth build is more suitable for production deployments because it avoids hard-coding your API key. ## Scope These quickstarts intentionally request only: - `SmartSpectraConfig.breathingMetrics` - `SmartSpectraConfig.cardioMetrics` - `MetricType.expressions` Please see the detailed documents for additional features. To supply frames from your own camera or video decoder, use [`useCustomInput()`](docs/headless-mode.md#use-your-own-camera-or-video-source). The SDK also supports its default camera capture through `start()`. The demo app's [Video Testing sample](https://github.com/Presage-Security/SmartSpectra/blob/main/swift/samples/demo-app/VideoInput/VideoTestingView.swift) decodes a selected clip and submits frames through this public API. ## Important Implementation Rules Start by creating a new iOS app project named `Cool Vitals`. The Quick Start is intended so that the developer can replace `Cool Vitals/ContentView.swift` as a full file. - Import `SwiftUI`, `SmartSpectra`, and `AVFoundation`. - Use `let sdk = SmartSpectraSDK.shared`. - Buffer pulse, breathing, arterial pressure, chest, and abdomen samples locally before drawing charts. ## Logging SDK log verbosity defaults to warnings and errors only. To change it, set `logLevel` on the SDK config: ```swift SmartSpectraSDK.shared.config.logLevel = .info ``` Levels are cumulative — `.debug`, `.info`, `.warning` (default), `.error`, `.none`. `.debug` cannot restore debug-only statements compiled out of the release engine binary. ## Choose Your Guide Use [Option 1: API Key](docs/option-1-api-key.md) for the fastest manual setup. Use [Option 2: OAuth](docs/option-2-oauth.md) if you need OAuth. Either way, an AI assistant connected to the [SmartSpectra MCP Server](../docs/mcp-server.md) can do the account side for you — fetch your API key, or register your bundle ID and Apple team ID and download `PresageService-Info.plist`. ## LLM Insights See [LLM Insights](docs/llm-insights.md) for natural-language analysis of the measured vitals. ## Swift API Reference Source: https://presagetech.com/docs/smartspectra-sdk/swift/api-reference --- title: Swift API Reference description: API reference for the SmartSpectra Swift SDK — observable state, lifecycle methods, metrics, and error codes. sidebarTitle: API Reference --- ## SmartSpectraSDK Entry point for the SmartSpectra SDK. Most apps use the shared instance: ```swift let sdk = SmartSpectraSDK.shared sdk.config.apiKey = "your-key" try await sdk.start() ``` Tests and advanced integrations can create an isolated instance via ``init(config:)``. The underlying authentication handler and C++ preprocessing runtime are process-global, so only one SDK instance can drive an active measurement at a time. ### Initializers - ```swift public init (config: SmartSpectraConfig) ``` Create an isolated SDK instance backed by the given configuration. The authentication handler and preprocessing runtime are process-global, so only one instance can drive an active measurement at a time. Use this initializer for tests or advanced integrations where you intentionally need a separate observable state from ``shared``. Most apps should just use ``SmartSpectraSDK/shared``. ### Methods - ```swift public func start () async throws ``` Begin processing the selected input (device camera by default). In custom mode, await this call before submitting frames. - ```swift public func stop () async throws ``` Stop processing. Call ``start()`` again to resume. - ```swift public func useCustomInput (frameTransform: FrameTransform = .none) throws -> CustomInput ``` Select caller-owned video input while stopped. No SDK camera discovery, capture, or camera permission is required. Selecting a source invalidates older custom-input handles. Throws `.invalidState` during startup or processing. - ```swift public func useCamera (_ selection: CameraSelection) throws ``` Select camera capture while stopped. Default prefers front, then the first discovered camera. Explicit selections never fall back. Selection is resolved at capture startup; an unavailable camera fails start with `.inputUnavailable`. Throws `.configurationFailed` for an empty ID and `.invalidState` while busy. Rejected requests preserve the previous source. Stop/start/reset retain the selection until another source is selected. This call does not open a camera. - ```swift @available(*, deprecated, message: "Pass an explicit CameraSelection, such as .default or .byId(id).") public func useCamera () throws ``` Select camera capture using the legacy config.cameraPosition behavior. - ```swift public nonisolated static func availableCameras () throws -> [CameraInfo] ``` Discover cameras without constructing or authenticating an SDK instance, requesting permission, or starting capture. Visibility depends on platform permissions; a snapshot does not guarantee capture will succeed later. Throws `.inputUnavailable` if discovery fails. - ```swift public func reset () async throws ``` Stop processing and clear measurement output, retaining configuration, the selected source, transform, and custom-input handle. Safe while idle; await before restarting. - ```swift @discardableResult public func requestInsight (_ text: String) throws -> Int32 ``` Dispatch an on-demand insight request alongside the vitals samples buffered since the last send. The provided `text` is sent as the current turn's prompt; prior calls' text is not retained or re-sent by the SDK. The delivered `Insight` always has `type == .vitals` today (`.speech` and `.combined` are reserved and not emitted). Vitals snapshots auto-fire — one as each of pulse rate, breathing rate, and HRV first stabilises, then one every 60 seconds — carrying a fixed SDK-supplied prompt rather than caller text, and are published through ``SmartSpectraSDK/insight`` alongside on-demand replies, so match a reply to its prompt with `requestID`, not `type`. - ```swift @_spi(Testing) public func setVideoInput (path: String) ``` Configures a video file as input source instead of the live camera. Supported formats: .mov, .mp4, .qt - ```swift @_spi(Testing) public func setVideoTimestampInput (path: String) ``` Configures optional timestamp file (one ms value per line) for frame timing. - ```swift @_spi(Testing) public func setVideoInterframeDelay (milliseconds: Int) ``` Configures optional video-file playback throttling in milliseconds. - ```swift @_spi(Testing) public func setVideoInputEnabled (_ enabled: Bool) ``` Selects testing file input or camera input while stopped. An active-session selection is rejected and reported through `error`. ### Properties - ```swift public static let shared = SmartSpectraSDK() ``` The shared SDK instance. - ```swift public internal(set) var metrics : Metrics? ``` Real-time metrics emitted while processing is active. - ```swift public internal(set) var imageOutput : UIImage? ``` Live camera preview image. - ```swift public internal(set) var processingStatus : ProcessingStatus = .idle ``` Current processing pipeline status. - ```swift public internal(set) var error : SmartSpectraError? ``` Latest error from the SDK. - ```swift public internal(set) var validationStatus : ValidationStatus? ``` Measurement readiness: face position, lighting, etc. Only meaningful while ``processingStatus`` is `.running`. - ```swift public internal(set) var insight : Insight? ``` Latest insight from the AI insights service. - ```swift public nonisolated static var version : String ``` SDK version string. - ```swift public let config : SmartSpectraConfig ``` Configuration for this SDK instance. Mutate properties on this object rather than replacing it. ## SmartSpectraConfig Configuration for ``SmartSpectraSDK``. Access configuration through ``SmartSpectraSDK/config`` rather than constructing one directly: ```swift let sdk = SmartSpectraSDK.shared sdk.config.apiKey = "your-key" ``` Standalone construction is available for tests and advanced integrations that pass a custom config into ``SmartSpectraSDK/init(config:)``. ### Initializers - ```swift public init () ``` Creates a configuration with default values. Prefer ``SmartSpectraSDK/config`` over constructing a standalone config unless you are driving an isolated ``SmartSpectraSDK`` instance. ### Properties - ```swift @available(*, deprecated, message: "Use sdk.useCamera(.front), .back, or .byId instead.") public var cameraPosition : AVCaptureDevice.Position = .front ``` Legacy camera position, used until an explicit camera selector is chosen. - ```swift public var logLevel : SmartSpectraLogLevel = .default ``` Verbosity of SDK logging — both the SDK's Swift-side logging and the native engine. Takes effect immediately when set (like ``imageOutputEnabled``). Defaults to ``SmartSpectraLogLevel/warning`` (warnings and errors only). - ```swift public var imageOutputEnabled : Bool = true ``` Controls whether the SDK publishes preview frames to ``SmartSpectraSDK/imageOutput``. When disabled, camera frames are still processed for vitals analysis, but the SDK skips `CVPixelBuffer` to `UIImage` conversion and preview updates. This is useful for custom integrations that do not display a live camera feed. Example: ```swift let sdk = SmartSpectraSDK.shared sdk.config.imageOutputEnabled = false ``` - Note: Changes take effect immediately and do not require restarting processing. - ```swift public var enableTelemetry : Bool = true ``` Controls whether the SDK reports anonymous, aggregate usage telemetry. Telemetry is on by default; set this to `false` to opt out. When enabled, the SDK sends a per-session summary. It does not include raw frames, metric values, file paths, user identifiers, or device identifiers. Reporting is best-effort and never blocks or affects a measurement session. Example: ```swift let sdk = SmartSpectraSDK.shared sdk.config.enableTelemetry = false ``` - Note: Read when a measurement session starts. - ```swift public var apiKey : String? ``` API key from the Presage developer portal. Setting this automatically configures authentication. - ```swift public var requestedMetrics : [MetricType]? ``` Which metrics to compute during processing. This property provides granular control over which metric families the SDK requests. Derived feature flags such as cardio and face processing are inferred from the selected metric types. Duplicate metrics are removed automatically while preserving the original order. If you do not set this property, the SDK defaults to breathing-only metrics: chest breathing, abdomen breathing, breathing rate, breathing amplitude, apnea, respiratory line length, baseline, and inhale/exhale ratio. Example: ```swift let sdk = SmartSpectraSDK.shared sdk.config.requestedMetrics = [ .breathingRate, .pulseRate, .faceLandmarks ] ``` Example using the predefined bundles: ```swift config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics ``` - ```swift public nonisolated static let breathingMetrics : [MetricType] ``` Breathing metric bundle. Equivalent to leaving ``SmartSpectraConfig/requestedMetrics`` unset. - ```swift public nonisolated static let cardioMetrics : [MetricType] ``` Cardio metric bundle (pulse rate, arterial pressure trace, HRV). Combine with ``breathingMetrics`` for the typical "vitals" bundle. - ```swift public nonisolated static let faceMetrics : [MetricType] ``` Face metric bundle (landmarks, blinking, talking, expressions). - ```swift public nonisolated static let edaMetrics : [MetricType] ``` EDA Proxy (electrodermal activity) trace metric bundle. ## CustomInput Caller-owned video input obtained from ``SmartSpectraSDK/useCustomInput(frameTransform:)``. The handle survives stop/start and reset. Selecting another source invalidates it. Await `sdk.start()` before submitting frames from a serial worker queue. Keep each buffer alive and unchanged until submission returns; the SDK retains no borrowed pixels. Calls are serialized with native teardown. This type is safe to share across threads; callers remain responsible for ordering timestamps and synchronizing buffer mutations. ### Methods - ```swift public func sendFrame (_ pixelBuffer: CVPixelBuffer, timestampUs: Int64) -> FrameSubmissionResult ``` Submits BGRA or 8-bit bi-planar NV12 (video or full range) pixels. Timestamps are nonnegative, strictly increasing microseconds on one monotonic timeline per run, below `Int64.max - 2`. Gaps greater than two seconds are rejected; stop and start again after an interruption. Row padding and separate NV12 planes are supported. NV12 is converted to BGRA before submission. Unsupported formats return `.rejected` with `.frameConversionFailed`. Pixels must already have the desired orientation, or select a session transform. Camera position and preview mirroring do not transform caller-owned frames. - ```swift public func sendFrame (_ sampleBuffer: CMSampleBuffer) -> FrameSubmissionResult ``` Submits the image and presentation timestamp of an uncompressed video sample. The timestamp is converted to microseconds; invalid, indefinite, infinite, negative, or out-of-range timestamps are rejected. Orientation is not inferred from metadata. ## FrameTransform Spatial transform applied to every frame in a custom-input session. - `case none = 0` - `case rotate90CW = 1` - `case rotate90CCW = 2` - `case rotate180 = 3` - `case mirrorHorizontal = 4` - `case mirrorVertical = 5` ## ProcessingStatus Indicates the current state of the preprocessing pipeline. - `case idle` - `case starting` - `case running` - `case stopping` - `case error` ## ValidationStatus Measurement readiness: a stable code plus a human-readable hint. Orthogonal to ``ProcessingStatus`` (engine lifecycle). ### Properties - ```swift public let code : ValidationCode ``` The stable, machine-readable readiness code. - ```swift public let hint : String ``` A human-readable hint describing what needs to change for a valid measurement. ## ValidationCode Measurement-readiness codes. - `case ok = 0` - `case noFaceFound = 1` - `case multipleFacesFound = 2` - `case faceNotCentered = 3` - `case faceSizeOutOfRange = 4` - `case tooDark = 5` - `case tooBright = 6` - `case chestNotVisible = 7` - `case cameraTuning = 10` - `case frameRateTooLow = 11` - `case excessiveMotion = 12` - `case faceTooClose = 13` - `case faceTooFar = 14` - `case faceTooHigh = 15` - `case faceTooLow = 16` - `case faceNotForward = 17` ## SmartSpectraError A typed error from the SmartSpectra SDK. Thrown by ``SmartSpectraSDK/start()`` and ``SmartSpectraSDK/stop()``, and published on ``SmartSpectraSDK/error`` for async pipeline failures. ### Properties - ```swift public let code : Code ``` The error category. - ```swift public let message : String ``` A human-readable description of what went wrong. - ```swift public let retryable : Bool ``` Whether the operation that produced this error can be retried. - ```swift public var errorDescription : String? ``` A human-readable description of the error; returns ``message``. ## SmartSpectraError.Code SDK error codes. No `.ok` case — in Swift, success means no error thrown. Raw values are stable across SDK versions and match the C++/Android wire values. - `case invalidState = 1` - `case authenticationFailed = 2` - `case configurationFailed = 3` - `case creditExhausted = 4` - `case networkError = 5` - `case serverError = 6` - `case inputUnavailable = 7` - `case processingFailed = 8` - `case frameConversionFailed = 9` - `case nonMonotonicTimestamp = 10` - `case timestampGap = 11` ## CameraSelection A camera request resolved when capture starts. Explicit selections never fall back. - `` case `default` `` - `case front` - `case back` - `case byId(String)` ## CameraFacing Direction reported by the camera. External cameras may report `.unknown`. - `case front` - `case back` - `case unknown` ## CameraLensType Best-effort lens classification, independent of digital zoom. Missing metadata and cameras combining multiple lenses report `.unknown`. - `case unknown` - `case wideAngle` - `case ultraWide` - `case telephoto` ## CameraInfo A discoverable camera. IDs are opaque and device-local; availability can change. ### Initializers - ```swift public init (id: String, name: String? = nil, facing: CameraFacing = .unknown, lensType: CameraLensType = .unknown) ``` ### Properties - ```swift public let id : String ``` - ```swift public let name : String? ``` - ```swift public let facing : CameraFacing ``` - ```swift public let lensType : CameraLensType ``` ## FrameSubmissionResult Acceptance means the pixels were consumed, not that measurement processing has finished. - `case accepted` - `case rejected(SmartSpectraError)` ## SmartSpectraLogLevel Verbosity of SDK logging, set via ``SmartSpectraConfig/logLevel``. Levels are cumulative: a level shows its own messages plus everything more severe. The setting covers both the SDK's Swift-side logging and the native engine. ``debug`` cannot restore debug-only statements that were compiled out of the release engine binary. Raw values are stable across SDK versions and match the C++/Android wire values. - `case debug = 0` - `case info = 1` - `case warning = 2` - `case error = 3` - `case none = 4` ## Headless Mode on Swift Source: https://presagetech.com/docs/smartspectra-sdk/swift/headless-mode --- title: Headless Mode on Swift description: Use SmartSpectra with your own iOS UI, AVFoundation camera, CVPixelBuffer, or CMSampleBuffer video input. sidebarTitle: Headless Mode --- # Headless Mode (iOS) The SDK doesn't ship UI. `SmartSpectraSDK.shared` is observable — read its properties (`metrics`, `validationStatus`, `error`, `imageOutput`, `processingStatus`) directly from SwiftUI views and SwiftUI auto-tracks reads. Outside SwiftUI, use `withObservationTracking` and re-arm the observation after each change. The sample apps include a measurement UI; your own integration looks however you want. Use this when you want to: - Monitor vitals in the background while the app shows other content - Build a custom measurement UI ## Use Your Own Camera or Video Source Use this for a camera shared with another feature, or for frames decoded by your app. The demo app's [Video Testing sample](https://github.com/Presage-Security/SmartSpectra/blob/main/swift/samples/demo-app/VideoInput/VideoTestingView.swift) shows public frame submission from `AVAssetReader`, including clip orientation, presentation timestamps, rejected frames, cancellation, and source cleanup. Select custom input while the SDK is stopped. Await `start()` before sending frames; you do not need to wait for `processingStatus == .running`. ```swift import AVFoundation import SmartSpectra final class CameraFrames: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate { private let input: CustomInput init(input: CustomInput) { self.input = input } func captureOutput( _ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection ) { switch input.sendFrame(sampleBuffer) { case .accepted: break case .rejected(let error): // Handle invalid state, unsupported pixels, or timestamp errors. print("Frame rejected: \(error.code): \(error.message)") } } } @MainActor func connectCameraOutput(_ output: AVCaptureVideoDataOutput) async throws -> CameraFrames { let sdk = SmartSpectraSDK.shared sdk.config.apiKey = "YOUR_API_KEY" let input = try sdk.useCustomInput(frameTransform: .none) try await sdk.start() let receiver = CameraFrames(input: input) output.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA] output.alwaysDiscardsLateVideoFrames = true output.setSampleBufferDelegate(receiver, queue: DispatchQueue(label: "camera.frames")) return receiver // Retain this receiver for as long as capture runs. } ``` Configure and attach the output to your app's capture session before calling `connectCameraOutput`. Start capture afterward, on your capture session's worker queue. To finish, stop capture and detach the delegate, drain the delegate queue, then await `sdk.stop()` before releasing the receiver. Call `try sdk.useCamera(.default)` while stopped if the next measurement should use SDK capture. Serialize these lifecycle operations and handle thrown errors; frame-error callbacks run on the delegate queue, so dispatch UI updates to the main actor. The host app owns its capture session, camera permissions, preview, and camera settings. Configure the capture connection to deliver upright pixels, or select a `FrameTransform` when calling `useCustomInput()`. Avoid applying the same rotation twice. Custom input itself requires no SDK camera permission or capture session; normal SDK authentication is still required. For a decoded video or another source, submit a pixel buffer with its timestamp: ```swift let result = input.sendFrame(pixelBuffer, timestampUs: presentationTimestampUs) ``` | Input | Supported layout | | --- | --- | | `CVPixelBuffer` | `kCVPixelFormatType_32BGRA`, including padded rows | | `CVPixelBuffer` | 8-bit bi-planar NV12, full or video range, even dimensions | | `CMSampleBuffer` | A ready, uncompressed video sample containing one of these pixel formats | The sample-buffer overload uses the presentation timestamp. Explicit timestamps are nonnegative, strictly increasing microseconds below `Int64.max - 2`, on one monotonic timeline per run. Do not use wall-clock arrival times. Duplicate or backward timestamps return `.nonMonotonicTimestamp`; gaps over two seconds return `.timestampGap`. Stop and start again after an interruption to begin a fresh timeline. Invalid or unsupported pixels return `.frameConversionFailed`. Submit from a serial worker queue. Keep pixels unchanged until `sendFrame` returns; afterward you can release or reuse them. `.accepted` confirms input consumption, not completed measurement output. Observe SDK metrics separately. The handle survives `stop()`/`start()` and `reset()`. Reset stops processing and clears measurement output while preserving configuration and the selected input. Calling `useCamera(.default)` or selecting another custom input invalidates older handles. Source selection is stopped-only and throws `.invalidState` during startup, measurement, or teardown. Stop the host's frame delivery before switching sources; the SDK does not stop the host's camera. ## Processing Status Lifecycle states: | Status | Meaning | | --- | --- | | **Idle** | Pipeline is not running | | **Starting** | Pipeline is initializing | | **Running** | Actively measuring — data is flowing | | **Stopping** | Teardown in progress, will return to Idle | | **Error** | Something went wrong | ## Example Use `SmartSpectraSDK.shared` directly for headless processing: ```swift import SwiftUI import SmartSpectra struct HeadlessExample: View { private let sdk = SmartSpectraSDK.shared @State private var isMonitoring = false @State private var showCameraFeed = false init() { sdk.config.apiKey = "YOUR_API_KEY" // The default camera selection prefers front when available. } var body: some View { VStack { if let metrics = sdk.metrics, metrics.hasBreathing, let rate = metrics.breathing.rate.last { Text("Breathing: \(Int(rate.value.rounded())) bpm") } if let metrics = sdk.metrics, metrics.hasCardio, let pulse = metrics.cardio.pulseRate.last { Text("Pulse: \(Int(pulse.value.rounded())) bpm") } Text("Status: \(sdk.validationStatus?.hint ?? "")") if let error = sdk.error { Text(error.message) .foregroundStyle(.red) } Toggle("Camera Preview", isOn: $showCameraFeed) .onChange(of: showCameraFeed) { sdk.config.imageOutputEnabled = showCameraFeed } if showCameraFeed, let image = sdk.imageOutput { Image(uiImage: image) .resizable() .scaledToFit() } Button(isMonitoring ? "Stop" : "Start") { isMonitoring.toggle() if isMonitoring { Task { try? await sdk.start() } } else { Task { try? await sdk.stop() } } } // Disable the start button when the SDK has an unrecoverable // input-unavailable error (e.g. camera permission denied). // All other error states either recover on `start()` or // return a throwable error that you can surface to the user. .disabled(sdk.error?.code == .inputUnavailable && !isMonitoring) } } } ``` ## Reading Metrics `sdk.metrics` is the same observable property in headless integrations as elsewhere — there's no separate "headless" API. See [iOS Metrics](metrics.md) for the metric request configuration and the field-by-field reading guide. ## Headless Testing in CI on iOS Source: https://presagetech.com/docs/smartspectra-sdk/swift/headless-testing-in-ci --- title: Headless Testing in CI on iOS description: Test SmartSpectra on the iOS Simulator with public custom input, decoded CVPixelBuffer or CMSampleBuffer frames, or testing-only video playback. sidebarTitle: Headless Testing in CI --- # Headless Testing in CI (iOS) See [Headless Testing in CI](../../docs/headless-testing-in-ci.md) for the cross-platform overview of what's automatable and why. This page covers the iOS specifics. ## What's different on iOS For a caller-owned decoder or capture pipeline, use the public [`useCustomInput()` API](headless-mode.md#use-your-own-camera-or-video-source). It accepts decoded `CVPixelBuffer` and `CMSampleBuffer` frames without a testing opt-in or SDK camera access. The file-playback helper below remains testing-only. The SDK normally measures from the live camera, but it also ships a **testing-only video-input API**: point it at a recorded video file and it plays the clip through the same pipeline as a live camera. A CI simulator has no camera — with video input, that no longer matters, so CI can run a **full video-fed measurement** in an XCTest. The API is gated behind an SPI group so it can't leak into production code by accident: it is only visible to targets that opt in with `@_spi(Testing) import SmartSpectra`. ```swift @_spi(Testing) import SmartSpectra sdk.setVideoInput(path: path) // .mov, .mp4, .qt sdk.setVideoTimestampInput(path: ts) // optional: one ms value per line sdk.setVideoInputEnabled(true) // select file input while stopped ``` While video input is enabled the SDK does not open the camera, so the test needs no camera hardware and no camera permission. Two levels of CI coverage, pick per test: 1. **[Video-fed measurement](#option-1-the-video-fed-test)** — a full measurement from a recorded clip, asserting that real readings came out. 2. **[Build-integration smoke](#option-2-the-build-integration-smoke)** — no clip needed; proves the SDK builds, launches, and initializes. ## Option 1: The video-fed test ### Public custom input Your test can decode its own clip and submit frames with no SPI import. Use `import SmartSpectra`, select `try sdk.useCustomInput()` while stopped, and await `sdk.start()` before the first frame. In your decoder's serial loop: ```swift switch input.sendFrame(sampleBuffer) { case .accepted: break case .rejected(let error): XCTFail("Frame rejected: \(error.code): \(error.message)") throw error } ``` Use the sample's presentation timestamp and pace decoding to the clip's timing. For a pixel buffer, call `input.sendFrame(pixelBuffer, timestampUs: timestampUs)`. Assert that requested metrics appear, not merely that frames were accepted. Stop submission before awaiting `sdk.reset()`, then call `try sdk.useCamera(.default)` to restore the shared SDK for later tests. Perform cleanup on failure as well as success. See the [custom-input contract](headless-mode.md#use-your-own-camera-or-video-source) and the [app-owned decoder sample](https://github.com/Presage-Security/SmartSpectra/blob/main/swift/samples/demo-app/VideoInput/VideoTestingView.swift). Choose a decoder and codec supported by your test device or simulator; the SDK accepts uncompressed frames and does not decode files through this public API. ### Testing-only file playback Drive `SmartSpectraSDK.shared` directly, the same way you would for any [headless integration](headless-mode.md), from an XCTest hosted by your app on the **iOS Simulator**. Feed the clip, poll the observable `metrics` property, and assert that real readings appeared — a pulse rate and a breathing rate — not their exact values. ```swift import XCTest @_spi(Testing) import SmartSpectra final class VideoMeasurementTests: XCTestCase { @MainActor func testMeasuresFromRecordedVideo() async throws { let sdk = SmartSpectraSDK.shared sdk.config.apiKey = ProcessInfo.processInfo.environment["SMARTSPECTRA_API_KEY"] ?? "" // The default request is breathing-only; ask for cardio too so a // pulse rate can appear. See the metrics guide. sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics // A short clip of a well-lit, mostly still face, bundled with the // test target. let videoURL = try XCTUnwrap( Bundle(for: Self.self).url(forResource: "face", withExtension: "mov"), "face.mov missing from the test bundle" ) sdk.setVideoInput(path: videoURL.path) sdk.setVideoInputEnabled(true) do { try await sdk.start() var sawPulse = false var sawBreathing = false let deadline = Date().addingTimeInterval(120) while Date() < deadline, !(sawPulse && sawBreathing) { if let metrics = sdk.metrics { if metrics.hasCardio, metrics.cardio.pulseRate.contains(where: { $0.value > 0 }) { sawPulse = true } if metrics.hasBreathing, metrics.breathing.rate.contains(where: { $0.value > 0 }) { sawBreathing = true } } try await Task.sleep(for: .milliseconds(250)) } try await sdk.stop() XCTAssertTrue(sawPulse, "no pulse reading came out of the recorded clip") XCTAssertTrue(sawBreathing, "no breathing reading came out of the recorded clip") } catch { try await sdk.reset() try sdk.useCamera(.default) throw error } try await sdk.reset() try sdk.useCamera(.default) } } ``` ## Option 2: The build-integration smoke If you don't have a recorded clip yet (or want a faster job on every push), skip the video calls entirely and keep the check at smoke level — no SPI import needed: ```swift import XCTest import SmartSpectra final class HeadlessSmokeTests: XCTestCase { @MainActor func testSDKInitializesHeadless() async throws { let sdk = SmartSpectraSDK.shared sdk.config.apiKey = ProcessInfo.processInfo.environment["SMARTSPECTRA_API_KEY"] ?? "" do { try await sdk.start() try await sdk.stop() } catch let error as SmartSpectraError { print("SmartSpectra reported: \(error.message)") } } } ``` The simulator has no camera, so don't assert on a measurement result here: `start()` returning at all — whether it succeeds or throws a typed `SmartSpectraError` — is the smoke signal that the SDK built, launched, and initialized correctly end to end. ## The recorded video Supply your own short clip and keep it in your test assets: - Around **30–60 seconds** of a **well-lit, mostly still face**, framed like a real measurement — long enough for the pipeline to compute rates (a measurement runs about 30 seconds); a clip of only a few seconds won't produce readings. - Container/codec: `.mov`, `.mp4`, or `.qt`; H.264 is a safe choice. - Record in **standard (video/limited) range** — full-range clips can be rejected by the brightness validation before a measurement starts. - Frame timing comes from the clip itself. If your clip needs an external time base, supply a sidecar file via `setVideoTimestampInput(path:)` with one millisecond timestamp per line, one line per frame. See [iOS Metrics](metrics.md) for which metrics to request and how to read them. ## A CI pipeline, in general terms 1. **Expose the API key** as a job secret. 2. **Run the test target headlessly** on a booted iOS Simulator via `xcodebuild test`. 3. **Fail the job** if the test target doesn't build or the test fails. A minimal, provider-neutral sketch (GitHub Actions) — adapt the scheme name and simulator to your project: ```yaml name: smartspectra-ios-headless-video on: [push] jobs: headless: runs-on: macos-14 steps: - uses: actions/checkout@v4 - name: Video-fed measurement test env: SMARTSPECTRA_API_KEY: ${{ secrets.SMARTSPECTRA_API_KEY }} run: | xcodebuild test \ -scheme YourAppVideoTests \ -destination 'platform=iOS Simulator,name=iPhone 15' \ -only-testing:YourAppVideoTests/VideoMeasurementTests ``` ## Limitations - **Testing only.** The video-input API is SPI-gated for a reason: keep `@_spi(Testing)` imports out of production targets. The API may change without a migration path. - **No offline mode.** Like every SmartSpectra SDK, a measurement authenticates against the SmartSpectra service, so the runner needs network access. - **iOS 26 simulator decoder issue.** Video decoding on the iOS 26 simulator can fail with a `-12900` decoder error; run video-fed tests on an iOS 17 or 18 simulator. - **Smoke, not accuracy.** A recorded-clip run confirms the integration and model pipeline end to end; it is not an accuracy benchmark. ## LLM Insights on Swift Source: https://presagetech.com/docs/smartspectra-sdk/swift/llm-insights --- title: LLM Insights on Swift description: Ask natural-language questions about a measurement and receive LLM Insights through the SmartSpectra Swift SDK, alongside the metrics stream. sidebarTitle: LLM Insights --- # Swift LLM Insights Platform-specific usage for the Swift SDK. For what LLM Insights are, the request/response model, required metrics, and the privacy notice, see the [LLM Insights overview](../../docs/llm-insights/index.md). ## Enable the required metrics Insights summarize the buffered vitals, so breathing (the default set) and cardio must both be active. Assign both bundles to `requestedMetrics`: ```swift sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics ``` `cardioMetrics` includes `.arterialPressureTrace`, which drives the on-screen pulse waveform. When `requestedMetrics` is left unset the SDK measures breathing only. ## Receive responses `SmartSpectraSDK` is `@Observable` (`@MainActor`). The latest insight is exposed as an observable property, updated on the main actor: ```swift public internal(set) var insight: Insight? ``` Observe it like any other observable state — read it in a SwiftUI `body`, or react to changes: ```swift .onChange(of: sdk.insight) { _, insight in guard let insight else { return } switch insight.result { case .analysis(let text): // the LLM text show(text) case .error: // insight.error holds the message show("The insights service is currently unavailable.") case .none: break } } ``` The same property delivers **both** the auto-fired periodic vitals insights and on-demand responses. Match `insight.requestID` against the value returned by `requestInsight` to tell them apart (auto-fired vitals won't match a request you made). Every insight is currently delivered with `type == .vitals`, so correlate on `requestID`, not `type`. ## Request an insight Call `requestInsight` on a running session (it throws otherwise). It returns the request ID used to correlate the asynchronous response: ```swift @discardableResult public func requestInsight(_ text: String) throws -> Int32 ``` ```swift do { pendingRequestId = try sdk.requestInsight("Summarize my current vital signs and flag anything unusual.") } catch { // e.g. processing not active } ``` The prompt is combined with the latest buffered metrics when they exist, otherwise sent prompt-only. ## Reading the Insight `Insight` is a generated SwiftProtobuf struct. There are **no** `hasAnalysis` / `hasError` accessors — switch on the `result` oneof: - `insight.result` → `.analysis(String)` on success, `.error(String)` on failure, or `nil`. (The convenience `insight.analysis` / `insight.error` properties return `""` for the case that isn't set.) - `insight.requestID` (`Int32`, note the capital `ID`) correlates the reply. - `insight.type` is currently always `.vitals` (the `.speech` and `.combined` cases are reserved and not emitted today). Full field documentation is in [Data Types → Insight](../../docs/data-types.md#insight). The first auto-fired insight arrives once pulse rate has a stable reading, typically about 15 seconds after the session starts; further snapshots follow as breathing rate and HRV stabilise, then every 60 seconds. Allow at least 15 seconds of valid measurement before an on-demand request can be grounded in the user's physiology. ## See also - [LLM Insights overview](../../docs/llm-insights/index.md) - [Swift API reference](api-reference.md) - [Data Types](../../docs/data-types.md) ## Configuring Metrics on Swift Source: https://presagetech.com/docs/smartspectra-sdk/swift/metrics --- title: Configuring Metrics on Swift description: Request pulse and breathing rates, HRV and EDA metrics from the SmartSpectra Swift SDK, and read published results. sidebarTitle: Configuring Metrics --- # Configuring iOS Metrics By default, Swift SDK measurements request the breathing metric set. Add pulse rate when your app needs a basic cardio value. ## Breathing and Pulse ### Request Metrics Request the default breathing metrics plus `.pulseRate` before calling `start()`: ```swift import SmartSpectra let sdk = SmartSpectraSDK.shared sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + [.pulseRate] ``` ### Read Metrics Read the latest breathing and pulse samples from `SmartSpectraSDK.metrics`: ```swift private let sdk = SmartSpectraSDK.shared if let metrics = sdk.metrics { let breathingRate = metrics.breathing.rate.last?.value let chestTrace = metrics.breathing.upperTrace.last?.value let abdomenTrace = metrics.breathing.lowerTrace.last?.value let pulseRate = metrics.cardio.pulseRate.last?.value } ``` Set `requestedMetrics = nil` to return to the default breathing-only set. Cardio fields are empty unless you request a cardio metric such as `.pulseRate`. ## Metric Update Patterns `SmartSpectraSDK.metrics` exposes the latest SDK metrics payload. Each payload contains the samples that became available since the previous metrics update; it is not guaranteed to contain every requested field every time. | Metric category | Examples | Expected cadence | Empty/null behavior | | --- | --- | --- | --- | | Peak/event-driven rate metrics | `metrics.breathing.rate`, `metrics.cardio.pulseRate`, `metrics.cardio.hrv` | Updated when a new physiological event, cycle, or analysis window produces a value | Arrays may be empty between valid updates during active capture | | Frame-driven metrics | `metrics.face.expression`, `metrics.face.landmarks`, `metrics.face.blinking`, `metrics.face.talking`, breathing traces | Updated near device frame cadence, with SDK callbacks rate-limited to about 30 Hz | Usually present more continuously when the metric is enabled and the input signal is valid | For example, `sdk.metrics?.cardio.pulseRate.last?.value` and `sdk.metrics?.breathing.rate.last?.value` may temporarily evaluate to `nil` between valid updates. This is expected and does not mean capture stopped or the metric was disabled. By contrast, face expression samples are frame-driven, so `sdk.metrics?.face.expression.last` can appear continuously while face metrics are enabled and the face signal is valid. Recommended UI handling: - Keep the last valid rate sample in app state and update it only when the array contains a new sample. - Show an initial loading or placeholder state until the first valid sample arrives. - Do not overwrite a displayed pulse rate or breathing rate with `nil` only because one metrics payload has no new sample. - Clear retained values when a capture session starts, stops, or when your app intentionally changes the requested metric set. - Prefer sample timestamps, and `stable` when present, to decide whether a retained value is fresh enough for your UI. ```swift private var lastPulseRate: Double? func pulseRateText(from metrics: Metrics?) -> String { if let pulse = metrics?.cardio.pulseRate.last(where: { $0.timestamp > 0 })?.value { lastPulseRate = pulse return "\(Int(pulse.rounded())) bpm" } guard let lastPulseRate else { return "-- bpm" } return "\(Int(lastPulseRate.rounded())) bpm" } ``` ## Advanced Request additional metrics only when your app needs them: ```swift sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + [ .pulseRate, .arterialPressureTrace, .hrv, .edaTrace, .faceLandmarks, .blinking, .talking, .expressions, ] ``` Read the advanced fields from the same metrics object: ```swift if let metrics = sdk.metrics { let pressureTrace = metrics.cardio.arterialPressureTrace.last?.value let hrvRmssd = metrics.cardio.hrv.last?.rmssd let edaTrace = metrics.eda.trace.last?.value let faceLandmarks = metrics.face.landmarks.last?.value let blinking = metrics.face.blinking.last?.detected let talking = metrics.face.talking.last?.detected let expression = metrics.face.expression.last } ``` ### Advanced Payload Types The Swift SDK uses the generated Swift protobuf types. Requested advanced metrics populate these fields: ```swift Metrics { breathing: Breathing eda: Eda face: Face cardio: Cardio } Cardio { pulseRate: [MeasurementWithConfidence] arterialPressureTrace: [MeasurementWithConfidence] hrv: [Hrv] } Hrv { rmssd: Double meanNn: Double sdnn: Double baevsky: Double timestamp: Int64 confidence: Float stable: Bool } Eda { trace: [Measurement] } Face { landmarks: [Landmarks] blinking: [DetectionStatus] talking: [DetectionStatus] expression: [Expression] } ``` EDA Proxy may take longer to produce its first sample than breathing or cardio outputs. See [Data Types](../../docs/data-types.md) for the complete protobuf schema. ## Swift Migration Guide Source: https://presagetech.com/docs/smartspectra-sdk/swift/migration-guide --- title: Swift Migration Guide description: "Migrate the SmartSpectra Swift SDK to typed camera selection, public custom video input, specific usage errors, and other release-by-release API changes." sidebarTitle: Migration Guide --- # SmartSpectra Swift SDK Migration Guide > Applies to SmartSpectra Swift SDK v3.x. > Migrating from a v3.0 release-candidate prior to rc.13, or from v2.x. ## Swift SDK v3.4.0 Migration ### Custom video input Camera capture remains the default. For camera selection changes, see below. Use `try sdk.useCustomInput()` to obtain a public frame-submission handle for your own camera or decoder. Await `sdk.start()` before submitting `CVPixelBuffer` or `CMSampleBuffer` frames. See the [custom camera example](headless-mode.md#use-your-own-camera-or-video-source). `try await sdk.reset()` stops processing and clears measurement output, retaining configuration and the custom-input handle. `try sdk.useCamera(.default)` switches back to SDK capture while stopped. Replacing an input invalidates older handles. The testing SPI `setVideoInputEnabled(_:)` now selects a source only while stopped. Calls during an active session leave the source unchanged and publish `.invalidState` on `sdk.error`. Move these calls after `stop()` or `reset()`. File playback remains testing-only; caller-owned frame input is public. The demo app's Video Testing tab now decodes clips in the app and calls `CustomInput.sendFrame` directly, with no testing SPI import. Optional timestamp sidecars still contain one integer millisecond value per decoded frame; the app converts these to microseconds. Decoding uses the formats supported by AVFoundation on the selected device or simulator. ### Camera selection Camera selection is opt-in. Existing applications retain the front-camera default and the behavior of `sdk.config.cameraPosition`. That property and the original `try sdk.useCamera()` method remain available, with deprecation warnings. To adopt discovery and strict selection, call `try sdk.useCamera(selection)` while stopped: ```swift // Discovery needs no SDK instance, authentication, or permission prompt. let cameras = try SmartSpectraSDK.availableCameras() let sdk = SmartSpectraSDK.shared try sdk.useCamera(.default) try sdk.useCamera(.front) try sdk.useCamera(.back) if let camera = cameras.first { try sdk.useCamera(.byId(camera.id)) } ``` Discovery returns `CameraInfo(id: name: facing: lensType:)`. IDs are opaque and device-local; names are optional. Facing is `.front`, `.back`, or `.unknown`, including external cameras whose direction is unknown. Visibility depends on platform permissions; discovery does not reserve a camera or guarantee a later capture will succeed. Discovery failures throw `SmartSpectraError` with `.inputUnavailable`. `lensType` reports `.wideAngle`, `.ultraWide`, `.telephoto`, or `.unknown`. Classification is best effort; missing metadata and cameras combining multiple lenses report `.unknown`. It does not describe digital zoom or change selection. Use it for picker labels and continue selecting by `id`. Explicit `.default` prefers front, then uses the first discovered camera. Explicit `.front`, `.back`, and `.byId` requests never fall back. Selection is resolved when capture starts; an unavailable camera fails `start()` with `.inputUnavailable`. `useCamera` throws `.configurationFailed` for an empty ID and `.invalidState` during startup, processing, stopping, or reset. A rejected request preserves the previous source. Stop/start/reset retain the selection until another source is selected. Once a typed selection is made, changes to the deprecated `cameraPosition` property do not override it. Calling the deprecated `useCamera()` returns to that property's selection behavior. To change cameras, await `stop()`, select, and start again: ```swift try await sdk.stop() try sdk.useCamera(.back) try await sdk.start() ``` Choose the input from an explicit app action or initial setup while stopped. SwiftUI view initializers can run repeatedly, so avoid selecting an input there. The SwiftUI demo's checkup, headless, and capture screens and the UIKit sample provide **Select Camera** controls. The picker lists discovered cameras by name, facing, and opaque ID, with **Default** and **Refresh** actions. Selection is disabled during startup, processing, and stopping. After a failed Start, opening the picker awaits `reset()` before allowing a new selection; this also recovers failures that occurred before a processing session existed. Navigating between screens does not change the selected camera. The iOS Simulator may return no cameras. The picker displays an empty-state message and still offers Default and Refresh; selecting an actual camera ID requires a device with discoverable cameras. ### Usage failures now report specific errors Usage entitlement checking continues throughout a measurement as before. What changes in v3.4.0 is how a terminal usage failure is reported. Observe `sdk.error` and handle: - `.authenticationFailed` for rejected credentials - `.creditExhausted` for an explicit quota denial - `.networkError` when entitlement cannot be refreshed in time The SDK also includes short measurements and the final partial interval before `stop()` in usage reporting. No public API changes are required. ## Swift SDK v3.3.1 Migration ### Stricter `PresageService-Info.plist` validation The SDK now validates the OAuth plist once, when the SDK loads, and surfaces problems as a non-retryable `configurationFailed` error instead of a generic auth-readiness failure at `start()`. What still works without changes: - No `PresageService-Info.plist` → API-key authentication, as before. - `IS_OAUTH_ENABLED` set to `false` (or legacy integer `0`) → API-key authentication, as before. - An older plist revision **missing** `IS_OAUTH_ENABLED` → treated as OAuth-off (API-key authentication), with a log message suggesting a re-download from the portal. - An OAuth plist **missing** `BUNDLE_ID` (older revisions) → the local bundle-identifier pre-check is skipped; the server still verifies your app's bundle identifier during authentication. What now fails fast (previously fell back silently or failed later with a generic error): - `IS_OAUTH_ENABLED` present with a non-boolean value (e.g. the string `"true"`) → `configurationFailed`; the API key, if any, is ignored. - OAuth enabled with a missing, blank, or wrong-typed `CLIENT_ID` or `SUB` → `configurationFailed`. - `BUNDLE_ID` present with a blank or non-string value, or not matching the running app target's Bundle Identifier → `configurationFailed`. If you hit any of these after upgrading, re-download the current `PresageService-Info.plist` for your app from [physiology.presagetech.com](https://physiology.presagetech.com) — error messages name the offending plist key but never echo its value. ## Swift SDK v3.3.0 Migration ### Default log verbosity is now warnings and errors only The SDK previously emitted informational log chatter by default. From v3.3.0 the default level is `SmartSpectraLogLevel.warning`. If you relied on the informational output, restore it via `SmartSpectraConfig.logLevel`: ```swift SmartSpectraSDK.shared.config.logLevel = .info ``` ## Protobuf Type Renames The Swift SDK's protobuf-generated types previously carried a `Presage_Physiology_` prefix derived from the proto package. The prefix has been stripped at the protoc-gen-swift level (`option swift_prefix = "";`), so all proto types are now exposed under their bare names. ### Quick reference | Before | After | | --- | --- | | `Presage_Physiology_Metrics` | `Metrics` | | `Presage_Physiology_Insight` | `Insight` | | `Presage_Physiology_InsightType` | `InsightType` | | `Presage_Physiology_FeatureType` | `FeatureType` | | `Presage_Physiology_MetricType` | `MetricType` | | `Presage_Physiology_Measurement` | `Measurement` | | `Presage_Physiology_MeasurementWithConfidence` | `MeasurementWithConfidence` | | `Presage_Physiology_DetectionStatus` | `DetectionStatus` | | `Presage_Physiology_ExpressionType` | `ExpressionType` | | `Presage_Physiology_StatusValue` | `StatusValue` | | `Presage_Physiology_StatusCode` | `StatusCode` | The same rule applies to every other type the proto schema exposes (`Pulse`, `Breathing`, `Trace`, `Strict`, `Face`, `Landmarks`, `Point2dFloat`, …). ### What to change Replace any reference to a `Presage_Physiology_*` symbol with the bare name. A repo-wide search-and-replace is sufficient: ```sh sed -i '' 's/Presage_Physiology_//g' ``` ### Name collisions Bare names can collide with types in modules the consumer also imports. Known collisions today: - `Measurement` collides with `Foundation.Measurement`. - `Trace` collides with `os.Trace` (OSLog signpost APIs). In files that import the colliding module, qualify the SmartSpectra type at the use site: ```swift import Foundation import SmartSpectra var breathingTrace: [SmartSpectra.Measurement] = [] let pulseTrace: SmartSpectra.Trace = ... ``` Everything else (`MeasurementWithConfidence`, `Pulse`, `Insight`, `Strict`, …) doesn't collide with anything in the standard Apple modules today — leave those bare. Wire format is unchanged (`swift_prefix` only affects Swift codegen). ## Package Rename The Swift SDK module and SPM product were renamed from `SmartSpectraSwiftSDK` to `SmartSpectra`. Update every import and the shared SDK type: ```swift // Before: import SmartSpectraSwiftSDK let sdk = SmartSpectraSwiftSDK.shared // After: import SmartSpectra let sdk = SmartSpectraSDK.shared ``` Most call sites should move from `SmartSpectraSwiftSDK.shared` to `SmartSpectraSDK.shared`. ## Edge Metrics Migration The `metricsBuffer` pathway has been removed. Swift apps should now read vitals data from `metrics` on `SmartSpectraSDK.shared`. ### What Changed - `metricsBuffer` and `$metricsBuffer` were removed - on-device `metrics` is now the vitals data source - public configuration now lives on `sdk.config` ### Field Mappings | Old (`metricsBuffer`) | New (`metrics`) | | --------------------- | --------------- | | `pulse.rate` | `cardio.pulseRate` | | `breathing.rate` | `breathing.rate` | | `pulse.trace` | `cardio.arterialPressureTrace` | | `breathing.upperTrace` | `breathing.upperTrace` | ### Important Cardio fields now require explicit opt-in through requested metrics configuration. ```swift import SmartSpectra let sdk = SmartSpectraSDK.shared sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics ``` **Removed:** - `sdk.metricsBuffer` - `sdk.$metricsBuffer` - `MetricsBuffer` **Replace with:** ```swift // Before: import SmartSpectraSwiftSDK let sdk = SmartSpectraSwiftSDK.shared sdk.$metricsBuffer.sink { buffer in // read buffer.pulse.rate, buffer.breathing.rate, ... } ``` ```swift // After: import SmartSpectra let sdk = SmartSpectraSDK.shared if let metrics = sdk.metrics { // read metrics.cardio.pulseRate, metrics.breathing.rate, ... } ``` ## Public Configuration Surface - authentication should be set through `sdk.config.apiKey` - metric selection should be set through `sdk.config.requestedMetrics` - camera selection should be set through `try sdk.useCamera(selection)` while stopped - preview frame publishing should be controlled through `sdk.config.imageOutputEnabled` ### Removed - `sdk.setApiKey("...")` - `sdk.setCameraPosition(...)` - `sdk.setImageOutputEnabled(...)` ### Replace With | Old | New | | --- | --- | | `sdk.setApiKey("...")` | `sdk.config.apiKey = "..."` | | `sdk.setCameraPosition(.front)` | `try sdk.useCamera(.front)` | | `sdk.setImageOutputEnabled(true)` | `sdk.config.imageOutputEnabled = true` | | no public metric-selection API | `sdk.config.requestedMetrics = [...]` | ```swift // Before: let sdk = SmartSpectraSwiftSDK.shared sdk.setApiKey("YOUR_API_KEY") sdk.setCameraPosition(.front) sdk.setImageOutputEnabled(true) ``` ```swift // After: let sdk = SmartSpectraSDK.shared sdk.config.apiKey = "YOUR_API_KEY" sdk.config.requestedMetrics = [.breathingRate, .pulseRate, .faceLandmarks] try sdk.useCamera(.front) sdk.config.imageOutputEnabled = true ``` ## Headless Lifecycle Migration ```swift // Before: let vitalsProcessor = SmartSpectraVitalsProcessor.shared vitalsProcessor.startProcessing { error in if let error { print(error.localizedDescription) } } vitalsProcessor.stopProcessing() ``` ```swift // After: let sdk = SmartSpectraSDK.shared try await sdk.start() try await sdk.stop() ``` ## Observable State Consolidation | Old | New | | --- | --- | | `sdk.metricsBuffer` | `sdk.metrics` | | `vitalsProcessor.processingStatus` | `sdk.processingStatus` | | `vitalsProcessor.lastStatusCode` + `vitalsProcessor.statusHint` | `sdk.validationStatus?.code` + `sdk.validationStatus?.hint` | | `sdk.resultErrorText` | `sdk.error?.message` | | `vitalsProcessor.imageOutput` | `sdk.imageOutput` | ## Validation Status ```swift // Before: Text(vitalsProcessor.statusHint) ``` ```swift // After: if let validationStatus = sdk.validationStatus { showBanner(validationStatus.hint) switch validationStatus.code { case .ok: break case .noFaceFound, .multipleFacesFound, .faceNotCentered, .tooDark, .tooBright, .chestNotVisible, .cameraTuning, .frameRateTooLow, .excessiveMotion, .faceTooClose, .faceTooFar, .faceTooHigh, .faceTooLow, .faceNotForward: break default: // .faceSizeOutOfRange is deprecated in favour of // .faceTooClose / .faceTooFar, and new codes may be added. break } } ``` ## Error Model ```swift // Before: if !sdk.resultErrorText.isEmpty { let message = sdk.resultErrorText showError(message) } ``` ```swift // After: if let error = sdk.error { showError(error.message) if error.retryable { showRetryAction() } } ``` ## Processing Status Migration ### Type Rename ```swift // Before: let status: PresageProcessingStatus = vitalsProcessor.processingStatus // After: let status: ProcessingStatus = sdk.processingStatus ``` ### Lifecycle Cases - `idle` - `starting` - `running` - `stopping` - `error` ### Case Mapping | Previous case | New case | | ------------- | -------- | | `idle` | `idle` | | `starting` | `starting` | | `processing` | `running` | | `processed` | `idle` | | `error` | `error` | ### Example Update ```swift // Before: if vitalsProcessor.processingStatus == .idle || vitalsProcessor.processingStatus == .processed { showResults() } ``` ```swift // After: if sdk.processingStatus == .idle { showResults() } ``` ## Observable Migration `SmartSpectraSDK` and `SmartSpectraConfig` moved from `ObservableObject` + `@Published` to the Swift 5.9 `@Observation` macro. The Combine-style `$` publishers (`sdk.$metrics`, `sdk.$error`, `sdk.$processingStatus`, `sdk.$imageOutput`, `sdk.$validationStatus`, `sdk.$insight`) are gone. The properties themselves remain and now track automatically in SwiftUI views. ### SwiftUI Views Drop `@ObservedObject` / `@StateObject` wrappers on `sdk`. SwiftUI re-renders when tracked properties change: ```swift // Before: struct MyView: View { @ObservedObject private var sdk = SmartSpectraSwiftSDK.shared var body: some View { Text("Status: \(sdk.processingStatus)") } } ``` ```swift // After: struct MyView: View { private let sdk = SmartSpectraSDK.shared var body: some View { Text("Status: \(sdk.processingStatus)") } } ``` For side effects on property change, use `.onChange(of:)` instead of `.onReceive($X)` or manual `.sink { }`: ```swift // Before: .onReceive(sdk.$metricsBuffer) { buffer in guard let buffer else { return } appendToChart(buffer.breathing.rate) } ``` ```swift // After: .onChange(of: sdk.metrics) { _, metrics in guard let metrics else { return } appendToChart(metrics.breathing.rate) } ``` ### UIKit Consumers UIKit has no `.onChange(of:)` equivalent. Replace Combine subscriptions with `withObservationTracking`, re-armed after each change: ```swift // Before: sdk.$metricsBuffer .receive(on: DispatchQueue.main) .sink { [weak self] buffer in self?.update(buffer) } .store(in: &cancellables) ``` ```swift // After: private func observeMetrics() { withObservationTracking { _ = sdk.metrics } onChange: { [weak self] in Task { @MainActor [weak self] in guard let self else { return } self.update(self.sdk.metrics) self.observeMetrics() } } } ``` A generic helper keeps multiple keypath observations tidy: ```swift private func observeSDK( _ keyPath: KeyPath, _ handler: @escaping (T) -> Void ) { withObservationTracking { _ = sdk[keyPath: keyPath] } onChange: { [weak self] in Task { @MainActor [weak self] in guard let self else { return } handler(self.sdk[keyPath: keyPath]) self.observeSDK(keyPath, handler) } } handler(sdk[keyPath: keyPath]) } // Usage: observeSDK(\.metrics) { [weak self] metrics in guard let self, let metrics else { return } self.update(metrics) } observeSDK(\.processingStatus) { [weak self] status in self?.updateStatus(status) } ``` ### Non-SwiftUI `@Observable` Classes View models that consume SDK state should mark themselves `@MainActor @Observable`. Combine `.sink` on `sdk.$X` inside those view models migrates to the same `withObservationTracking` re-arm pattern shown above. ## Configuration Access Configuration is now reached through the SDK instance only: ```swift // Before: let sdk = SmartSpectraSwiftSDK.shared sdk.setApiKey("YOUR_API_KEY") sdk.setCameraPosition(.front) ``` ```swift // After: let sdk = SmartSpectraSDK.shared sdk.config.apiKey = "YOUR_API_KEY" sdk.config.requestedMetrics = SmartSpectraConfig.cardioMetrics + SmartSpectraConfig.breathingMetrics ``` Older releases did not expose public `SmartSpectraConfig` access; configuration was applied through methods on `SmartSpectraSwiftSDK.shared`. Current releases keep SDK settings on `sdk.config`, so views observe the same configuration used by the SDK. Choose the input separately through `useCamera` or `useCustomInput`. ### Custom SDK Instances `SmartSpectraSDK` and `SmartSpectraConfig` now expose public initializers for callers that want an isolated SDK instance (for tests or advanced integrations): ```swift let customConfig = SmartSpectraConfig() customConfig.apiKey = "…" let sdk = SmartSpectraSDK(config: customConfig) ``` Most apps should keep using `SmartSpectraSDK.shared` — it's the intended entry point, matching the `URLSession.shared` pattern. #### What multi-instance gives you today **Isolated per instance:** - `@Observable` state (`metrics`, `error`, `processingStatus`, `validationStatus`, `insight`, `imageOutput`) - `config` — each SDK has its own - SwiftUI views bound via `init(sdk:)` or `.smartSpectraSDK(_:)` render against the correct instance **Still process-global (not isolated):** - Authentication — the underlying auth handler is a singleton, so setting `apiKey` on one instance affects the auth state that every instance sees - Camera — only one `AVCaptureSession` can be active at a time on iOS, and it is owned process-wide - The preprocessing runtime — only one instance can drive an *active measurement* at a time In practice this means custom instances are useful for **tests** (isolated state per test), **side-by-side UI** (show state from two SDKs without either actively processing), or **sequential lifecycles** (stop one SDK, start another). Two simultaneous live measurements on two SDK instances is not supported yet. #### Binding a custom instance into a SwiftUI hierarchy The SDK no longer ships SwiftUI views or environment-binding helpers. Custom instances flow into your views the same way any `@Observable` does — pass them directly, or define your own environment key: ```swift @main struct MyApp: App { @State private var sdk = SmartSpectraSDK(config: customConfig) var body: some Scene { WindowGroup { ContentView(sdk: sdk) } } } ``` For an environment-key pattern, see `samples/demo-app/UI/SDKEnvironmentKey.swift` — a tiny helper that defines `\.smartSpectraSDK` and `.smartSpectraSDK(_:)` for the moved screening views to read against. Copy it if you want the same pattern in your own app. Hosts that just use `SmartSpectraSDK.shared` need no binding at all. ## SwiftUI Surface Removal The SDK no longer ships any SwiftUI views or environment helpers. Removed from the `SmartSpectra` module: - `SmartSpectraView`, `SmartSpectraButtonView`, `SmartSpectraResultView` - The screening overlay / plot / processing views - Onboarding, tutorial, legal, and web views - `ContinuousVitalsPlotView` and its `TraceLineView` / `VitalSection` helpers - `StartupRecovery` helper - `\.smartSpectraSDK` environment key and `.smartSpectraSDK(_:)` view modifier ### What's still public - ``SmartSpectraSDK`` and ``SmartSpectraConfig`` — the data and lifecycle plane. - The full observable surface: `metrics`, `imageOutput`, `processingStatus`, `validationStatus`, `error`, `insight`, plus `try await sdk.start() / sdk.stop()`. - All proto types (`Metrics`, `Measurement`, `MeasurementWithConfidence`, etc.) and their `TimeStamped` / `appendProtoArray` extensions. ### What you need to do If you used `SmartSpectraView()` as a one-line integration point, copy the reference implementation from the demo-app sample at [`samples/demo-app/UI/`](https://github.com/Presage-Security/SmartSpectra/tree/main/swift/samples/demo-app/UI) into your project. The folder mirrors the previous SDK layout (`Components/`, `Screening/`, `Legal/`, `Web/`) and wires through public-only SDK API: - `ScreeningViewModel` calls `try await sdk.start()` / `sdk.stop()` instead of the (removed) `processor.startProcessing/stopProcessing`. - `SDKExtensions.swift` recomputes `cardioMeasurementsEnabled` / `facialExpressionEnabled` / `edaInferenceEnabled` from the public `requestedMetrics` surface. - `SDKEnvironmentKey.swift` defines a sample-local `\.smartSpectraSDK` environment key + `.smartSpectraSDK(_:)` modifier — copy it if you want the same SwiftUI binding pattern. - `StartupRecovery` takes an explicit `videoInputEnabled` argument; the host tracks it locally rather than reading SDK config. - Brand color lives in `BrandColor.swift` — customize for your product theme. - Tutorial images load from the host app's main bundle (no `bundle: .module`); the demo-app's `Assets.xcassets/tutorial_image*.imageset` entries can be copied as-is. If you only used ``ContinuousVitalsPlotView``, the same folder includes a sample-local copy at `samples/demo-app/UI/Screening/ContinuousVitalsPlotView.swift` (with `TraceLineView.swift` and `VitalSection.swift`). Drop the three files into your project, add the `SDKExtensions.swift` derived flags, and the view works unchanged. ```swift // Before: import SwiftUI import SmartSpectra struct ContentView: View { private let sdk = SmartSpectraSDK.shared init() { sdk.config.apiKey = "…" } var body: some View { SmartSpectraView() } } ``` ```swift // After (option 1 — keep the screening flow by copying the sample): import SwiftUI import SmartSpectra struct ContentView: View { private let sdk = SmartSpectraSDK.shared init() { sdk.config.apiKey = "…" } var body: some View { // SmartSpectraView is now a sample-local view copied from // samples/demo-app/UI/Components/SmartSpectraView.swift. SmartSpectraView() } } ``` ```swift // After (option 2 — bring your own UI, drive the SDK directly): import SwiftUI import SmartSpectra struct ContentView: View { private let sdk = SmartSpectraSDK.shared init() { sdk.config.apiKey = "…" } var body: some View { VStack { if let image = sdk.imageOutput { Image(uiImage: image).resizable().aspectRatio(contentMode: .fit) } Text("Status: \(String(describing: sdk.processingStatus))") Button(sdk.processingStatus == .running ? "Stop" : "Start") { Task { if sdk.processingStatus == .running { try? await sdk.stop() } else { try? await sdk.start() } } } } } } ``` ### Why this changed The shipped UI embedded opinionated decisions every customer wanted to override (full-screen vs sheet presentation, onboarding policy, legal-document hosting URL, theming, plot styling). Maintaining that as stable SDK API forced every customer to fight the same set of defaults. Moving every SwiftUI piece into a sample lets each project fork the part it cares about while the SDK owns only the data and lifecycle plane. ## `@MainActor` Isolation `SmartSpectraSDK` and `SmartSpectraConfig` are now `@MainActor`-isolated. Access from non-main contexts requires the standard hop: ```swift // From a background task: await MainActor.run { sdk.config.apiKey = "…" } ``` SwiftUI `View` bodies, UIKit `UIViewController` methods, and `XCTestCase`-subclass methods marked `@MainActor` access the SDK directly without extra ceremony. Unit tests that mutate SDK or config state should annotate the test class with `@MainActor`. ## Metric bundles moved to `SmartSpectraConfig` Older Swift releases did not expose public requested-metric bundles. Current releases expose public `nonisolated static let` bundles on `SmartSpectraConfig`, including `breathingMetrics`, `cardioMetrics`, `faceMetrics`, and `edaMetrics`. The namespace and `Metrics` suffix match the Android SDK's `SmartSpectraConfig.breathingMetrics` companion field and the C++ SDK's `SmartSpectraConfig::CardioMetrics()` static method. ```swift // Before // No public requested-metric selection API. // After sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics ``` Current releases also expose an EDA Proxy bundle: ```swift sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.edaMetrics ``` ## Option 1: API Key on Swift Source: https://presagetech.com/docs/smartspectra-sdk/swift/option-1-api-key --- title: "Option 1: API Key on Swift" description: "The fastest SmartSpectra Swift setup: add the package, drop in an API key, and run a first pulse and breathing measurement on device." sidebarTitle: "Option 1: API Key" --- # QuickStart - API Key Use this if you want the fastest manual path. ## What you will change manually You will touch exactly these things: 1. The app target package dependencies 2. The app target camera permission 3. `Cool Vitals/ContentView.swift` You do not need to create any new Swift files. ## Result you should get At the end, the app should show: - `Status` and `Validation` at the top - live camera preview - pulse rate, breathing rate, HRV RMSSD, and expression cards - white labels for those four cards - confidence-colored pulse and breath-rate values - one large arterial pressure waveform - chest and abdomen breathing waveforms - guidance text below the breathing waveforms - one portrait screen with no scrolling ![SmartSpectra iOS quickstart demo](ios-quickstart.gif) ## Register for your free API Key ### Create an Account 1. Navigate to the Presage [Developer Admin Portal Registration](https://physiology.presagetech.com/auth/register) 2. Click **Register** and fill in your email, password, and other required fields. 3. Check your email for a confirmation link and follow it to activate your account. ### Log In 1. Go to the Presage Developer Admin Portal [Login](https://physiology.presagetech.com/auth/login) 2. Enter your email and password, then click **Submit**. 3. After successful login you will be redirected to your Portal page, where you can manage your API key. ## Step 1 — Create the project In Xcode, create a new iOS app project: 1. Select `File` → `New` → `Project...` 2. Choose `iOS` → `App` 3. Set `Product Name` to `Cool Vitals` 4. Set `Interface` to `SwiftUI` 5. Set `Language` to `Swift` 6. Save the project If you already created the project, open it instead. In Finder, open: - `Cool Vitals/Cool Vitals.xcodeproj` Then select the app target in Xcode. ## Step 2 — Add the SmartSpectra package In Xcode: 1. Click `File` → `Add Package Dependencies...` 2. Paste `https://github.com/Presage-Security/SmartSpectra-Swift/` 3. For repeatable builds, choose `Exact Version` and enter a released tag such as `3.4.0`, from the [SmartSpectra-Swift releases](https://github.com/Presage-Security/SmartSpectra-Swift/releases) 4. Use `Branch` → `main` only when testing the latest final public release before pinning a version 5. Add the package to the `Cool Vitals` app target Manual check: - In the project navigator, you should now see `Package Dependencies` - `SmartSpectra` should be attached to the app target ## Step 3 — Add camera permission In Xcode: 1. Select the `Cool Vitals` target 2. Open the `Info` tab 3. Add a new key named `Privacy - Camera Usage Description` **NOTE** `Ctrl + Click` on the `Custom iOS Target Properties` and click `Add Row` 4. Set the value to `This app needs camera access to measure vitals.` ![FindInfo](FindInfo.png) Manual check: - The app target now has a camera usage description ## Step 4 — Replace `ContentView.swift` In Xcode: 1. Open `Cool Vitals/ContentView.swift` 2. Delete everything in the file 3. Paste the full file below 4. Replace `YOUR_API_KEY` with your real API key **NOTE:** [Log in](https://physiology.presagetech.com/auth/login) or [register](https://physiology.presagetech.com/auth/register) at the Presage developer portal for your API key. Paste this entire file: ```swift import SwiftUI import SmartSpectra import AVFoundation struct ContentView: View { private enum TraceWindow { static let rate = 120 static let arterialWaveform = 240 static let breathingWaveform = 180 } private let sdk = SmartSpectraSDK.shared @State private var didAutoStart = false @State private var pulseRateBuffer: [MeasurementWithConfidence] = [] @State private var breathingRateBuffer: [MeasurementWithConfidence] = [] @State private var arterialPressureBuffer: [MeasurementWithConfidence] = [] @State private var chestBuffer: [SmartSpectra.Measurement] = [] @State private var abdomenBuffer: [SmartSpectra.Measurement] = [] @State private var latestHrv: Hrv? @State private var latestExpressionScores: [ExpressionScore] = [] init() { sdk.config.apiKey = "YOUR_API_KEY" // The default camera selection prefers front when available. sdk.config.imageOutputEnabled = true sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics + [ .expressions, ] } private enum WaveformProminence { case primary case secondary } private var metrics: Metrics? { sdk.metrics } private var metricsUpdateToken: Int64 { [ metrics?.cardio.pulseRate.last?.timestamp, metrics?.breathing.rate.last?.timestamp, metrics?.cardio.arterialPressureTrace.last?.timestamp, metrics?.breathing.upperTrace.last?.timestamp, metrics?.breathing.lowerTrace.last?.timestamp, metrics?.cardio.hrv.last?.timestamp, metrics?.face.expression.last?.timestamp, ] .compactMap { $0 } .max() ?? 0 } private var pulseRateText: String { formatMetric(pulseRateBuffer.last.map { Double($0.value) }, digits: 0, suffix: " bpm") } private var breathingRateText: String { formatMetric(breathingRateBuffer.last.map { Double($0.value) }, digits: 0, suffix: " bpm") } private var hrvText: String { guard let value = latestHrv?.rmssd, value > 0 else { return "--" } return formatMetric(value, digits: 1, suffix: " ms") } private var latestExpressionScore: ExpressionScore? { latestExpressionScores.max(by: { $0.confidence < $1.confidence }) } private var latestExpressionLabel: String { guard let score = latestExpressionScore else { return "--" } let name = String(expressionName(score.type).prefix(8)) let paddedName = name + String(repeating: " ", count: max(0, 8 - name.count)) let percent = confidenceText(score.confidence) let paddedPercent = String(repeating: " ", count: max(0, 4 - percent.count)) + percent return "\(paddedName) \(paddedPercent)" } private var pulseConfidenceColor: Color { confidenceColor(pulseRateBuffer.last?.confidence) } private var breathingConfidenceColor: Color { confidenceColor(breathingRateBuffer.last?.confidence) } private var arterialPressureSamples: [Double] { arterialPressureBuffer.map { Double($0.value) } } private var chestSamples: [Double] { chestBuffer.map { Double($0.value) } } private var abdomenSamples: [Double] { abdomenBuffer.map { Double($0.value) } } private var statusText: String { switch sdk.processingStatus { case .idle: return "Idle" case .starting: return "Starting" case .running: return "Running" case .stopping: return "Stopping" case .error: return "Error" @unknown default: return "Unknown" } } private var validationTitle: String { guard let validationStatus = sdk.validationStatus else { return "Waiting" } return validationStatus.hint } private var statusColor: Color { switch sdk.processingStatus { case .running: return .green case .starting, .stopping: return .orange case .error: return .red case .idle: return .gray @unknown default: return .gray } } private var validationColor: Color { guard let validationStatus = sdk.validationStatus else { return .gray } switch validationStatus.code { case .ok: return .green case .cameraTuning: return .orange default: return .yellow } } var body: some View { GeometryReader { geometry in let compact = geometry.size.height < 820 let horizontalPadding: CGFloat = compact ? 12 : 16 let topSpacing: CGFloat = compact ? 8 : 12 let previewHeight = min(max(geometry.size.height * 0.26, 190), 250) VStack(spacing: topSpacing) { statusBar(compact: compact) .zIndex(1) previewCard .frame(height: previewHeight) .zIndex(0) HStack(spacing: topSpacing) { metricCard( title: "Pulse Rate", value: pulseRateText, valueColor: pulseConfidenceColor, accent: .red, compact: compact ) metricCard( title: "Breathing Rate", value: breathingRateText, valueColor: breathingConfidenceColor, accent: .cyan, compact: compact ) } .frame(maxHeight: compact ? 82 : 92) HStack(spacing: topSpacing) { metricCard( title: "HRV RMSSD", value: hrvText, valueColor: .white, accent: .mint, compact: compact ) metricCard( title: "Expression", value: latestExpressionLabel, valueColor: .white, accent: .orange, compact: compact, monospacedValue: true ) } .frame(maxHeight: compact ? 82 : 92) waveformCard( title: "Arterial Pressure", samples: arterialPressureSamples, accent: .purple, compact: compact, prominence: .primary ) .frame(height: compact ? 154 : 182) HStack(spacing: topSpacing) { waveformCard( title: "Chest Waveform", samples: chestSamples, accent: .cyan, compact: compact, prominence: .secondary ) waveformCard( title: "Abdomen Waveform", samples: abdomenSamples, accent: .blue, compact: compact, prominence: .secondary ) } .frame(height: compact ? 130 : 146) guidanceText } .padding(.horizontal, horizontalPadding) .padding(.vertical, compact ? 10 : 14) .background(backgroundGradient.ignoresSafeArea()) } .task { await startIfNeeded() } .task(id: metricsUpdateToken) { mergeCurrentMetrics() } } // The SDK's `hint` is written for end users — surface it as-is. private var guidanceText: some View { Text(sdk.validationStatus?.hint ?? "Getting ready\u{2026}") .font(.footnote) .foregroundStyle(.white.opacity(0.85)) .multilineTextAlignment(.center) .frame(maxWidth: .infinity) .accessibilityLabel("Measurement guidance") } private var previewCard: some View { ZStack { if let image = sdk.imageOutput { Image(uiImage: image) .resizable() .aspectRatio(contentMode: .fill) .frame(maxWidth: .infinity, maxHeight: .infinity) .clipped() } else { LinearGradient( colors: [Color(red: 0.16, green: 0.24, blue: 0.46), Color.black], startPoint: .topLeading, endPoint: .bottomTrailing ) VStack(spacing: 10) { Image(systemName: "camera.viewfinder") .font(.system(size: 40, weight: .semibold)) Text("Camera preview will appear here") .font(.headline) } .foregroundStyle(.white.opacity(0.92)) } LinearGradient( colors: [.black.opacity(0.68), .black.opacity(0.12), .clear], startPoint: .bottom, endPoint: .top ) } .frame(maxWidth: .infinity, maxHeight: .infinity) .clipShape(RoundedRectangle(cornerRadius: 26, style: .continuous)) .overlay( RoundedRectangle(cornerRadius: 26, style: .continuous) .stroke(.white.opacity(0.12), lineWidth: 1) ) .shadow(color: .black.opacity(0.35), radius: 18, x: 0, y: 10) } private func statusBar(compact: Bool) -> some View { HStack(spacing: compact ? 8 : 10) { badge(title: "Status", value: statusText, color: statusColor) badge(title: "Validation", value: validationTitle, color: validationColor) Spacer(minLength: 8) Button(action: toggleMeasurement) { Text(sdk.processingStatus == .running ? "Stop" : "Start") .font(.caption.bold()) .padding(.horizontal, compact ? 14 : 18) .padding(.vertical, 10) .background(.white, in: Capsule()) .foregroundStyle(.black) } } } private func metricCard( title: String, value: String, valueColor: Color, accent: Color, compact: Bool, monospacedValue: Bool = false ) -> some View { VStack(alignment: .leading, spacing: compact ? 6 : 8) { HStack(spacing: 6) { Circle() .fill(accent) .frame(width: 8, height: 8) Text(title) .font(.caption.weight(.semibold)) .foregroundStyle(.white) } Text(value) .font(.system(size: compact ? 21 : 24, weight: .bold, design: monospacedValue ? .monospaced : .rounded)) .foregroundStyle(valueColor) .monospacedDigit() .lineLimit(1) .minimumScaleFactor(0.7) } .dashboardCard() } private func waveformCard( title: String, samples: [Double], accent: Color, compact: Bool, prominence: WaveformProminence ) -> some View { VStack(alignment: .leading, spacing: compact ? 6 : 8) { VStack(alignment: .leading, spacing: 2) { Text(title) .font(.caption.weight(.semibold)) .foregroundStyle(.white) } ZStack { RoundedRectangle(cornerRadius: 14, style: .continuous) .fill(accent.opacity(0.12)) if samples.count > 1 { WaveformView( samples: samples, strokeColor: accent, verticalPaddingFraction: prominence == .primary ? 0.14 : 0.08 ) .padding(prominence == .primary ? 8 : 10) } } .frame(maxHeight: .infinity) .overlay( RoundedRectangle(cornerRadius: 14, style: .continuous) .stroke(accent.opacity(0.3), lineWidth: 1) ) } .dashboardCard() } private func badge(title: String, value: String, color: Color) -> some View { HStack(spacing: 6) { Circle() .fill(color) .frame(width: 8, height: 8) Text("\(title): \(value)") .font(.caption.weight(.semibold)) } .padding(.horizontal, 10) .padding(.vertical, 8) .background(.white.opacity(0.12), in: Capsule()) .foregroundStyle(.white) } private func mergeCurrentMetrics() { guard let metrics else { return } if !metrics.cardio.pulseRate.isEmpty { pulseRateBuffer.appendProtoArray(contentsOf: metrics.cardio.pulseRate) pulseRateBuffer = Array(pulseRateBuffer.suffix(TraceWindow.rate)) } if !metrics.breathing.rate.isEmpty { breathingRateBuffer.appendProtoArray(contentsOf: metrics.breathing.rate) breathingRateBuffer = Array(breathingRateBuffer.suffix(TraceWindow.rate)) } if !metrics.cardio.arterialPressureTrace.isEmpty { arterialPressureBuffer.appendProtoArray(contentsOf: metrics.cardio.arterialPressureTrace) arterialPressureBuffer = Array(arterialPressureBuffer.suffix(TraceWindow.arterialWaveform)) } if !metrics.breathing.upperTrace.isEmpty { chestBuffer.appendProtoArray(contentsOf: metrics.breathing.upperTrace) chestBuffer = Array(chestBuffer.suffix(TraceWindow.breathingWaveform)) } if !metrics.breathing.lowerTrace.isEmpty { abdomenBuffer.appendProtoArray(contentsOf: metrics.breathing.lowerTrace) abdomenBuffer = Array(abdomenBuffer.suffix(TraceWindow.breathingWaveform)) } if let hrv = metrics.cardio.hrv.last { latestHrv = hrv } if let scores = metrics.face.expression.last?.scores, !scores.isEmpty { latestExpressionScores = scores } } private func resetBuffers() { pulseRateBuffer.removeAll(keepingCapacity: true) breathingRateBuffer.removeAll(keepingCapacity: true) arterialPressureBuffer.removeAll(keepingCapacity: true) chestBuffer.removeAll(keepingCapacity: true) abdomenBuffer.removeAll(keepingCapacity: true) latestHrv = nil latestExpressionScores.removeAll(keepingCapacity: true) } private func toggleMeasurement() { Task { if sdk.processingStatus == .running || sdk.processingStatus == .starting { try? await sdk.stop() } else { resetBuffers() try? await sdk.start() } } } private func startIfNeeded() async { guard !didAutoStart else { return } didAutoStart = true guard sdk.processingStatus == .idle else { return } resetBuffers() try? await sdk.start() } private func confidenceText(_ confidence: Float?) -> String { guard let confidence, confidence.isFinite else { return "--" } let percent = min(max(Double(confidence), 0), 100) return "\(Int(percent.rounded()))%" } private func confidenceColor(_ confidence: Float?) -> Color { guard let confidence, confidence.isFinite else { return .white.opacity(0.65) } let percent = min(max(Double(confidence), 0), 100) switch percent { case 85...: return .green case 60..<85: return .yellow default: return .red } } private func formatMetric(_ value: Double?, digits: Int = 0, suffix: String = "") -> String { guard let value else { return "--" } if digits == 0 { return "\(Int(value.rounded()))\(suffix)" } return String(format: "% .\(digits)f", value).replacingOccurrences(of: " ", with: "") + suffix } private func expressionName(_ type: ExpressionType) -> String { switch type { case .unspecified: return "Unspecified" case .angry: return "Angry" case .contempt: return "Contempt" case .disgust: return "Disgust" case .fear: return "Fear" case .happy: return "Happy" case .neutral: return "Neutral" case .sad: return "Sad" case .surprise: return "Surprise" case .UNRECOGNIZED(_): return "Unknown" @unknown default: return "Unknown" } } private var backgroundGradient: LinearGradient { LinearGradient( colors: [ Color(red: 0.03, green: 0.05, blue: 0.12), Color(red: 0.07, green: 0.09, blue: 0.18), Color.black, ], startPoint: .topLeading, endPoint: .bottomTrailing ) } } private struct WaveformView: View { let samples: [Double] let strokeColor: Color let verticalPaddingFraction: Double var body: some View { GeometryReader { geometry in Path { path in guard samples.count > 1 else { return } let minValue = samples.min() ?? 0 let maxValue = samples.max() ?? 1 let rawRange = max(maxValue - minValue, 0.0001) let padding = rawRange * verticalPaddingFraction let lowerBound = minValue - padding let upperBound = maxValue + padding let range = max(upperBound - lowerBound, 0.0001) for (index, sample) in samples.enumerated() { let x = geometry.size.width * CGFloat(index) / CGFloat(samples.count - 1) let normalized = (sample - lowerBound) / range let y = geometry.size.height * (1 - normalized) if index == 0 { path.move(to: CGPoint(x: x, y: y)) } else { path.addLine(to: CGPoint(x: x, y: y)) } } } .stroke(strokeColor, style: StrokeStyle(lineWidth: 2.2, lineCap: .round, lineJoin: .round)) } } } private extension View { func dashboardCard() -> some View { self .padding(12) .background( RoundedRectangle(cornerRadius: 20, style: .continuous) .fill(Color.white.opacity(0.08)) ) .overlay( RoundedRectangle(cornerRadius: 20, style: .continuous) .stroke(Color.white.opacity(0.08), lineWidth: 1) ) } } ``` ## Step 5 — Build and run on a phone In Xcode: 1. Choose a physical iPhone as the run destination 2. Build and run the app 3. Allow camera access when iOS asks 4. Wait a few seconds for camera tuning and signal stabilization Run this on a physical device — the simulator has no camera. (For automated tests, where frames come from a video file instead, the simulator is supported; see [Headless testing in CI](headless-testing-in-ci.md).) ## What success looks like When your program is running, you should see all of these: - `Status` and `Validation` chips are visible at the top - the preview is below the chips - the arterial pressure waveform is larger than the breathing waveforms - chest and abdomen waveforms both appear on screen - the guidance text is below those waveforms - the pulse and breath-rate numbers change color with confidence - expressions and HRV are reported ## Expected log note for API key mode This log is expected in API key mode and is not a failure: - `PresageService-Info.plist not found. OAuth authentication will be disabled. Using API key authentication instead.` ## Common manual mistakes If the screen does not match the target state, check these first: - the package was added to the wrong target - `ContentView.swift` was only partially replaced - `YOUR_API_KEY` was not replaced with a real key - the app is still running an older installed build on the phone - the app was run in the simulator instead of on a real device ## Option 2: OAuth on Swift Source: https://presagetech.com/docs/smartspectra-sdk/swift/option-2-oauth --- title: "Option 2: OAuth on Swift" description: "Set up SmartSpectra on iOS with OAuth instead of a hard-coded API key: register the app, add PresageService-Info.plist, and verify it works." sidebarTitle: "Option 2: OAuth" --- # QuickStart - OAuth Use this if you want to use SmartSpectra OAuth instead of the API key. ## What you will change manually You will touch exactly these things: 1. The app target package dependencies 2. The app target camera permission 3. The OAuth plist target membership 4. `Cool Vitals/ContentView.swift` You do not need to create any new Swift files. ## Result you should get At the end, the app should show: - `Status` and `Validation` at the top - live camera preview - pulse rate, breathing rate, HRV RMSSD, and expression cards - white labels for those four cards - confidence-colored pulse and breath-rate values - one large arterial pressure waveform - chest and abdomen breathing waveforms - guidance text below the breathing waveforms - one portrait screen with no scrolling ![SmartSpectra iOS quickstart demo](ios-quickstart.gif) ## Step 1 — Create the project In Xcode, create a new iOS app project: 1. Select `File` → `New` → `Project...` 2. Choose `iOS` → `App` 3. Set `Product Name` to `Cool Vitals` 4. Set `Interface` to `SwiftUI` 5. Set `Language` to `Swift` 6. Save the project If you already created the project, open it instead. In Finder, open: - `Cool Vitals/Cool Vitals.xcodeproj` Then select the app target in Xcode. ## Step 2 — Add the SmartSpectra package In Xcode: 1. Click `File` → `Add Package Dependencies...` 2. Paste `https://github.com/Presage-Security/SmartSpectra-Swift/` 3. For repeatable builds, choose `Exact Version` and enter a released tag such as `3.4.0`, from the [SmartSpectra-Swift releases](https://github.com/Presage-Security/SmartSpectra-Swift/releases) 4. Use `Branch` → `main` only when testing the latest final public release before pinning a version 5. Add the package to the `Cool Vitals` app target Manual check: - In the project navigator, you should now see `Package Dependencies` - `SmartSpectra` should be attached to the app target ## Step 3 — Add camera permission In Xcode: 1. Select the `Cool Vitals` target 2. Open the `Info` tab 3. Add a new key named `Privacy - Camera Usage Description` **NOTE** `Ctrl + Click` on the `Custom iOS Target Properties` and click `Add Row` 4. Set the value to `This app needs camera access to measure vitals.` ![FindInfo](FindInfo.png) ## Step 4 — Get `PresageService-Info.plist` This file does **not** come from Xcode. This file is **not** created when you add the SmartSpectra package. This file is **not** already inside your app project unless you or your team already downloaded it. You need to get it from Presage first: 1. Sign in to the Presage developer portal: `https://physiology.presagetech.com/auth/login` 2. Register for OAuth in the [Presage developer portal](https://physiology.presagetech.com/portal/apps) - Enter the app target's Bundle Identifier from `Signing & Capabilities` in the Application ID field. It must match exactly, including capitalization. - Enter your Team ID in the Organization ID field. Find it in **Xcode → Settings → Accounts**, select your Apple ID and team, and read the `Team ID` value — or in your [Apple Developer Account](https://developer.apple.com) under `Membership Details`. A Team ID is exactly 10 uppercase alphanumeric characters; it is not a certificate fingerprint. - Enable sandbox to additionally accept development App Attest identities from locally signed builds. Sandbox mode still accepts production identities from TestFlight and the App Store. When sandbox is disabled, only production identities are accepted. 3. Download the iOS OAuth config file named `PresageService-Info.plist` Use this `Signing & Capabilities` view in Xcode to find the Application ID and Organization ID inputs mentioned above: ![Signing & Capabilities view showing where Xcode exposes the Bundle Identifier for the Application ID and the signing identity used to look up the Organization ID](../../cpp/docs/images/macos-xcode-signing-source.png) An AI assistant connected to the [SmartSpectra MCP Server](../../docs/mcp-server.md) can do this step for you: it registers the bundle identifier and team ID (`apps.register`, including sandbox mode) and fetches `PresageService-Info.plist` (`apps.get_config`) without you opening the portal. Registration asks you to confirm before it takes effect. If you cannot find a download for `PresageService-Info.plist`, stop here. That means you do not have the OAuth config file yet. Ask [Presage support](mailto:support@presagetech.com) or your Presage contact for the iOS OAuth plist for this app. ## Step 5 — Add `PresageService-Info.plist` to Xcode In Xcode: 1. In Finder, locate the downloaded `PresageService-Info.plist` 2. Drag that file from Finder into the Xcode project navigator 3. When Xcode shows the add-file dialog: - enable `Copy items if needed` - make sure the `Cool Vitals` target is checked 4. Click `Finish` Then select the app target, open `Signing & Capabilities`, and add the **App Attest** capability. OAuth uses Apple's App Attest service, which requires a supported physical iOS or iPadOS device and a correctly provisioned app. Confirm `DCAppAttestService.shared.isSupported` is `true` on the device. The simulator cannot create an App Attest identity. `IS_OAUTH_ENABLED` is the authentication-mode selector. Set it to `true` for OAuth; set it to `false`, omit the key (older plist revisions), or omit the plist entirely, to use `sdk.config.apiKey`. If OAuth is enabled and an API key is also supplied, OAuth takes precedence and the API key is ignored. Boolean values and legacy plist integer values `0` and `1` are accepted; strings such as `"true"` are rejected as a non-retryable configuration error — a present-but-invalid value never falls back to API-key authentication. When OAuth is enabled, `PresageService-Info.plist` must contain non-blank string values for `CLIENT_ID` and `SUB`. `BUNDLE_ID`, when present, must exactly match the Bundle Identifier of the running app target (surrounding whitespace is ignored); a plist without `BUNDLE_ID` (older plist revisions) skips this local pre-check — the server still verifies the app's bundle identifier during authentication. SmartSpectra treats an invalid static field or mismatch as a non-retryable configuration error before contacting the authentication service. ## Step 6 — Replace `ContentView.swift` In Xcode: 1. Open `Cool Vitals/ContentView.swift` 2. Delete everything in the file 3. Paste the full file below Paste this entire file: ```swift import SwiftUI import SmartSpectra import AVFoundation struct ContentView: View { private enum TraceWindow { static let rate = 120 static let arterialWaveform = 240 static let breathingWaveform = 180 } private let sdk = SmartSpectraSDK.shared @State private var didAutoStart = false @State private var pulseRateBuffer: [MeasurementWithConfidence] = [] @State private var breathingRateBuffer: [MeasurementWithConfidence] = [] @State private var arterialPressureBuffer: [MeasurementWithConfidence] = [] @State private var chestBuffer: [SmartSpectra.Measurement] = [] @State private var abdomenBuffer: [SmartSpectra.Measurement] = [] @State private var latestHrv: Hrv? @State private var latestExpressionScores: [ExpressionScore] = [] init() { // The default camera selection prefers front when available. sdk.config.imageOutputEnabled = true sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics + [ .expressions, ] } private enum WaveformProminence { case primary case secondary } private var metrics: Metrics? { sdk.metrics } private var metricsUpdateToken: Int64 { [ metrics?.cardio.pulseRate.last?.timestamp, metrics?.breathing.rate.last?.timestamp, metrics?.cardio.arterialPressureTrace.last?.timestamp, metrics?.breathing.upperTrace.last?.timestamp, metrics?.breathing.lowerTrace.last?.timestamp, metrics?.cardio.hrv.last?.timestamp, metrics?.face.expression.last?.timestamp, ] .compactMap { $0 } .max() ?? 0 } private var pulseRateText: String { formatMetric(pulseRateBuffer.last.map { Double($0.value) }, digits: 0, suffix: " bpm") } private var breathingRateText: String { formatMetric(breathingRateBuffer.last.map { Double($0.value) }, digits: 0, suffix: " bpm") } private var hrvText: String { guard let value = latestHrv?.rmssd, value > 0 else { return "--" } return formatMetric(value, digits: 1, suffix: " ms") } private var latestExpressionScore: ExpressionScore? { latestExpressionScores.max(by: { $0.confidence < $1.confidence }) } private var latestExpressionLabel: String { guard let score = latestExpressionScore else { return "--" } let name = String(expressionName(score.type).prefix(8)) let paddedName = name + String(repeating: " ", count: max(0, 8 - name.count)) let percent = confidenceText(score.confidence) let paddedPercent = String(repeating: " ", count: max(0, 4 - percent.count)) + percent return "\(paddedName) \(paddedPercent)" } private var pulseConfidenceColor: Color { confidenceColor(pulseRateBuffer.last?.confidence) } private var breathingConfidenceColor: Color { confidenceColor(breathingRateBuffer.last?.confidence) } private var arterialPressureSamples: [Double] { arterialPressureBuffer.map { Double($0.value) } } private var chestSamples: [Double] { chestBuffer.map { Double($0.value) } } private var abdomenSamples: [Double] { abdomenBuffer.map { Double($0.value) } } private var statusText: String { switch sdk.processingStatus { case .idle: return "Idle" case .starting: return "Starting" case .running: return "Running" case .stopping: return "Stopping" case .error: return "Error" @unknown default: return "Unknown" } } private var validationTitle: String { guard let validationStatus = sdk.validationStatus else { return "Waiting" } return validationStatus.hint } private var statusColor: Color { switch sdk.processingStatus { case .running: return .green case .starting, .stopping: return .orange case .error: return .red case .idle: return .gray @unknown default: return .gray } } private var validationColor: Color { guard let validationStatus = sdk.validationStatus else { return .gray } switch validationStatus.code { case .ok: return .green case .cameraTuning: return .orange default: return .yellow } } var body: some View { GeometryReader { geometry in let compact = geometry.size.height < 820 let horizontalPadding: CGFloat = compact ? 12 : 16 let topSpacing: CGFloat = compact ? 8 : 12 let previewHeight = min(max(geometry.size.height * 0.26, 190), 250) VStack(spacing: topSpacing) { statusBar(compact: compact) .zIndex(1) previewCard .frame(height: previewHeight) .zIndex(0) HStack(spacing: topSpacing) { metricCard( title: "Pulse Rate", value: pulseRateText, valueColor: pulseConfidenceColor, accent: .red, compact: compact ) metricCard( title: "Breathing Rate", value: breathingRateText, valueColor: breathingConfidenceColor, accent: .cyan, compact: compact ) } .frame(maxHeight: compact ? 82 : 92) HStack(spacing: topSpacing) { metricCard( title: "HRV RMSSD", value: hrvText, valueColor: .white, accent: .mint, compact: compact ) metricCard( title: "Expression", value: latestExpressionLabel, valueColor: .white, accent: .orange, compact: compact, monospacedValue: true ) } .frame(maxHeight: compact ? 82 : 92) waveformCard( title: "Arterial Pressure", samples: arterialPressureSamples, accent: .purple, compact: compact, prominence: .primary ) .frame(height: compact ? 154 : 182) HStack(spacing: topSpacing) { waveformCard( title: "Chest Waveform", samples: chestSamples, accent: .cyan, compact: compact, prominence: .secondary ) waveformCard( title: "Abdomen Waveform", samples: abdomenSamples, accent: .blue, compact: compact, prominence: .secondary ) } .frame(height: compact ? 130 : 146) guidanceText } .padding(.horizontal, horizontalPadding) .padding(.vertical, compact ? 10 : 14) .background(backgroundGradient.ignoresSafeArea()) } .task { await startIfNeeded() } .task(id: metricsUpdateToken) { mergeCurrentMetrics() } } // The SDK's `hint` is written for end users — surface it as-is. private var guidanceText: some View { Text(sdk.validationStatus?.hint ?? "Getting ready\u{2026}") .font(.footnote) .foregroundStyle(.white.opacity(0.85)) .multilineTextAlignment(.center) .frame(maxWidth: .infinity) .accessibilityLabel("Measurement guidance") } private var previewCard: some View { ZStack { if let image = sdk.imageOutput { Image(uiImage: image) .resizable() .aspectRatio(contentMode: .fill) .frame(maxWidth: .infinity, maxHeight: .infinity) .clipped() } else { LinearGradient( colors: [Color(red: 0.16, green: 0.24, blue: 0.46), Color.black], startPoint: .topLeading, endPoint: .bottomTrailing ) VStack(spacing: 10) { Image(systemName: "camera.viewfinder") .font(.system(size: 40, weight: .semibold)) Text("Camera preview will appear here") .font(.headline) } .foregroundStyle(.white.opacity(0.92)) } LinearGradient( colors: [.black.opacity(0.68), .black.opacity(0.12), .clear], startPoint: .bottom, endPoint: .top ) } .frame(maxWidth: .infinity, maxHeight: .infinity) .clipShape(RoundedRectangle(cornerRadius: 26, style: .continuous)) .overlay( RoundedRectangle(cornerRadius: 26, style: .continuous) .stroke(.white.opacity(0.12), lineWidth: 1) ) .shadow(color: .black.opacity(0.35), radius: 18, x: 0, y: 10) } private func statusBar(compact: Bool) -> some View { HStack(spacing: compact ? 8 : 10) { badge(title: "Status", value: statusText, color: statusColor) badge(title: "Validation", value: validationTitle, color: validationColor) Spacer(minLength: 8) Button(action: toggleMeasurement) { Text(sdk.processingStatus == .running ? "Stop" : "Start") .font(.caption.bold()) .padding(.horizontal, compact ? 14 : 18) .padding(.vertical, 10) .background(.white, in: Capsule()) .foregroundStyle(.black) } } } private func metricCard( title: String, value: String, valueColor: Color, accent: Color, compact: Bool, monospacedValue: Bool = false ) -> some View { VStack(alignment: .leading, spacing: compact ? 6 : 8) { HStack(spacing: 6) { Circle() .fill(accent) .frame(width: 8, height: 8) Text(title) .font(.caption.weight(.semibold)) .foregroundStyle(.white) } Text(value) .font(.system(size: compact ? 21 : 24, weight: .bold, design: monospacedValue ? .monospaced : .rounded)) .foregroundStyle(valueColor) .monospacedDigit() .lineLimit(1) .minimumScaleFactor(0.7) } .dashboardCard() } private func waveformCard( title: String, samples: [Double], accent: Color, compact: Bool, prominence: WaveformProminence ) -> some View { VStack(alignment: .leading, spacing: compact ? 6 : 8) { VStack(alignment: .leading, spacing: 2) { Text(title) .font(.caption.weight(.semibold)) .foregroundStyle(.white) } ZStack { RoundedRectangle(cornerRadius: 14, style: .continuous) .fill(accent.opacity(0.12)) if samples.count > 1 { WaveformView( samples: samples, strokeColor: accent, verticalPaddingFraction: prominence == .primary ? 0.14 : 0.08 ) .padding(prominence == .primary ? 8 : 10) } } .frame(maxHeight: .infinity) .overlay( RoundedRectangle(cornerRadius: 14, style: .continuous) .stroke(accent.opacity(0.3), lineWidth: 1) ) } .dashboardCard() } private func badge(title: String, value: String, color: Color) -> some View { HStack(spacing: 6) { Circle() .fill(color) .frame(width: 8, height: 8) Text("\(title): \(value)") .font(.caption.weight(.semibold)) } .padding(.horizontal, 10) .padding(.vertical, 8) .background(.white.opacity(0.12), in: Capsule()) .foregroundStyle(.white) } private func mergeCurrentMetrics() { guard let metrics else { return } if !metrics.cardio.pulseRate.isEmpty { pulseRateBuffer.appendProtoArray(contentsOf: metrics.cardio.pulseRate) pulseRateBuffer = Array(pulseRateBuffer.suffix(TraceWindow.rate)) } if !metrics.breathing.rate.isEmpty { breathingRateBuffer.appendProtoArray(contentsOf: metrics.breathing.rate) breathingRateBuffer = Array(breathingRateBuffer.suffix(TraceWindow.rate)) } if !metrics.cardio.arterialPressureTrace.isEmpty { arterialPressureBuffer.appendProtoArray(contentsOf: metrics.cardio.arterialPressureTrace) arterialPressureBuffer = Array(arterialPressureBuffer.suffix(TraceWindow.arterialWaveform)) } if !metrics.breathing.upperTrace.isEmpty { chestBuffer.appendProtoArray(contentsOf: metrics.breathing.upperTrace) chestBuffer = Array(chestBuffer.suffix(TraceWindow.breathingWaveform)) } if !metrics.breathing.lowerTrace.isEmpty { abdomenBuffer.appendProtoArray(contentsOf: metrics.breathing.lowerTrace) abdomenBuffer = Array(abdomenBuffer.suffix(TraceWindow.breathingWaveform)) } if let hrv = metrics.cardio.hrv.last { latestHrv = hrv } if let scores = metrics.face.expression.last?.scores, !scores.isEmpty { latestExpressionScores = scores } } private func resetBuffers() { pulseRateBuffer.removeAll(keepingCapacity: true) breathingRateBuffer.removeAll(keepingCapacity: true) arterialPressureBuffer.removeAll(keepingCapacity: true) chestBuffer.removeAll(keepingCapacity: true) abdomenBuffer.removeAll(keepingCapacity: true) latestHrv = nil latestExpressionScores.removeAll(keepingCapacity: true) } private func toggleMeasurement() { Task { if sdk.processingStatus == .running || sdk.processingStatus == .starting { try? await sdk.stop() } else { resetBuffers() try? await sdk.start() } } } private func startIfNeeded() async { guard !didAutoStart else { return } didAutoStart = true guard sdk.processingStatus == .idle else { return } resetBuffers() try? await sdk.start() } private func confidenceText(_ confidence: Float?) -> String { guard let confidence, confidence.isFinite else { return "--" } let percent = min(max(Double(confidence), 0), 100) return "\(Int(percent.rounded()))%" } private func confidenceColor(_ confidence: Float?) -> Color { guard let confidence, confidence.isFinite else { return .white.opacity(0.65) } let percent = min(max(Double(confidence), 0), 100) switch percent { case 85...: return .green case 60..<85: return .yellow default: return .red } } private func formatMetric(_ value: Double?, digits: Int = 0, suffix: String = "") -> String { guard let value else { return "--" } if digits == 0 { return "\(Int(value.rounded()))\(suffix)" } return String(format: "% .\(digits)f", value).replacingOccurrences(of: " ", with: "") + suffix } private func expressionName(_ type: ExpressionType) -> String { switch type { case .unspecified: return "Unspecified" case .angry: return "Angry" case .contempt: return "Contempt" case .disgust: return "Disgust" case .fear: return "Fear" case .happy: return "Happy" case .neutral: return "Neutral" case .sad: return "Sad" case .surprise: return "Surprise" case .UNRECOGNIZED(_): return "Unknown" @unknown default: return "Unknown" } } private var backgroundGradient: LinearGradient { LinearGradient( colors: [ Color(red: 0.03, green: 0.05, blue: 0.12), Color(red: 0.07, green: 0.09, blue: 0.18), Color.black, ], startPoint: .topLeading, endPoint: .bottomTrailing ) } } private struct WaveformView: View { let samples: [Double] let strokeColor: Color let verticalPaddingFraction: Double var body: some View { GeometryReader { geometry in Path { path in guard samples.count > 1 else { return } let minValue = samples.min() ?? 0 let maxValue = samples.max() ?? 1 let rawRange = max(maxValue - minValue, 0.0001) let padding = rawRange * verticalPaddingFraction let lowerBound = minValue - padding let upperBound = maxValue + padding let range = max(upperBound - lowerBound, 0.0001) for (index, sample) in samples.enumerated() { let x = geometry.size.width * CGFloat(index) / CGFloat(samples.count - 1) let normalized = (sample - lowerBound) / range let y = geometry.size.height * (1 - normalized) if index == 0 { path.move(to: CGPoint(x: x, y: y)) } else { path.addLine(to: CGPoint(x: x, y: y)) } } } .stroke(strokeColor, style: StrokeStyle(lineWidth: 2.2, lineCap: .round, lineJoin: .round)) } } } private extension View { func dashboardCard() -> some View { self .padding(12) .background( RoundedRectangle(cornerRadius: 20, style: .continuous) .fill(Color.white.opacity(0.08)) ) .overlay( RoundedRectangle(cornerRadius: 20, style: .continuous) .stroke(Color.white.opacity(0.08), lineWidth: 1) ) } } ``` ## Step 7 — Build and run on a phone In Xcode: 1. Choose a supported physical iOS or iPadOS device as the run destination 2. Confirm the app target has the **App Attest** capability 3. Build and run the app 4. Allow camera access when iOS asks 5. Wait a few seconds for camera tuning and signal stabilization Run this on a physical device — the simulator has no camera. (For automated tests, where frames come from a video file instead, the simulator is supported; see [Headless testing in CI](headless-testing-in-ci.md).) ## What success looks like When your program is running, you should see all of these: - `Status` and `Validation` chips are visible at the top - the preview is below the chips - the arterial pressure waveform is larger than the breathing waveforms - chest and abdomen waveforms both appear on screen - the guidance text is below those waveforms - the pulse and breath-rate numbers change color with confidence - expressions and HRV are reported ## Expected OAuth check If OAuth is wired correctly, you should not see this fallback log at launch: - `PresageService-Info.plist not found. OAuth authentication will be disabled. Using API key authentication instead.` If you do see it, the plist is missing from the target or was not copied into the app. ## Common manual mistakes If the screen does not match the target state, check these first: - the package was added to the wrong target - `ContentView.swift` was only partially replaced - `PresageService-Info.plist` is not in target membership - the app is still running an older installed build on the phone - the app was run in the simulator instead of on a supported physical iOS or iPadOS device ## Swift Troubleshooting Source: https://presagetech.com/docs/smartspectra-sdk/swift/troubleshooting --- title: Swift Troubleshooting description: Troubleshoot SmartSpectra Swift setup, camera permissions, custom input frame rejections, timestamps, and iOS Simulator measurements. sidebarTitle: Troubleshooting --- # iOS Troubleshooting ## Installation & Setup ### Package not found in Xcode Ensure you're adding the package via **File → Add Package Dependencies...**, entering `https://github.com/Presage-Security/SmartSpectra-Swift`, and selecting the release tag `3.4.0`, from the [SmartSpectra-Swift releases](https://github.com/Presage-Security/SmartSpectra-Swift/releases), for repeatable builds. Pin the current release rather than an older one — the [migration guide](migration-guide.md) documents behaviour changes since 3.0. Use **Branch → main** only when testing the latest final public release before pinning a version. If you pasted a subdirectory URL such as `/tree/main/swift/sdk`, replace it with the repository root URL above. Swift Package Manager resolves the package from the repo root. --- ### Measurement does not start on the simulator The simulator has no camera, so camera-driven measurement needs a physical device. Select a real device target in Xcode for normal development. The simulator supports supplied video frames through public [`useCustomInput()`](headless-mode.md#use-your-own-camera-or-video-source), or the testing-only file-playback API — see [Headless testing in CI](headless-testing-in-ci.md), which runs a full measurement on the iOS Simulator. --- ## Camera & Permissions ### `NSCameraUsageDescription` missing This key is required when your app opens a camera, including SDK-owned capture. Custom input does not check or request camera access; a decoder-only integration does not need camera permission. Your own camera capture still needs it. In Xcode: 1. Select your app target. 2. Open the `Info` tab. 3. Add a new row for `Privacy - Camera Usage Description`. 4. Set the value to `This app needs camera access to measure vitals.` With SDK-owned capture, the SDK fails gracefully if this key is absent or empty. Or add the entry directly to your `Info.plist`: ```xml NSCameraUsageDescription This app needs camera access to measure vitals. ``` --- ### Camera permission denied at runtime For SDK-owned capture, the reference sample offers an action to open iOS Settings when camera permission is denied. With custom input, your app handles permission for its own capture source. Ensure your `Info.plist` description explains why your app needs camera access. --- ## Authentication ### Auth errors / measurements not starting 1. Verify your API key is correct, or that your OAuth plist is present and valid. 2. Ensure the device has an active internet connection. 3. Check that the key or app registration is active in [physiology.presagetech.com](https://physiology.presagetech.com/auth/login). If processing fails immediately with a missing-auth error, make sure you set `sdk.config.apiKey = "YOUR_KEY"` before calling `try await SmartSpectraSDK.shared.start()`. --- ### OAuth not working When registering your OAuth app, enter the app target's Bundle Identifier exactly as Xcode shows it, including capitalization. Enter your **Apple Org ID** (Team ID, e.g. `AB12CDE34F`) for the Organization ID: exactly 10 uppercase alphanumeric characters, not a certificate fingerprint. Find it in **Xcode → Settings → Accounts**, select your Apple ID and team, and read the `Team ID` value — or in your [Apple Developer Account](https://developer.apple.com) under `Membership Details`. Place the downloaded `PresageService-Info.plist` in your app, enable its app-target membership, and add the **App Attest** capability under the target's `Signing & Capabilities` tab. Run on a supported physical iOS or iPadOS device and confirm `DCAppAttestService.shared.isSupported` is `true`; the simulator cannot create an App Attest identity. The plist must set `IS_OAUTH_ENABLED` to `true` and contain non-blank string values for `CLIENT_ID` and `SUB`. `BUNDLE_ID`, when present, must exactly match the running app target's Bundle Identifier; older plists without `BUNDLE_ID` skip the local check and rely on the server's verification. An invalid field or mismatch produces a non-retryable `configurationFailed` error as soon as the SDK loads, before any authentication network request. The error message identifies the invalid field or mismatch without exposing its value. Portal sandbox behavior controls which App Attest identities the server accepts: - Enabled: accepts development identities from locally signed builds and production identities from TestFlight or the App Store. - Disabled: accepts production identities only. Your app repo should look roughly like this: ![Example plist location](plist_location_in_repo.png) > **Note:** Each bundle identifier can only be registered once. You cannot create multiple OAuth configs for the same bundle ID. --- ## Metrics & Data ### Pulse rate / cardio metrics not appearing Breathing metrics are enabled by default. Cardio metrics are not. Enable them explicitly: ```swift let sdk = SmartSpectraSDK.shared sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics // or sdk.config.requestedMetrics = [ .breathingRate, .pulseRate, .hrv ] ``` --- ### `metricsBuffer` / `$metricsBuffer` unresolved `MetricsBuffer` was removed. Replace `metricsBuffer` with `sdk.metrics`. The SDK now uses Swift Observation, so Combine-style `$` publishers such as `sdk.$metrics` are no longer available. ```swift // Before sdk.$metricsBuffer.sink { buffer in let pulse = buffer?.pulse.rate.last?.value } // After in SwiftUI if let metrics = sdk.metrics { let pulse = metrics.cardio.pulseRate.last?.value } ``` SwiftUI views automatically track reads of `sdk.metrics`. For UIKit or other non-SwiftUI code, observe SDK properties with `withObservationTracking` and re-arm the observation after each change. Field mapping: | Old (`metricsBuffer`) | New (`metrics`) | | --- | --- | | `pulse.rate` | `cardio.pulseRate` | | `pulse.trace` | `cardio.arterialPressureTrace` | | `breathing.rate` | `breathing.rate` | | `breathing.upperTrace` | `breathing.upperTrace` | > **Important:** Cardio fields now require cardio metrics to be requested explicitly, for example through `requestedMetrics`. Previously, `MetricsBuffer` provided pulse rate regardless of configuration. --- ## Headless Mode ### Custom input stays in `.starting` Await `sdk.start()`, then begin submitting frames. Do not wait for `processingStatus == .running` before sending the first frame. An accepted frame is not a completed measurement; continue observing `sdk.metrics`, `sdk.validationStatus`, and `sdk.error`. ### Custom frames are rejected Inspect the `SmartSpectraError` in `FrameSubmissionResult.rejected`: | Code | What to check | | --- | --- | | `.invalidState` | Await start before sending. Stop before selecting a source. A replaced input handle cannot be reused. | | `.frameConversionFailed` | Supply BGRA or 8-bit bi-planar NV12 with even dimensions for NV12. Sample buffers must contain ready, uncompressed pixels. | | `.nonMonotonicTimestamp` | Use finite, nonnegative, strictly increasing microsecond timestamps below `Int64.max - 2`. Sample buffers use their presentation timestamps. | | `.timestampGap` | Stop and start after a gap over two seconds; do not rewrite live capture timestamps to conceal an interruption. | Keep pixels alive and unchanged until submission returns. Supply upright pixels or select a fixed `FrameTransform`; the SDK does not infer sample orientation from metadata. See [custom input](headless-mode.md#use-your-own-camera-or-video-source) for supported layouts and lifecycle details. ### `processingStatus` cases don't match `SmartSpectraSDK.processingStatus` uses the current lifecycle states. Update any `switch` or comparisons: | Old case | New case | | --- | --- | | `.processing` | `.running` | | `.processed` | `.idle` | | `.idle` | `.idle` | | `.starting` | `.starting` | | `.stopping` | `.stopping` | | `.error` | `.error` | --- ### `startProcessing()` / `stopProcessing()` unresolved or inaccessible `SmartSpectraVitalsProcessor` is no longer part of the public Swift API. Use the async lifecycle methods on `SmartSpectraSDK.shared`: ```swift do { try await SmartSpectraSDK.shared.start() // Observe SmartSpectraSDK.shared.metrics, // SmartSpectraSDK.shared.processingStatus, // SmartSpectraSDK.shared.validationStatus, etc. try await SmartSpectraSDK.shared.stop() } catch { print("SmartSpectra error: \(error)") } ``` For older-to-current mappings, see the [iOS Migration Guide](migration-guide.md). --- ## Getting Help - Email: [support@presagetech.com](mailto:support@presagetech.com) - [Submit a GitHub issue](https://github.com/Presage-Security/SmartSpectra-Swift/issues) - API reference: [Swift API reference](api-reference.md) ## Use Case Examples Source: https://presagetech.com/docs/smartspectra-sdk/swift/use-case-examples --- title: Use Case Examples description: SmartSpectra Swift examples for metrics, face landmarks, custom camera and decoded video input, data export, and iOS UI state. sidebarTitle: Use Case Examples --- # iOS Use Case Examples These are intentionally short, partial examples. They are meant to show one pattern at a time, not serve as complete drop-in apps. ## Shared Setup Most examples assume you already own an SDK instance: ```swift import SwiftUI import AVFoundation import Charts import SmartSpectra struct ExampleHostView: View { private let sdk = SmartSpectraSDK.shared init() { sdk.config.apiKey = "YOUR_API_KEY" } var body: some View { Text("Ready") } } ``` ## Accessing Face Mesh Read the latest face landmarks from `sdk.metrics` and render them into your own overlay. Face landmarks are only populated when face metrics are requested. Landmarks are **pixel coordinates in the capture frame**, so scale `x` by the frame width and `y` by the frame *height* — they are different numbers. Capture defaults to 1280x720; if you change the capture resolution, change these constants to match. ```swift let captureWidth: CGFloat = 1280 let captureHeight: CGFloat = 720 sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.faceMetrics if let latestLandmarks = sdk.metrics?.face.landmarks.last { GeometryReader { geometry in ZStack { ForEach(Array(latestLandmarks.value.enumerated()), id: \.offset) { _, landmark in Circle() .fill(.blue) .frame(width: 3, height: 3) .position( x: CGFloat(landmark.x) * geometry.size.width / captureWidth, y: CGFloat(landmark.y) * geometry.size.height / captureHeight ) } } } } ``` ## Accessing Metrics Read the latest metrics directly from the SDK object. Pulse and HRV values require cardio metrics to be requested. ```swift sdk.config.requestedMetrics = SmartSpectraConfig.breathingMetrics + SmartSpectraConfig.cardioMetrics if let metrics = sdk.metrics { if let breathingRate = metrics.breathing.rate.last { Text("Breathing: \(breathingRate.value.formatted()) RPM") } if let pulseRate = metrics.cardio.pulseRate.last { Text("Pulse: \(pulseRate.value.formatted()) BPM") } } ``` ## LineChartView Use a small reusable chart view for breathing, pulse, or confidence values. ```swift struct LineChartView: View { let orderedPairs: [(time: Date, value: Float)] let title: String var body: some View { VStack(alignment: .leading) { Text(title) .font(.headline) Chart { ForEach(orderedPairs, id: \.time) { pair in LineMark( x: .value("Time", pair.time), y: .value("Value", pair.value) ) } } .frame(height: 180) } } } ``` You can feed it from metrics like this: ```swift if let breathingSeries = sdk.metrics?.breathing.rate { LineChartView( orderedPairs: breathingSeries.map { ( time: Date(timeIntervalSince1970: Double($0.timestamp) / 1_000_000.0), value: $0.value ) }, title: "Breathing Rate" ) } ``` ## Metrics Data Export Convert the latest metrics into your own export format before writing to disk or uploading. ```swift struct MetricsExportRow: Codable { let timestampUs: Int64 let breathingRate: Float? let pulseRate: Float? } if let metrics = sdk.metrics { let row = MetricsExportRow( timestampUs: metrics.breathing.rate.last?.timestamp ?? metrics.cardio.pulseRate.last?.timestamp ?? 0, breathingRate: metrics.breathing.rate.last?.value, pulseRate: metrics.cardio.pulseRate.last?.value ) let jsonData = try JSONEncoder().encode(row) let jsonString = String(decoding: jsonData, as: UTF8.self) print(jsonString) } ``` ## State Management Own the SDK once and bind UI directly to its observable state. ```swift struct MonitoringView: View { private let sdk = SmartSpectraSDK.shared init() { sdk.config.apiKey = "YOUR_API_KEY" } var body: some View { VStack { Text("Status: \(String(describing: sdk.processingStatus))") if let validationStatus = sdk.validationStatus { Text(validationStatus.hint) } if let error = sdk.error { Text(error.message) .foregroundStyle(.red) } } } } ``` ## Camera Handling For SDK-owned capture, select the camera while stopped before calling `try await sdk.start()`. ```swift let sdk = SmartSpectraSDK.shared sdk.config.apiKey = "YOUR_API_KEY" try sdk.useCamera(.front) ``` To change cameras, stop, select the camera, and start again. Explicit selections never fall back to another camera. ```swift @MainActor func switchToBackCamera() async throws { let sdk = SmartSpectraSDK.shared try await sdk.stop() try sdk.useCamera(.back) try await sdk.start() } ``` ## Your Own Camera or Video Decoder If your app already captures video, select `try sdk.useCustomInput()` while stopped, await `sdk.start()`, and pass each uncompressed sample to `input.sendFrame(sampleBuffer)`. Handle `.rejected(let error)` and observe metrics as above. Your app keeps ownership of its camera and buffers. The [AVFoundation example](headless-mode.md#use-your-own-camera-or-video-source) shows capture delegation and buffer requirements. The demo app's [Video Testing sample](https://github.com/Presage-Security/SmartSpectra/blob/main/swift/samples/demo-app/VideoInput/VideoTestingView.swift) uses `AVAssetReader` to decode a clip and submit its frames through the public API, including cancellation, frame rejection, and cleanup. Both paths require normal SDK authentication.