--- title: SmartSpectra Metrics Payload Data Types description: The language-agnostic SmartSpectra metrics payload — every message, enum, and field the SDK emits. sidebarTitle: Data Types --- > SmartSpectra SDK developer docs for v3.4.0, the latest stable release. Web page: https://presagetech.com/docs/smartspectra-sdk/data-types > Index of all pages: https://presagetech.com/docs/smartspectra-sdk/llms.txt ## InsightType Origin of an Insight. Only INSIGHT_TYPE_VITALS is emitted on delivered Insights today; SPEECH and COMBINED are reserved for a future release. Correlate an on-demand reply with its prompt via Insight.request_id, not this field. ```proto enum InsightType { INSIGHT_TYPE_VITALS = 0; INSIGHT_TYPE_SPEECH = 1; INSIGHT_TYPE_COMBINED = 2; } ``` - `INSIGHT_TYPE_VITALS` -- Auto-fired vitals snapshot dispatched periodically by InsightSession with the accumulated metrics buffer and no caller prompt. The only value set on delivered Insights today. - `INSIGHT_TYPE_SPEECH` -- Reserved, not emitted. Describes a request from RequestInsight with a prompt but no buffered metrics at dispatch time (prompt-only nudge). - `INSIGHT_TYPE_COMBINED` -- Reserved, not emitted. Describes a request from RequestInsight carrying both the caller's prompt and a non-empty metrics buffer. ## Insight LLM-generated analysis or error returned by the insights endpoint. ### Properties ```proto message Insight { int32 request_id = 1; string processed_at = 2; oneof result { string analysis = 3; string error = 4; } InsightType type = 5; } ``` - `int32` `request_id` -- Client-generated request ID. Populated client-side by the async callback closure (not returned by the server). Lets callers correlate responses to requests. - `string` `processed_at` -- ISO timestamp of when the analysis was produced - `string` `analysis` -- The LLM analysis text, set on success. - `string` `error` -- Error message, set on failure. - [`InsightType`](#insighttype) `type` -- Origin of this insight. Always INSIGHT_TYPE_VITALS today — SPEECH and COMBINED are reserved and never set — so route on request_id rather than on this field to tell auto-fired snapshots from on-demand replies. ## FeatureType FeatureType defines high-level physiological measurement categories. Each feature represents a super-metric that may encompass multiple individual metrics. ```proto enum FeatureType { BREATHING = 0; EDA = 2; FACE = 3; CARDIO = 4; } ``` > Some wire values are intentionally omitted from this view; the gaps in the numeric sequence preserve compatibility with the underlying proto. Use the listed names — do not renumber. - `BREATHING` -- Breathing measurements (chest and abdomen) - `EDA` -- EDA Proxy (electrodermal activity) measurement - `FACE` -- Facial tracking and analysis - `CARDIO` -- Cardiovascular metrics (pulse, HRV, relative arterial pressure waveform) ## MetricType MetricType defines individual physiological measurements that can be requested and produced during video-based vital signs analysis. ```proto enum MetricType { CHEST_BREATHING = 0; ABDOMEN_BREATHING = 1; BREATHING_RATE = 2; BREATHING_AMPLITUDE = 3; APNEA = 4; RESPIRATORY_LINE_LENGTH = 5; BASELINE = 6; INHALE_EXHALE_RATIO = 7; EDA_TRACE = 10; FACE_LANDMARKS = 11; BLINKING = 12; TALKING = 13; EXPRESSIONS = 14; PULSE_RATE = 15; ARTERIAL_PRESSURE_TRACE = 16; HRV = 17; } ``` > Some wire values are intentionally omitted from this view; the gaps in the numeric sequence preserve compatibility with the underlying proto. Use the listed names — do not renumber. - `CHEST_BREATHING` -- Breathing upper (chest) metrics - `ABDOMEN_BREATHING` -- Breathing lower (abdomen) metrics - `BREATHING_RATE` -- Breathing aggregate metrics - `EDA_TRACE` -- EDA Proxy metrics - `FACE_LANDMARKS` -- Face metrics - `PULSE_RATE` -- Cardio metrics ## RequestedMetrics Wrapper message for passing a list of requested metrics through MediaPipe packets ### Properties ```proto message RequestedMetrics { repeated MetricType metrics = 1; } ``` ## Measurement Represents a single measurement with timestamp and stability information. Used for various physiological measurements throughout the system. ### Properties ```proto message Measurement { float value = 1; bool stable = 2; int64 timestamp = 3; } ``` - `float` `value` -- The measured or estimated value - `bool` `stable` -- Whether the measurement is considered stable/reliable - `int64` `timestamp` -- Absolute timestamp at which the measurement was taken, in microseconds, since Linux epoch ## DetectionStatus Represents detection status with timestamp information. Used to track whether a particular physiological feature or state is detected. ### Properties ```proto message DetectionStatus { bool detected = 1; bool stable = 2; int64 timestamp = 3; } ``` - `bool` `detected` -- Whether the feature/state was detected - `bool` `stable` -- Whether the detection is considered stable/reliable - `int64` `timestamp` -- Absolute timestamp at which the detection status was updated, in microseconds, since Linux epoch ## MeasurementWithConfidence Represents a measurement with an associated confidence score. Extends basic measurement with confidence information for quality assessment. ### Properties ```proto message MeasurementWithConfidence { float value = 1; bool stable = 2; float confidence = 3; int64 timestamp = 4; } ``` - `float` `value` -- The measured value - `bool` `stable` -- Whether the measurement is considered stable/reliable - `float` `confidence` -- Confidence score for the measurement, expressed as a percentage in the range [0.0, 100.0] - `int64` `timestamp` -- Absolute timestamp at which the measurement was taken, in microseconds, since Linux epoch ## ExpressionType Enumerates the supported facial expression types output by the model. ```proto enum ExpressionType { UNSPECIFIED = 0; ANGRY = 1; CONTEMPT = 2; DISGUST = 3; FEAR = 4; HAPPY = 5; NEUTRAL = 6; SAD = 7; SURPRISE = 8; } ``` - `UNSPECIFIED` -- Expression is unspecified or unknown - `ANGRY` -- Angry expression - `CONTEMPT` -- Contempt expression - `DISGUST` -- Disgust expression - `FEAR` -- Fear expression - `HAPPY` -- Happy expression - `NEUTRAL` -- Neutral expression - `SAD` -- Sad expression - `SURPRISE` -- Surprise expression ## ExpressionScore Associates an expression type with its confidence score. ### Properties ```proto message ExpressionScore { ExpressionType type = 1; float confidence = 2; } ``` - [`ExpressionType`](#expressiontype) `type` -- Expression type identifier - `float` `confidence` -- Confidence score for the expression, expressed as a percentage in the range [0.0, 100.0] ## Expression Represents a detected expression with metadata. Used for facial expression analysis and emotion detection. ### Properties ```proto message Expression { bool stable = 1; int64 timestamp = 2; repeated ExpressionScore scores = 3; } ``` - `bool` `stable` -- Whether the detection is considered stable/reliable - `int64` `timestamp` -- Absolute timestamp at which the expression was detected, in microseconds, since Linux epoch - `repeated` [`ExpressionScore`](#expressionscore) `scores` -- Confidence distribution across all expression types ## Hrv Represents a single HRV measurement computed over some period of pulse data. ### Properties ```proto message Hrv { double rmssd = 1; double mean_nn = 2; double sdnn = 3; double baevsky = 4; int64 timestamp = 5; float confidence = 6; bool stable = 7; } ``` - `double` `rmssd` -- root mean square of successive differences between normal heartbeats - `double` `mean_nn` -- mean normal-to-normal (NN) interval length - `double` `sdnn` -- Standard Deviation of normal-to-normal (NN) Intervals - `double` `baevsky` -- Baevsky's Stress Index: a measure of autonomic balance derived from the NN interval distribution, computed as `AMo / (2 * Mo * MxDMn)`. Reported without a unit, matching the HRV model card. - `int64` `timestamp` -- Absolute timestamp at which the HRV measurement was taken, in microseconds, since Linux epoch - `float` `confidence` -- Confidence score for the HRV measurement, expressed as a percentage in the range [0.0, 100.0] - `bool` `stable` -- Whether the HRV measurement is considered stable/reliable ## Pulse Comprehensive pulse-related measurements and derived metrics. Contains heart rate, pulse trace, and respiratory coupling information. ### Properties ```proto message Pulse { repeated MeasurementWithConfidence rate = 1; repeated Measurement trace = 2; repeated Measurement pulse_respiration_quotient = 3; } ``` - `repeated` [`MeasurementWithConfidence`](#measurementwithconfidence) `rate` -- Heart rate measurements with confidence scores - `repeated` [`Measurement`](#measurement) `trace` -- Raw pulse trace measurements - `repeated` [`Measurement`](#measurement) `pulse_respiration_quotient` -- Pulse-respiration quotient measurements indicating cardio-respiratory coupling ## Breathing Comprehensive breathing/respiratory measurements and derived metrics. Contains respiratory rate, traces, and various breathing pattern indicators. ### Properties ```proto message Breathing { repeated MeasurementWithConfidence rate = 1; repeated Measurement upper_trace = 2; repeated Measurement lower_trace = 3; repeated Measurement amplitude = 4; repeated DetectionStatus apnea = 5; repeated Measurement respiratory_line_length = 6; repeated Measurement baseline = 7; repeated Measurement inhale_exhale_ratio = 8; } ``` - `repeated` [`MeasurementWithConfidence`](#measurementwithconfidence) `rate` -- Respiratory rate measurements with confidence scores. `stable` marks confidence at or above the minimum accepted accuracy standard for breathing rate (+/-1 br/min), which corresponds to confidence >= 45; see the [breathing model card](https://vandv.presagetech.com/breathing_model_card.html) for the confidence-to-error mapping. - `repeated` [`Measurement`](#measurement) `upper_trace` -- Chest breathing movement trace measurements. The trace has no confidence of its own; `stable` is inherited from the breathing rate verdict (>= 45); see the [breathing model card](https://vandv.presagetech.com/breathing_model_card.html). - `repeated` [`Measurement`](#measurement) `lower_trace` -- Abdominal breathing movement trace measurements. The trace has no confidence of its own; `stable` is inherited from the breathing rate verdict (>= 45); see the [breathing model card](https://vandv.presagetech.com/breathing_model_card.html). - `repeated` [`Measurement`](#measurement) `amplitude` -- Breathing amplitude measurements - `repeated` [`DetectionStatus`](#detectionstatus) `apnea` -- Apnea (breathing cessation) detection status - `repeated` [`Measurement`](#measurement) `respiratory_line_length` -- Respiratory line length measurements for breathing pattern analysis - `repeated` [`Measurement`](#measurement) `baseline` -- Baseline breathing measurements - `repeated` [`Measurement`](#measurement) `inhale_exhale_ratio` -- Inhale to exhale duration ratio measurements ## Landmarks Facial landmark coordinates with temporal and stability information. Used for face tracking and facial feature analysis. ### Properties ```proto message Landmarks { repeated Point2dFloat value = 1; bool stable = 2; bool reset = 3; int64 timestamp = 4; } ``` - `repeated` [`Point2dFloat`](#point2dfloat) `value` -- Array of 2D coordinate points representing facial landmarks - `bool` `stable` -- Whether the landmark detection is considered stable/reliable - `bool` `reset` -- Indicates whether the landmark set was reset (cannot be directly associated with previous set) - `int64` `timestamp` -- Absolute timestamp at which the landmarks were detected, in microseconds, since Linux epoch ## Face Comprehensive facial analysis measurements and detections. Contains blinking, talking detection, landmarks, and expressions. ### Properties ```proto message Face { repeated DetectionStatus blinking = 1; repeated DetectionStatus talking = 2; repeated Landmarks landmarks = 3; repeated Expression expression = 4; } ``` - `repeated` [`DetectionStatus`](#detectionstatus) `blinking` -- Blinking detection status over time - `repeated` [`DetectionStatus`](#detectionstatus) `talking` -- Talking/speech detection status over time - `repeated` [`Landmarks`](#landmarks) `landmarks` -- Facial landmark coordinates over time - `repeated` [`Expression`](#expression) `expression` -- Detected expressions over time ## Eda EDA Proxy (electrodermal activity) measurements. Tracks skin conductance changes related to autonomic nervous system activity. ### Properties ```proto message Eda { repeated Measurement trace = 1; } ``` - `repeated` [`Measurement`](#measurement) `trace` -- EDA Proxy trace measurements over time ## Cardio ### Properties ```proto message Cardio { repeated MeasurementWithConfidence pulse_rate = 1; repeated MeasurementWithConfidence arterial_pressure_trace = 2; repeated Hrv hrv = 3; } ``` - `repeated` [`MeasurementWithConfidence`](#measurementwithconfidence) `pulse_rate` -- Heart rate measurements with confidence scores. `stable` marks confidence at or above the minimum accepted accuracy standard for heart rate (+/-3 bpm), which corresponds to confidence >= 40; see the [arterial pressure model card](https://vandv.presagetech.com/arterial_pressure_model_card.html) for the confidence-to-error mapping. Heart rate is derived from the same signal. - `repeated` [`MeasurementWithConfidence`](#measurementwithconfidence) `arterial_pressure_trace` -- Arterial pressure trace (uncalibrated, unitless) measurements with confidence scores. `stable` uses the same threshold as heart rate -- the minimum accepted accuracy standard (+/-3 bpm), confidence >= 40 -- since both derive from the arterial pressure signal; see the [arterial pressure model card](https://vandv.presagetech.com/arterial_pressure_model_card.html). - `repeated` [`Hrv`](#hrv) `hrv` -- Heart rate variability measurements with confidence scores. `stable` marks confidence at or above the minimum accepted accuracy standard for HRV (+/-5 ms), which corresponds to confidence >= 50; see the [HRV model card](https://vandv.presagetech.com/hrv_model_card.html) for the confidence-to-error mapping. ## Metrics Comprehensive physiological metrics container. Contains all available physiological measurements and analysis results. ### Properties ```proto message Metrics { Breathing breathing = 1; Eda eda = 3; Face face = 4; Cardio cardio = 5; } ``` - [`Breathing`](#breathing) `breathing` -- Breathing and respiratory analysis results - [`Eda`](#eda) `eda` -- EDA Proxy (electrodermal activity) measurements. Note: processing needs to run for over 35 seconds to generate the first EDA Proxy result. - [`Face`](#face) `face` -- Facial analysis results - [`Cardio`](#cardio) `cardio` -- Cardiovascular measurements (pulse rate, arterial pressure, HRV) ## Point2dInt32 Represents a 2D point with integer coordinates. Used for pixel-based coordinate systems and discrete positioning. ### Properties ```proto message Point2dInt32 { int32 x = 1; int32 y = 2; } ``` - `int32` `x` -- X coordinate as 32-bit signed integer - `int32` `y` -- Y coordinate as 32-bit signed integer ## Point2dFloat Represents a 2D point with floating-point coordinates. Used for precise positioning, normalized coordinates, and sub-pixel accuracy. ### Properties ```proto message Point2dFloat { float x = 1; float y = 2; } ``` - `float` `x` -- X coordinate as floating-point value - `float` `y` -- Y coordinate as floating-point value ## Point3dFloat Represents a 3D point with floating-point coordinates. Used for spatial positioning, depth information, and 3D landmark representation. ### Properties ```proto message Point3dFloat { float x = 1; float y = 2; float z = 3; } ``` - `float` `x` -- X coordinate as floating-point value - `float` `y` -- Y coordinate as floating-point value - `float` `z` -- Z coordinate (depth) as floating-point value