Multimodal interaction¶
One intent, many native experiences
Meaning meets each person in the right form
Unison accepts intent through the modalities available to a person and composes the outcome through forms suited to their needs, preferences, device, environment, and current context.
Meaning is the common integration point¶
Every experience begins with a semantic outcome. It describes the meaning, relationships, status, uncertainty, provenance, urgency, choices, actions, confirmation, privacy state, cancellation, errors, and recovery that a person needs to understand.
A modality adapter contributes two focused translations:
- It turns speech, text, touch, Braille input, sign, images, device events, or future signals into evidence of intent.
- It turns a governed semantic outcome into a native expression for the person and situation.
Identity, consent, policy, and actuation authority remain with their owning platform services. This common contract lets independent modality teams innovate while preserving equivalent understanding and control.
Native experiences use each modality's strengths¶
Conversation and voice
Dialogue, pacing, interruption, prosody, spatial audio, and concise spoken structure support hands-free and auditory interaction.
Visual
Composition, typography, imagery, spatial relationships, direct manipulation, and glanceable state support sighted interaction.
Braille and tactile
Structured tactile regions, routing, concise labels, status cells, and direct navigation support native reading and control.
Touch and physical controls
Buttons, switches, gestures, haptics, and tangible controls provide direct input and perceivable state.
Sign language
Language-aware recognition and expressive signing support communication through a person's selected sign language.
Future BCI
Qualified signals can support constrained intent expression, selection, timing, and control through the same governed contract.
A person can combine modalities. Conversation can accompany a tactile map. Braille can carry exact values while audio provides context. Sign can express intent while visual composition presents detailed evidence. Haptics can convey urgent state while another modality provides explanation.
Composition responds to the moment¶
Modality selection considers the person's preferences, available devices, privacy setting, environment, task demands, fatigue, urgency, and the assurance required for the intended action. The composition can change during an interaction while semantic state remains continuous.
A person might begin research visually, continue through conversation while walking, and review exact sources in Braille. A driving context can prioritize short dialogue and defer dense detail. A quiet shared room can favor sign, Braille, touch, or visual interaction. Each transition preserves current intent, pending choices, confirmation state, and recovery.
Shared understanding across different experiences¶
People can collaborate through different native expressions of the same semantic outcome. Each person receives the meaning, choices, attribution, uncertainty, and authority appropriate to them. Shared state records which facts and decisions are mutual while private context remains in each person's governed space.
Follow the shared-understanding journeys.
Privacy follows the modality path¶
Microphones, cameras, tactile devices, displays, headsets, and biological-signal devices each have explicit identity, presence, capture, retention, and disclosure policy. Local processing and derived representations can minimize raw media. Indicators communicate active sensing in forms perceivable to the person using the system.
A modality adapter declares the data it captures, the representations it derives, where processing occurs, how long information remains, which assurance it can establish, and how a person pauses, revokes, or recovers its use.
A stable contract for independent teams¶
Voice, visual, Braille, sign, tactile, switch, AAC, haptic, and BCI contributors integrate through the same intent and semantic-outcome contract. It also carries identity and assurance claims, capability declarations, policy references, confirmation state, cancellation, recovery, provenance, and evidence labels.