Pure emotions
Happy, calm, sad, angry, scared, surprised, confused, tired, hurt, overloaded, bored, sound.
Assistive Biometrics Lab
Local-first test queue for the camera collector: richer emotions, AAC-style wants, hands/body/audio, and AR mask experiments. No raw biometric storage by default.
AAC + child UI vocabulary
The prototype now separates pure emotion hints from practical communication cards. Wants include daily needs plus core words that are useful across many situations.
Wants + core words
Drink, eat, toilet, outside, play, sleep, help, break, quiet, hug, more, stop, yes, no, music.
Open-source / free candidates to test
MediaPipe Face Landmarker
Current base layer for face mesh, blendshapes, head pose, eye openness, and mouth movement.
Official web docsMediaPipe Hands + Pose
Current base layer for fingertips, pinch/open hand, body keypoints, leaving-frame, and restlessness.
Hand docsTensorFlow.js face landmarks
Alternative browser model to compare face stability, long-distance detection, and runtime cost.
GitHubJeeliz FaceFilter
AR-mask candidate for face-attached effects if the current canvas mask needs stronger tracking.
GitHubMindAR face tracking
Another browser AR candidate for face filters and anchored overlays.
GitHubAAC references
Vocabulary direction: wants, feelings, routine needs, and reusable core words for communication.
ASHA AAC overviewWearables
Future optional layer: Apple Watch or smart band heart rate, HRV, movement, activity, and rest context.
Apple health and fitness docsNext test queue
Mouth + tongue
Test mouth crop classifier or segmentation; current tongue value is only a mouth-open heuristic.
Gesture layer
Detect hand-to-mouth, pointing, self-touch, stop gesture, reach, wave, and repeated movement bursts.
Expression classifier
Train per-child classifier on blendshapes, pose, hands, audio, time context, and caregiver labels.
Audio layer
Classify nonverbal vocalization intensity, pitch contour, repeated hums, and distress bursts.
Wearable physiology
Add consented Apple Watch / smart band signals as context, not diagnosis: heart rate, HRV, motion, sleep/rest.