EyesAI:
Eyes for the phone
A Kotlin and Jetpack Compose Android app with a FastAPI backend that controls the device end-to-end through an AccessibilityService — a Planner → Executor → Verifier agent at 87% task completion, backed by on-device vision and TTS-first feedback.
No public repository or live deployment.
CORE CAPABILITIES
Autonomous Agent
Planner → Executor → Verifier owned by a Controller, reaching 87% task completion.
Accessibility Tree Control
A perception pass over the accessibility tree drives open_app, tap_element, tap_element_input_text_and_enter and done.
Dual-Pipeline Vision
On-device TFLite with COCO labels alongside an ML Kit labeler — 88.2% accuracy at 151ms latency.
Live Narration
Gemini Live scene narration over a throttled 1-fps frame feed from CameraX.
Multilingual Voice
English, Hindi and Hinglish commands at 94% accuracy — Levenshtein static matching with a Gemini function-calling fallback.
Non-Visual Feedback
TTS and haptics first, with displayLarge 36/44 and bodyLarge 17/24 accessibility typography.
On-Device Interface
Accessibility-first screens; the detection overlay is a debug view of the vision pipeline.
Home
Vision Pipeline
Autonomous Agent
AGENT LOOP
A Controller owns the loop: perception reads the accessibility tree, the Planner decides the next action, the Executor performs it on the device, and the Verifier confirms the result before the next step.
ACTION SPACE
open_app
Launch a target application.
tap_element
Tap a node resolved from the accessibility tree.
tap_element_input_text_and_enter
Focus a field, type, and submit.
done
Signal task completion back to the Controller.
TECHNICAL ARCHITECTURE
android
Kotlin
Application language
Jetpack Compose + Material 3
UI layer
CameraX
PreviewView, ImageCapture, ImageAnalysis
intelligence
TFLite + ML Kit
Dual-pipeline object detection
Gemini Live
Live scene narration
Porcupine
"Hey Eyes AI" wake word
backend
FastAPI
Service layer with WebSocket streaming
FAISS
Vector search
Vertex AI
Hosted model access