Saffron & Smoke:
Order by talking
Voice-to-voice restaurant ordering on a Pipecat pipeline with Gemini Live native audio — 21 agent-callable tools drive both the order state and the client canvas, over a hybrid pgvector + keyword menu search and a complete restaurant back-office.
CORE CAPABILITIES
Voice-to-Voice
Gemini Live native audio over a full-duplex WebSocket, 24kHz mono PCM16 out, with Silero VAD turn-taking.
21 Agent Tools
search_menu, add_to_order, check_delivery_area, confirm_order, request_bill, set_table and more drive both order state and the UI canvas.
Hybrid Menu Search
Keyword ranking fused with pgvector cosine similarity via Reciprocal Rank Fusion (k=60), on 1024-dim mistral-embed vectors.
Delivery Gate
Pincode-based service-area check backed by a dedicated service and repository.
Session Guardrails
Single-use Redis WebSocket tickets with a 30s TTL, a global 3-session concurrency cap, and a 900s max session length.
Full Back-Office
11 admin routes with role-based access: kitchen display, floor plan, tables & QR, menu CMS, analytics and more.
Ordering Experience
A conversation on the left, an agent-driven canvas on the right.
Landing
Voice Ordering
Agent-Driven Canvas
Order Confirmed
AGENT TOOLS (SELECTED)
search_menu
Hybrid keyword + vector menu lookup.
add_to_order
Push an item into the live order state.
check_delivery_area
Validate a pincode against service areas.
set_table
Bind the session to a dine-in table.
confirm_order
Finalise the order and update the canvas.
request_bill
Apply GST and service-charge bill policy.
6 of 21 tools shown.
BACK-OFFICE
11 admin routes gated by role-based access control. Dine-in runs through QR table binding, customers are recalled with 4-digit codes, and billing applies a configurable GST and service-charge policy.
TECHNICAL ARCHITECTURE
voice
Pipecat
Full-duplex realtime pipeline
Gemini Live
gemini-3.1-flash-live-preview, voice "Charon"
Silero VAD
Via LLMUserAggregatorParams
backend
FastAPI + SQLAlchemy
API and data layer
PostgreSQL + pgvector
Menu embeddings, Alembic migrations
Redis
WebSocket tickets and concurrency limiter
client
Next.js 16 / React 19
Ordering client and back-office
Tailwind v4
Styling layer
ProtobufFrameSerializer
24kHz mono PCM16 audio over WebSocket