SpeakEn: Building an AI Speaking Coach End-to-End

Speaking a new language fluently requires consistent practice with feedback. Most learners lack access to native speakers for regular conversation. The goal was to build an AI-powered speaking coach that provides real-time feedback, tracks progress, and keeps learners engaged through gamification and varied practice scenarios.

SpeakEn (AI English Coach) is a complete speaking practice product: user onboarding with level assessment, multiple conversation personas with distinct personalities, real-time speech recognition and AI feedback, session scoring on multiple dimensions (fluency, grammar, vocabulary, pronunciation), progress tracking with XP and streaks, flashcard review for mistakes, and IELTS-specific practice mode.

The session model captures user speech input, AI persona responses, per-turn feedback, and aggregate scoring. Each session is linked to a persona and topic. The schema supports both free-form conversation and structured IELTS prompts. Progress is computed from session aggregates rather than stored separately, keeping the data model normalized.

Users speak via browser microphone. Audio is sent to a speech-to-text service (STT) for transcription. The transcript is analyzed by an LLM for grammar, vocabulary, fluency, and pronunciation feedback. The response is synthesized back as persona dialogue. The pipeline runs asynchronously with loading states to keep UX smooth.

Decision: Client-side STT vs server-side. Chose server-side for model flexibility and easier provider switching. Cost: higher latency per turn (~2–3s) but better accuracy and upgradability. Decision: Session history vs real-time conversation only. History adds DB complexity but enables progress tracking and review — a must for learner retention.

Add spaced-repetition review for vocabulary from mistakes. Implement conversation branching for more natural dialogue. Explore WebRTC for lower-latency streaming speech. Add cohort analytics if user base grows.