How I Built a Fast, Accurate, and Context-Aware Gmail Writing Chrome Extension Using Web Speech API ,Laravel and LLMs

The problem this extension, which I named 'IntelCoPilot,' solves is the pain point of writing accurate technical emails. The Chrome extension handles voice-to-text conversion, AI-driven response generation, and auto-insertion of responses in Gmail's compose window. In this blog i go through its architecture which may inspire you to build solutions by leveraging Web Speech API and generative AI.

What problem does this Chrome extension solve?

As a backend engineer, writing emails to clients and third parties during implementations can be time-consuming and tedious. IntelCoPilot streamlines this process by helping you draft accurate, context-aware technical emails in seconds. Instead of typing, you can simply use your voice—just click a button in Gmail’s compose window, and the extension will generate a precise, well-structured email instantly.

IntelCoPilot Chrome Extension Architecture

1. Chrome Extension (Frontend)

The frontend of the IntelCoPilot Chrome Extension is responsible for capturing user voice input, processing it into text, and integrating AI-generated responses directly into the Gmail compose window.

Speech Recognition

• Utilizes the Web Speech API for real-time voice-to-text transcription.
• Recognition stops automatically on user click or when finished speaking.

UI Components

• A Icon/Mic button appears in Gmail’s compose window.
• AI-generated email drafts are auto-inserted into the email body.

DOM Observers

• Uses MutationObserver and ResizeObserver to detect new compose windows and track UI changes dynamically.

API Communication

• Sends transcribed text and email context from the dom or user input from compose window to the Laravel backend.
• Receives AI-generated responses and inserts them into the compose window.

2. Laravel Backend (API)

The backend, built with Laravel, processes AI requests, handles authentication, and ensures security and efficiency.

AI Processing Layer

• Integrates multiple LLM APIs using factory design pattern to generate email responses based on context and user voice prompts.
• Supports customization for different response tones and formats.

Authentication & Security

• Uses Laravel Sanctum for secure API access.
• Encrypts user data to prevent unauthorized access.
• Implements rate limiting to manage API usage effectively.

3. Request-Response Cycle

Step 1: User Clicks Mic Button

• Speech recognition begins.
• Voice input is converted to text.

Step 2: Data Sent to Laravel

• Transcribed text and email context are sent to the API.
• Laravel processes the AI request.

Step 3: Laravel Calls LLM's API

• GPT-4 generates a professional email response with proper role,prompt and context.

Step 4: AI Response Sent Back

• The Chrome Extension receives the AI-generated response.
• The response is inserted into the Gmail compose window for the user to review and send.

Conclusion

The IntelCoPilot Chrome Extension help in writing fast,accurate and context aware emails by leveraging AI for speech recognition and smart email drafting. With its seamless integration with Laravel, OpenAI, and future Gmail API enhancements, it aims to enhance productivity while maintaining top-tier security. Stay tuned for upcoming updates that will make this tool even more indispensable! I will may publish this extension as an open source project.