Drape is a virtual try-on application that lets a user see how a garment may look in two different ways: as a high-quality generated photo or as a continuously transformed live-camera experience.

The most interesting part is not only the AI model. It is the system around it. Drape coordinates image validation, asynchronous inference, provider selection, caching, hardware detection, secure API access, and a responsive browser interface in one local-first application.

At a high level, the system follows this architecture:

In parallel:

Webcam + Garment Reference ↓ Short-Lived Decart Token ↓ WebRTC Live Session ↓ Continuous Virtual Try-On

## Two Different Try-On Problems

A still photo and a live camera feed may look like versions of the same feature, but they create different engineering requirements.

Photo try-on can spend several seconds producing a polished result. It can validate both images, place the request in a queue, run an expensive inference job, cache the output, and let the user download the finished image.

Live try-on has a stricter latency budget. It must keep a video session active, transform frames continuously, and allow style changes without restarting the camera. Drape therefore treats the two experiences as separate inference paths instead of forcing one model and one API design to handle everything.

## FastAPI as the Coordination Layer

The backend is built with FastAPI and acts as the boundary between the browser, local model runtime, and external AI services.

It is responsible for:

- accepting person and garment uploads
- validating file size, format, and pixel count
- selecting a configured photo provider
- creating and tracking inference jobs
- serializing expensive local GPU work
- serving generated images
- reporting model, device, and provider status
- creating short-lived credentials for live sessions
- serving the static web application

This keeps provider credentials and inference details out of the frontend. The browser works with a stable set of Drape endpoints even when the selected AI provider changes.

## Input Validation Before Inference

AI image processing is expensive, so invalid input should be rejected before it reaches a model or consumes a paid API credit.

Drape limits each upload to 15 MB and rejects images above 30 million pixels. Pillow verifies the image data, applies EXIF orientation, and converts accepted JPG, PNG, or WebP files into a consistent RGB representation.

The interface also asks the user to identify the garment category as a top, bottom, or one-piece and to specify whether the source is a model photo or a flat-lay product image. These values give the selected provider more precise context and produce more predictable outputs.

## Provider Routing for Photo Try-On

The photo pipeline is deliberately provider-aware. An environment setting chooses the preferred route, while the backend checks whether the required credentials are actually available.

The current choices are:

- Genlook for a specialized hosted virtual try-on workflow
- OpenAI image editing for a two-reference, high-fidelity generation path
- local FASHN VTON v1.5 as the credential-free fallback

If the requested cloud provider is not configured, the application falls back to local inference instead of exposing a broken UI. This makes the same application useful for a polished demonstration, API experimentation, and offline development.

## Genlook Photo Pipeline

When a Genlook key is configured, Drape sends the person and garment as a multipart request to the virtual try-on API.

Before upload, the backend resizes images to satisfy the provider's resolution requirements while preserving aspect ratio. It also sends structured product context describing the garment category and whether the reference is flat-lay or worn by a model.

The service may complete synchronously or return a generation ID. In the asynchronous case, the backend polls until the result is ready, downloads the signed result URL immediately, and stores the image locally.

That download step matters because remote results have limited retention. Drape takes ownership of the completed output rather than depending on a temporary provider URL.

## OpenAI Two-Reference Editing

Drape also includes an OpenAI image-editing path. The person image is the first reference and the garment is the second.

The prompt is designed around two constraints: preserve the person and preserve the product. It asks the model to retain identity, pose, body shape, lighting, background, and framing while also retaining the garment's colour, print, material, logos, seams, sleeves, and hem.

The output dimensions are selected from the person's aspect ratio. Portrait, landscape, and square inputs therefore receive an appropriate generation size instead of being forced into one crop.

This route demonstrates an important fashion-tech lesson: visual quality is not only about generating plausible clothing. Product fidelity and identity preservation are equally important.

## Local FASHN VTON Fallback

Without a cloud key, Drape runs FASHN VTON v1.5 locally. The pretrained model performs maskless virtual try-on in pixel space and supports both model photos and flat-lay garment references.

The local pipeline uses supporting pose and human-parsing components, including DWPose, YOLOX, and a human parser. Model assets are loaded lazily so that the web application can start before the full inference pipeline is needed.

The default quality path uses a balanced 30-step schedule at a native 864 × 576 resolution. Device selection can be controlled through an environment variable, while the runtime detects CUDA availability and reports the active GPU or CPU to the interface.

Automatic low-VRAM behavior makes the application practical on hardware such as a 4 GB GTX 1650, with CPU available as a slower fallback.

## Job Queue and Concurrency Control

Photo inference is not executed directly inside the upload request. The backend creates a job with one of four states: queued, processing, complete, or failed.

A single-worker thread pool processes the expensive task. The frontend polls the job endpoint and updates the user-facing status until an image is available.

Serializing local inference is a sensible choice for a desktop-oriented application. Multiple simultaneous model runs can exhaust limited GPU memory and make every request slower. A one-worker queue trades throughput for stability and predictable resource use.

## Content-Based Caching

Every generated photo does not need to consume another API credit or repeat local inference.

Drape builds a SHA-256 cache key from the person image, garment image, category, photo type, provider, model settings, and prompt version. Identical inputs with identical generation settings resolve to the same cached output.

The cache has two useful effects:

- repeated demonstrations can return almost instantly
- paid provider credits are not spent on duplicate requests

The Genlook path also writes a small recovery record containing the completed remote result URL before the final local image is committed. If the application stops at the wrong moment, it can recover the already-generated result instead of paying for it again.

Images and metadata are saved atomically through temporary files and replacement. This reduces the chance that an interrupted write leaves a corrupt cache entry.

## Live Camera Try-On with Decart

The live path uses Decart Lucy VTON 3.6 over WebRTC. The webcam stream supplies the person, while a garment image acts as the visual reference for continuous transformation.

The permanent Decart API key remains on the backend. The browser requests a short-lived token scoped to the local origin and uses that token to establish the live session. This prevents the long-lived credential from being embedded in JavaScript or exposed through developer tools.

Once connected, the user can change the garment or styling instructions inside the active session without restarting the camera. This preserves the feeling of a live fitting room rather than turning every change into a new upload-and-wait cycle.

The interface also makes the cost boundary explicit: the session continues consuming live-generation credits until the camera is stopped.

## Frontend Experience

The browser layer is intentionally lightweight. It provides a responsive upload flow, category and source controls, job progress, provider and hardware diagnostics, result previews, downloads, and live-camera controls.

The frontend does not need to know how FASHN VTON loads its weights or how Genlook polling works. It submits inputs, observes job state, displays the returned image, and manages the live WebRTC experience.

This division keeps the interface responsive while the backend handles slower and more failure-prone operations.

## Privacy and Security Boundaries

Drape is local-first, but not every mode is fully offline.

Local FASHN inference keeps photo processing on the machine. When Genlook or OpenAI is selected, the person and garment images leave the computer for generation. During Decart live mode, camera frames are transmitted while the session is active.

The design makes several useful security choices:

- permanent API keys stay on the backend
- live browser access uses short-lived, origin-scoped tokens
- uploads are validated before processing
- generated provider results are downloaded to local storage
- configuration is supplied through environment variables
- photo providers fall back safely when credentials are missing

A production internet-facing version would still need authentication, per-user authorization, rate limiting, encrypted persistent storage, and a formal retention policy for uploaded and generated images.

## End-to-End Photo Flow

A typical photo request works like this:

1. The user uploads a person photo and garment reference.
2. The frontend submits both files with category and source metadata.
3. FastAPI validates the images and creates a queued job.
4. The worker computes a provider-specific cache key.
5. A cache hit returns the existing image without new inference.
6. Otherwise, the configured provider generates the try-on result.
7. The backend saves the image atomically and marks the job complete.
8. The frontend detects the completed state and displays a downloadable result.

## Why This Architecture Works

Drape separates product experience from model execution.

The browser owns interaction. FastAPI owns policy and orchestration. Provider adapters own external API details. The local model package owns inference. The queue protects constrained hardware, while content-based caching protects both time and API credits.

Most importantly, photo and live try-on are allowed to use different technologies. The photo path optimizes for fidelity and reproducibility; the live path optimizes for continuity and low latency. The user still experiences both as one coherent fashion application.

## What I Learned While Building It

The first lesson was that the best model is not automatically the best product architecture. A specialized API can produce an excellent demonstration, a local model can protect privacy and remove per-generation cost, and a general image-editing model can offer flexibility. Supporting more than one path made provider selection a backend concern instead of a limitation imposed on the user interface.

The second lesson was that caching is part of the product, not just a performance optimization. During repeated demos, identical generations should not consume more credits or force the audience to watch the same loading state. By including both the input images and generation configuration in the cache key, Drape can reuse a result without accidentally returning an image produced by different settings.

The third lesson was that failure recovery matters around paid inference. A request can finish successfully at the provider while the local process fails before saving the final file. Persisting recovery metadata lets the application retrieve work that has already been paid for.

The fourth lesson was that live video needs its own experience. A photo job can expose progress states and finish after several seconds. A live session must feel continuous, respond to garment changes, protect the permanent credential, and make ongoing credit usage visible. Treating live mode as a first-class path produced a cleaner design than trying to disguise repeated photo requests as video.

Finally, privacy needs to be described precisely. Calling an application local-first does not mean that every operation stays on the device. Drape keeps local inference available, but its cloud photo and live-video modes transmit media to their respective providers. Making that boundary visible is part of building a trustworthy AI product.

## Current Limitations

The current version is designed as a local showcase rather than a multi-user production platform. A larger deployment would need:

- durable jobs backed by a database or queue service
- authentication and tenant isolation
- object storage with explicit retention controls
- automatic retry policies and provider circuit breakers
- concurrency limits per user and per provider
- observability for latency, cache hits, failures, and credit usage
- evaluation datasets for identity and garment fidelity
- clearer consent and deletion controls for personal images
- horizontal workers for higher photo throughput

Virtual try-on also remains an approximation. Difficult poses, occlusion, unusual garments, poor lighting, and low-quality references can still produce unrealistic fits or altered details.

## Conclusion

Drape demonstrates that an AI fashion product is more than a model endpoint. A useful try-on experience needs validation, routing, secure credentials, asynchronous jobs, caching, hardware awareness, privacy boundaries, and a frontend that explains what the system is doing.

By combining specialized cloud generation, a local open-source fallback, and WebRTC-based live transformation, the project supports both polished still images and an interactive fitting-room experience without tying the entire application to one inference provider.

The main idea I am taking forward is simple: AI features become products only when the surrounding architecture makes them reliable, understandable, affordable, and safe enough to use.

The goal is to give a merchant a single place to inspect orders, customers, products, revenue, and sales trends without manually exporting data from Shopify.

At a high level, the application follows this flow:

Shopify Store
     ↓
OAuth Installation + Token Exchange
     ↓
Node.js / Express Backend
     ↓
MySQL Persistence
     ↓
Analytics APIs
     ↓
React Dashboard + Chart.js

In parallel:
Shopify orders/create Webhook
     ↓
HMAC Verification
     ↓
Order Upsert
     ↓
Updated Dashboard Metrics

Shopify Integration Layer

The application begins with the Shopify installation flow. When a merchant installs the app, the backend redirects the store through Shopify OAuth and receives a callback containing an authorization code.

The backend exchanges that code for a store-specific access token and saves the token on the server. The frontend never receives the secret token.

This layer is responsible for:

• starting the OAuth flow
• validating the store domain
• handling the OAuth callback
• exchanging the authorization code
• storing the access token per shop
• registering the store for later API requests

The stored token is then used when the backend calls the Shopify GraphQL Admin API for product data.

Webhook Ingestion Layer

Analytics should update when store activity happens, not only when a user refreshes the app. Xeno Analytics Pro uses Shopify webhooks to receive new order events.

When Shopify sends an orders/create webhook, the Express backend verifies the HMAC signature before trusting the payload. This step confirms that the event was signed with the application secret.

After verification, the order is written to MySQL. Webhook-driven ingestion keeps the local analytics store synchronized without repeatedly downloading every order.

The webhook layer provides:

• real-time order ingestion
• signature verification
• normalization of Shopify payloads
• persistent storage for reporting
• a clean boundary between external events and internal data

Express Backend Layer

The Node.js and Express backend coordinates authentication, Shopify API access, storage, and analytics endpoints.

Its main routes include:

• /auth to begin installation
• /auth/callback to complete OAuth and save the token
• /webhooks/orders/create to ingest signed order events
• /api/products to retrieve the live product catalogue
• /api/orders to return synchronized orders
• /api/customers to aggregate customer history
• /api/analytics to calculate dashboard metrics

Keeping these responsibilities behind the backend prevents sensitive credentials from reaching the browser and gives the frontend a consistent API even when the underlying data comes from different sources.

Data Persistence Layer

The application uses MySQL hosted on Aiven with SSL connections.

Two core data groups drive the system:

• shop tokens, which associate a store with its server-side access token
• shop orders, which contain webhook-sourced order information

The backend uses SQL queries and aggregations to produce merchant-facing views. Customer totals can be calculated by grouping orders by customer, while time-based revenue trends can be created by grouping orders by date.

Persisting normalized order data locally has two benefits. Dashboard queries do not need to call Shopify for every chart, and historical analytics remain available through a database designed for aggregation.

Product Retrieval Layer

Products are fetched through Shopify's GraphQL Admin API rather than being copied into the local order tables.

The backend reads the correct access token for the current shop, sends a GraphQL query to Shopify, and returns a simplified product response to the React application.

This is a useful split: order events are stored for analytics, while the current catalogue remains sourced from Shopify.

Frontend Layer

The frontend is built with React 18, React Router, Vite, and Chart.js. It provides separate views for the dashboard, analytics, products, orders, and customers.

The interface is responsible for showing:

• live KPI cards
• synchronized order tables
• customer order history and total spend
• product catalogue data
• revenue-per-customer charts
• orders-per-day trends

Custom hooks handle the current shop parameter and reusable API requests. A layout component provides navigation and guards pages that require shop context.

Analytics Layer

The analytics layer converts raw order records into information a merchant can act on.

For example, the API can group orders by day to show sales volume over time, or group by customer to identify high-value buyers. Chart.js then turns those aggregates into bar and line charts.

The separation is intentional: MySQL performs the grouping and aggregation, the Express API shapes the response, and React focuses on presentation.

End-To-End Flow

A typical store session works like this:

1. A merchant installs the application from Shopify.
2. Shopify OAuth returns an authorization code to the backend.
3. The backend exchanges the code and stores the shop token.
4. Shopify sends new-order webhooks as purchases occur.
5. The backend verifies each HMAC signature and saves the order.
6. The dashboard requests orders, customers, and analytics data.
7. MySQL aggregations produce revenue and activity metrics.
8. React and Chart.js display the results.
9. Product views are populated through server-side GraphQL calls to Shopify.

Security Design

The project includes several important security boundaries:

• webhook HMAC verification before payload processing
• server-side access-token storage
• SSL for database connections
• environment variables for credentials
• no Shopify secrets exposed to the frontend

For a production release, the OAuth flow should also enforce state or nonce validation, sessions should be hardened, and authorization should be checked for every shop-scoped API request.

Why This Architecture Works

The application separates external integration, data ingestion, analytics, and presentation.

Shopify remains the source of truth for the store and product catalogue. Webhooks deliver changes efficiently. MySQL provides queryable history. Express protects credentials and exposes stable APIs. React presents the results without needing to understand Shopify authentication details.

This separation also makes the system easier to extend. New webhook topics can be added without redesigning the dashboard, and new charts can reuse the existing analytics endpoints.

Current Limitations

The current version is a complete prototype, but a larger production deployment would still need:

• durable retry and dead-letter handling for failed webhooks
• idempotency guarantees for duplicate webhook delivery
• background jobs for historical order backfills
• pagination for large stores
• token encryption at rest
• stronger session and tenant authorization
• monitoring for webhook failures and Shopify API limits
• automated data-retention and uninstall cleanup

These are natural next steps when moving from a functional embedded app to a multi-store production service.

Conclusion

Xeno Analytics Pro connects Shopify's OAuth, GraphQL, and webhook systems to a secure Node.js backend, an SSL-enabled MySQL database, and a React analytics dashboard.

The architecture demonstrates how event-driven commerce data can move from a store action to a persistent record and finally to a useful visualization. It is a practical example of integrating an external platform while keeping credentials, data processing, and UI concerns cleanly separated.

If you found the project useful, consider giving it a star.

https://github.com/Kushal1213/shopify

The system spans three surfaces: a Chrome extension that works inside the messaging page, an Express API that performs validation and AI orchestration, and a Next.js dashboard for testing and using the same capabilities outside the extension.

Messaging Page
      ↓
Chrome Content Script
      ↓
Background Service Worker
      ↓
Node.js / Express API
  ↙                 ↘
Local Rules       OpenAI Analysis
      ↘             ↙
Summary, Replies, Scam Risk, Sentiment
      ↓
Injected Side Panel Or Next.js Dashboard

Browser Extension Layer

The extension uses Chrome Manifest V3. A content script observes the messaging page, while a background service worker relays requests to the API.

The content layer is responsible for:

• detecting newly added messages with MutationObserver
• blurring text that appears to contain sensitive information
• temporarily revealing a blurred message on click
• collecting a bounded set of visible messages
• displaying summaries, reply suggestions, and warnings
• rendering a slide-in assistant panel

Using MutationObserver is more efficient and responsive than continuously scanning the document on a timer. It lets WhisperShield react when the page changes.

Sensitive-Text Protection Layer

Smart Blur uses local pattern matching for content such as OTPs, passwords, bank details, CVVs, and account numbers.

This feature is deliberately local. Obvious sensitive patterns can be hidden without sending the text to an AI provider. The user can reveal a message temporarily when needed.

Rule-based detection is fast and predictable, but it can produce both false positives and false negatives. The feature should be treated as an additional privacy aid rather than a guarantee that all sensitive information will be recognized.

Background Service Worker

The Manifest V3 service worker acts as a boundary between the injected page logic and the external API.

It receives requests from the content script, calls the configured backend, and returns structured results. This design keeps API communication and extension lifecycle handling out of the page-facing code.

The service worker also avoids placing provider credentials in the extension. The browser calls the project's backend, and the backend owns model access.

Express API Layer

The Node.js and Express backend exposes endpoints for:

• summarization
• three-tone reply generation
• scam detection
• sentiment analysis
• service health

Security middleware includes Helmet, request rate limiting, configured CORS, and input validation. Errors are returned to the caller instead of being hidden behind a hardcoded response.

When an OpenAI key is available, the API uses GPT-4o-mini. A mock fallback keeps the product demonstrable without a provider key.

Scam Detection Layer

Scam detection combines deterministic signals with AI analysis.

Local rules check for patterns commonly associated with phishing or financial scams. The AI layer can consider broader context across the recent conversation. A warning banner is shown when the combined result crosses the configured confidence boundary.

This hybrid approach is useful because rules are consistent for known patterns, while the model can identify suspicious language that does not match an exact expression.

However, the result must remain advisory. It should not claim that a message is safe simply because no pattern or model signal was found.

Summarization And Reply Layer

The extension can collect a bounded window of recent visible messages and send it for summarization. Keeping the window small reduces latency and prevents the system from processing more conversation than necessary.

Smart Reply returns three options:

• professional
• friendly
• brief

The user chooses and copies a reply. The system does not send the message automatically, which preserves an important human-review step.

Next.js Dashboard Layer

The web dashboard provides a standalone interface to the same API. It is built with Next.js, TypeScript, and Tailwind CSS.

This surface is useful for testing the endpoints, demonstrating the product without installing the extension, and giving users access to analysis features outside the messaging page.

Keeping the dashboard and extension on one API also prevents the AI logic from being duplicated across clients.

End-To-End Flow

A typical scam check works like this:

1. The content script collects the most recent visible messages.
2. The extension sends the text to its background service worker.
3. The service worker calls the Express scam-detection endpoint.
4. The backend validates the request.
5. Regex patterns identify known warning signals.
6. The AI layer evaluates the broader conversation context.
7. The API returns a risk assessment and explanation.
8. The content script displays a warning when the result is sufficiently suspicious.

Smart Blur follows a shorter local path and does not require the API for basic pattern detection.

Privacy And Security Boundaries

WhisperShield handles message content, so privacy boundaries are central to the architecture.

Useful protections already include local blurring, bounded message collection, server-side model credentials, rate limiting, input validation, and explicit user selection before copying a reply.

A production version should go further with:

• clear disclosure before message text leaves the browser
• encryption in transit and strict log redaction
• no retention of message content by default
• allowlisted page origins
• minimal extension permissions
• prompt-injection defenses for message content
• provider data-retention documentation
• user controls for each AI feature

Why This Architecture Works

The extension handles page context and local privacy actions. The service worker handles browser-to-server communication. Express owns validation and AI orchestration. Next.js provides a separate product surface.

This separation keeps provider secrets out of the browser and allows deterministic privacy features to work even when the AI service is unavailable.

Current Limitations

The current prototype still depends on the structure of the target messaging page. DOM changes can break selectors or panel placement.

Other limitations include:

• rule-based sensitive-data detection is incomplete
• model-based scam detection can be wrong
• localhost API configuration is intended for development
• message context is limited to visible content
• the extension is not a substitute for platform security features
• mock mode is useful for demos but cannot validate real model behavior

Conclusion

WhisperShield demonstrates how local browser protections and AI-assisted communication tools can coexist in one architecture.

The key design is a layered workflow: keep simple privacy actions local, send only user-requested analysis through the service worker and API, and leave the final communication decision with the user.

If you found the project useful, consider giving it a star.

https://github.com/Kushal1213/Whispershield

At a high level, Veyra connects voice input, AI retrieval, realtime analytics, and agent assistance into one workflow.

Customer Voice
     ↓
Speech-to-Text
     ↓
Node.js API Gateway
     ↓
RAG Knowledge Service
     ↓
AI Response + Sources
     ↓
Voice Reply + Agent Dashboard

In parallel:
Conversation Transcript
     ↓
Realtime Insights Service
     ↓
Nudges, Risk Signals, Escalation Alerts

Frontend Layer

The frontend is built with React, Vite, Tailwind CSS, Framer Motion, and Recharts. It provides the main user interface for the voice demo, live transcript, insights dashboard, analytics views, and knowledge hub.

In the browser demo, Veyra uses the browser’s speech recognition API to convert customer speech into text. It also uses browser speech synthesis to speak the assistant’s answer back to the user. This makes the prototype easy to run without depending on a full telephony setup.

The frontend is responsible for showing:

• live conversation transcript

• assistant responses

• detected customer signals

• suggested agent nudges

• call/session state

• analytics-style dashboards

API Gateway Layer

The Node.js Express API gateway is the central coordination layer. It receives voice queries from the frontend, manages call sessions, sends conversation turns to the RAG service, and broadcasts realtime updates using Socket.IO.

This gateway keeps track of active sessions in memory. For each call, it stores things like customer name, intent, loan amount, income, frustration level, missing fields, and conversation turns.

The gateway has three major responsibilities:

• receive and validate user speech text

• call the RAG service to get grounded answers

• send transcript and insight updates to the frontend in realtime

It also supports optional integration points such as Vapi custom LLM webhooks and Deepgram transcript routes, although the main browser demo uses browser speech recognition.

RAG Knowledge Layer

The RAG service is built with Python FastAPI. RAG stands for Retrieval-Augmented Generation. In Veyra, this means the system does not answer only from a generic model. Instead, it searches a knowledge base first and then uses the retrieved information to form an answer.

Veyra uses Google Gemini embeddings to convert text into vectors. These vectors are stored in FAISS, a fast similarity search engine. When a customer asks a question, the RAG service embeds the query, searches FAISS for the most relevant chunks, and returns an answer with source information.

For example, if a customer asks about a home loan, Veyra can retrieve relevant policy chunks about loan eligibility, tenure, rates, or documents and use them in the response.

The RAG layer includes:

• query embedding

• vector similarity search

• market-aware retrieval

• answer synthesis

• source citation

• fallback handling when no strong match is found

Knowledge Ingestion Layer

Before the RAG service can answer questions, documents need to be added to the knowledge base. That is handled by the ingestion service, also built with Python FastAPI.

The ingestion service can process:

• PDF files

• website URLs

• raw text

It extracts text, breaks it into chunks, generates Gemini embeddings, and stores the vectors in FAISS along with metadata such as source, title, page, market, category, and version.

This design allows Veyra to be updated with new policy documents, FAQs, product brochures, compliance notes, or regional banking information.

Realtime Insights Layer

The realtime insights service analyzes conversation text while the call is happening. It is designed to help the agent notice important moments during the conversation.

This service uses rule-based detection with keyword patterns, regex rules, and conversation heuristics. It can identify signals such as:

• customer frustration

• payment difficulty

• buying intent

• missed cross-sell opportunity

• compliance disclosure gap

• need for human escalation

For example, if a customer says they are worried about monthly payments, the system may detect payment difficulty. If an agent discusses rates or fees without a required disclosure, it may raise a compliance nudge.

These nudges are sent back to the frontend through the gateway so the agent can respond while the conversation is still active.

CRM And Session Layer

Veyra also includes simple CRM-style functionality. It can create mock leads, callbacks, and customer records during a conversation. In the current prototype, this data is stored in memory using JavaScript maps.

This is useful for demonstrating how voice intelligence can connect to business workflows. For production, this layer could be connected to a real CRM like Salesforce, HubSpot, Zoho, or an internal banking CRM.

End-to-End Flow

A typical Veyra conversation works like this:

1. A customer speaks into the browser.

2. The frontend converts speech to text.

3. The text is sent to the Node.js API gateway.

4. The gateway stores the user turn in the session.

5. The gateway sends the query to the RAG service.

6. The RAG service retrieves relevant knowledge from FAISS.

7. The system returns an answer with sources.

8. The frontend displays and speaks the answer.

9. In parallel, the realtime insights service checks the conversation for nudges.

10. The dashboard updates with live transcript, signals, and recommendations.

Why This Architecture Works

Veyra uses separate services for separate responsibilities. The frontend handles the user experience, the gateway manages realtime coordination, the RAG service handles knowledge retrieval, the ingestion service manages document processing, and the insights service focuses on live call intelligence.

This separation makes the system easier to explain, test, and improve. For example, the RAG service can be upgraded without changing the frontend. The insights engine can become more advanced without rewriting the voice interface. A real CRM can be added later without changing the core conversation flow.

Current Prototype Limitations

The current version is a strong prototype, but it is not yet a full production system. Some parts are intentionally simple:

• session data is stored in memory

• CRM data is mock data

• realtime insights are rule-based

• browser speech recognition depends on browser support

• some dashboard analytics values are demo-style

• escalation events are emitted, but not connected to real call transfer

• production authentication, database storage, observability, and security hardening would still need to be added

Conclusion

Veyra’s architecture combines voice interaction, RAG-based knowledge retrieval, realtime insights, and agent assistance into one practical system. It shows how AI can support customer-facing teams by helping them answer questions faster, detect customer intent, reduce missed opportunities, and improve compliance during live conversations.

Consider giving a STAR if u find it helpful.

https://github.com/Kushal1213/darwinxvapi