--- url: https://yagneshlp.pmfolio.xyz/enterprise-conversational-ai-platform-reduced-drop-86352f title: Yagnesh L Pazhaniyappan — Reduced platform drop-offs by 25% through hierarchical intent mapping and HITL fallback strategies, while improving language detection speed from 3s+ to 100ms. description: Our incumbent NLP-based chatbot suffered from slow language detection (3+ seconds), high drop-off rates when users hit low-confidence or unsupported intents, an type: pm-case-study author: Yagnesh L Pazhaniyappan --- # Reduced platform drop-offs by 25% through hierarchical intent mapping and HITL fallback strategies, while improving language detection speed from 3s+ to 100ms. *IDFC First Bank · Associate Product Manager - Enterprise Conversational AI Platform · July 2025 – Present (Phase 1: ~6 months) · Cross-functional team of 12* ## Key Outcomes - Reduced drop-offs by 25% through HITL fallback and escalation strategies for low-confidence intents - Improved language detection speed from 3s+ to 100ms+ via multi-model ensemble system - Improved platform latency from 20+ sec to <5 sec through prompt and context engineering - Delivered 4/4 phase 1 modules on-time for go-live - Reduced system false negative rate to <1% through automated red-teaming system ## 01 — The Problem Our incumbent NLP-based chatbot suffered from slow language detection (3+ seconds), high drop-off rates when users hit low-confidence or unsupported intents, and 20+ second response latencies that frustrated customers. The probabilistic drift of LLMs made intent classification unreliable at scale, and we lacked the guardrails necessary to protect PII and filter toxic content before go-live. With 40+ products across 7 systems and growing user expectations, we needed a resilient, sub-second conversational AI platform that could handle multi-turn dialogue, personalization, and compliance from day one. > For a digital-first bank like IDFC First, the conversational AI platform was the front door to customer engagement and cross-sell opportunities. High drop-off rates meant lost revenue and poor NPS. Slow, generic responses eroded trust. And without robust safety and compliance guardrails, we risked regulatory penalties and reputational damage. Delivering a fast, personalized, and safe AI experience was critical to activating new business conversions, improving customer retention, and differentiating in a competitive fintech market. ## 03 — Solution I owned the AI platform development strategy and organized platform steerco discussions to align stakeholders. I designed a sub-second multi-model ensemble system for language detection and introduced hierarchical intent mapping to reduce disambiguation. To handle edge cases, I devised HITL (human-in-the-loop) fallback and escalation strategies for low-confidence, risky, or unsupported intents. I developed an 11-layered context engine to enable true multi-turn conversation and integrated 8 fragmented customer propensity models into an evolving "cDNA" for 6-layered personalization. I designed an agentic guardrail system with a closed feedback loop for toxicity filtering, PII protection, and response grounding, and deployed an automated red-teaming pipeline integrated into our test suite. I balanced cost, latency, and response quality through prompt and context engineering, and established baseline metrics from the incumbent NLP bot to track LLM impact via 8 self-serve dashboards. I also prepared a launch playbook to align stakeholders for cascading sign-offs. ### Key decisions & trade-offs Chose a multi-model ensemble over a single large model for language detection to optimize for speed and cost. Introduced hierarchical intent mapping instead of flat classification to improve scalability and reduce disambiguation rates. Opted for HITL fallback strategies rather than fully automated handling of low-confidence intents to balance user experience with safety. Prioritized prompt and context engineering over model fine-tuning to achieve sub-5s latency while controlling costs. Researched multiple guardrail providers and drove technical feasibility meetings to select and procure the best solution rather than building in-house. Deployed automated regression testing and red-teaming pipelines early to de-risk go-live. ## 04 — Results | Metric | Before | After | Delta | Timeframe | |--------|--------|-------|-------|-----------| | Platform drop-offs | Baseline (incumbent NLP bot) | 25% reduction | -25% | Phase 1 | | Language detection speed | 3s+ | 100ms+ | ~97% improvement | Phase 1 | | Platform latency | 20+ sec | <5 sec | >75% improvement | Phase 1 | | System false negative rate | Not specified | <1% | N/A | Post automated red-teaming deployment | | Phase 1 modules delivered on-time | | 4/4 | | Phase 1 go-live | | Customer propensity models integrated | | 8 | | Phase 1 | | Personalization layers enabled | | 6 | | Phase 1 | | Context engine layers | | 11 | | Phase 1 | | Products covered by automated regression testing | | 40+ | | Phase 1 | | Systems integrated for automated testing | | 7 | | Phase 1 | | Self-serve dashboards built for impact tracking | | 8 | | Phase 1 | ## 05 — Challenges & Learnings ### Challenges built the product ## Skills Demonstrated LLM Orchestration · Agentic Systems · Multi-model Ensembles · Platform Architecture · LLMOps Strategy · Guardrail Design · Prompt Engineering · Context Engineering · HITL Workflow Design · Intent Classification · Memory & Personalization · Observability & Monitoring · Automated Testing · Red-Teaming · Stakeholder Alignment · Product Strategy · A/B Testing · Python · Databricks · Langfuse --- *Back to Yagnesh L Pazhaniyappan's folio — https://yagneshlp.pmfolio.xyz* *Built with pmFolio — https://pmfolio.xyz*