Post Syndicated from Narendra Kumar original https://aws.amazon.com/blogs/big-data/how-razorpay-built-real-time-anomaly-detection-with-amazon-msk/
When you process over 500 million transactions per month, every second of undetected anomaly means failed payments, lost revenue, and eroded merchant trust. Static monitoring thresholds that worked for thousands of merchants collapse at the scale of millions, and the cost of missed detection compounds exponentially.
In this post, we explore Razorpay’s anomaly detection and alerting platform (ADA) architecture using Amazon Managed Streaming for Apache Kafka (Amazon MSK) and other AWS services. According to Razorpay the system detects transaction anomalies in under 30 seconds, supports thousands of merchant-level alerts, and reduced monitoring costs by approximately 80 percent. The platform maintains 99.99 percent uptime for over 500 million transactions per month.
Founded in 2014, Razorpay has become one of India’s largest full-stack financial solutions companies, powering payments, banking, and business growth for over 10 million businesses. With offerings spanning payment gateway, RazorpayX for business banking, and Razorpay Capital for lending, the company processes over 500 million transactions per month across payments, payroll, banking, and cross-border services.
At this scale, Razorpay’s data platform processes more than 5 billion events daily. Every transaction, settlement, and disbursement generates events that must be monitored in real time for anomalies. These range from systemic degradations and latency regressions to card-testing fraud attacks and velocity abuse at the merchant level.
For a regulated payments platform, undetected anomalies carry consequences far beyond technical metrics. A missed fraud pattern can mean direct financial losses running into millions of rupees. It can also bring regulatory scrutiny from the Reserve Bank of India and irreversible damage to merchant confidence, the foundation of Razorpay’s business. Razorpay needed real-time anomaly detection, but the existing infrastructure couldn’t keep pace with the company’s growth.
The problem: When static thresholds can’t keep up with scale
As Razorpay scaled from thousands to millions of merchants, the existing monitoring infrastructure hit critical limitations across four dimensions.
Anomaly blind spots
Systemic degradations, latency regressions, and success-rate drops went undetected until customers complained. By the time a human operator noticed a 15 percent drop in payment success rates for a specific gateway-merchant combination, thousands of transactions had already failed.
Fraud at velocity
Card-testing activity, velocity abuse, and geo-anomalies at the merchant level required sub-minute detection. Unauthorized users could generate hundreds of micro-transactions in seconds. Traditional batch detection was too slow to prevent damage.
Static thresholds don’t scale
The existing tooling relied on static thresholds with no adaptive baselines. This created a painful dilemma: set thresholds too tight and drown in false alarms (alert fatigue), or set them too loose and miss real incidents.
High cardinality equals high cost
Monitoring thousands of merchants individually on the previous architecture cost approximately $500K per year: $250K in licensing fees plus $250K in infrastructure, with fundamental scalability limits. ThirdEye queried a 21-day lookback at query time, enforcing a 1–2 minute service level agreement (SLA) minimum. The system was not designed for thousands of concurrent merchant-level alerts, a limitation confirmed by the vendor.
Solution overview: ADA: Anomaly detection and alerting
Razorpay built ADA (Anomaly Detection and Alerting), a configurable, multi-tenant engine for real-time anomaly detection and fraud prevention. The platform’s design centers on three core principles that address the limitations of the previous architecture.
First, ADA is declarative: users express what to detect, not how. A single domain-specific language (AdaDSL) drives both batch and streaming execution, eliminating the need for engineers to write custom detection code for each new alert. Second, ADA is adaptive. Dynamic baselines incorporate calendar-aware patterns (day-of-week, time-of-day, holiday adjustments) and machine learning (ML)-compatible thresholds that replace brittle static rules. Third, ADA is inherently multi-tenant: Payments, Payroll, and Banking each operate with isolated detection logic while sharing underlying infrastructure. This design removes the need to maintain separate monitoring stacks per business unit.
Amazon MSK serves as the event backbone of ADA, ingesting transaction events, distributing detection rules, and connecting the components of the real-time pipeline.
Architecture: Amazon MSK as the streaming backbone
The ADA architecture positions Amazon MSK as the core integration layer connecting event producers to detection engines and alert consumers. Payment authorization, settlement, and disbursement events flow through Kafka topics managed by Amazon MSK. With Razorpay processing over 500 million transactions per month and 5 billion events daily, the ingestion layer must absorb high throughput with zero data loss.
High-throughput event ingestion
The architecture uses tenant-partitioned topics. Each business unit (Payments, Payroll, Banking) publishes to logically isolated topics while sharing physical infrastructure. This design supports independent consumer groups per tenant with predictable throughput guarantees.
Change Data Capture (CDC) events from Razorpay’s core transactional databases (Amazon Aurora MySQL-Compatible Edition) flow through Debezium and a Kafka Streams-based Harvester service into Amazon MSK. Application events from payment services also publish directly to Amazon MSK topics via native Kafka producers.
Why Amazon MSK as the backbone
Amazon MSK serves as the architectural backbone of ADA, fulfilling four critical functions that together support reliable, real-time anomaly detection at scale. At the ingestion layer, Amazon MSK absorbs the full stream of transaction events with three-replica durability. If downstream consumers experience an outage, they resume from their last committed offset without data loss. Beyond ingestion, Amazon MSK is the event distribution backbone of detection rules. AdaDSL definitions authored by domain experts are serialized and published to a dedicated Kafka snapshot topic, which Flink jobs consume as a broadcast stream.
This delivers hot-reloadable rule updates without pipeline restarts, a critical capability when detection logic must evolve daily. Amazon MSK further supports tenant isolation at the topic level. Payments, Payroll, and Banking events flow through isolated topic partitions that support independent scaling and consumer group management per business unit. Finally, Amazon MSK fully decouples event producers from detection consumers, meaning new detection logic can be deployed, scaled, or rolled back without touching production payment flows.
Real-time stream processing with Apache Flink
Apache Flink acts as the stateful stream processing engine between Amazon MSK and the detection/alerting layer. The Flink pipeline implements five key stages:
- Kafka Source (tenant-partitioned topics) – Consumes events from Amazon MSK with exactly-once semantics using Flink’s Kafka connector.
- Event-Time Assignment + Watermarking – Assigns event timestamps and generates watermarks with a late-arrival tolerance of 2× the window size.
KeyBy(tenant_id,entity_key) + Windowed Aggregation – Partitions the stream by tenant and merchant, then computes windowed aggregates (success rates, latencies, transaction volumes).- Async I/O – Baseline Fetch from ClickHouse. Non-blocking lookups against pre-computed baselines stored in ClickHouse, supporting 1,024 concurrent requests.
- Rule Evaluation (threshold / ML / CEP) – Evaluates AdaDSL rules against the enriched stream. This includes Complex Event Processing (CEP) patterns for sequence detection (for example, five consecutive declines followed by a success, a signature of card-testing fraud).
The pipeline outputs to three sinks:
anomalies_fctto ClickHouse for anomaly persistence and historical analysis.- Alert Gateway to Slack/PagerDuty for immediate notification.
windows_fctfor reconciliation against batch baselines.
AdaDSL: Declarative detection at scale
AdaDSL abstracts detection logic into human-readable declarations that platform engineers and domain experts can author without understanding the underlying execution mechanics. A single definition compiles to both a ClickHouse Materialized View selector and a Flink CEP pattern, supporting consistent detection semantics across batch and streaming modes.
AdaDSL updates are distributed via the Amazon MSK snapshot topic. When an engineer modifies a rule, it’s serialized to Kafka and consumed by Flink as a broadcast state update. The change propagates to all running pipeline instances without redeployment. This is an important architectural advantage: the detection logic evolves independently of the infrastructure.
Reliability and fault tolerance
The architecture delivers 99.99 percent availability through multiple layers of resilience:
- Amazon MSK is deployed across three Availability Zones with
replication.factor=3andmin.insync.replicas=2, paired with producer-sideacks=all. No single broker failure causes data loss or ingestion interruption, because the durability guarantee depends on all three settings working together. Combined with configurable retention policies, Amazon MSK provides a meaningful replay window for consumer recovery. - Flink checkpointing to Amazon Simple Storage Service (Amazon S3) provides exactly-once processing semantics. If a Flink task fails, the job manager restores from the latest checkpoint and resumes processing from the corresponding Kafka offsets. No events are lost or duplicated.
- Idempotent sinks: Dedupe keys (
tenant:AdaDSL:version:entity:window_start) prevent reprocessed events from creating duplicate anomaly records or alerts. - Event-time watermarks: 2× window tolerance handles late-arriving events gracefully, supporting detection accuracy even under network delays.
Results and business impact
The migration from Pinot + ThirdEye to ADA on Amazon MSK and Apache Flink delivered measurable improvements. The platform achieved approximately 80 percent cost reduction compared to the previous architecture while maintaining a 99.99 percent uptime SLA. Anomaly detection latency in streaming mode is under 30 seconds, and the system processes over 5 billion events daily. It supports thousands of concurrent merchant-level alerts with full multi-tenant isolation across Payments, Payroll, and Banking.
Operational improvements
The ADA platform delivered significant operational improvements across detection accuracy, speed, and team autonomy:
- Alert fatigue removed – Adaptive baselines with calendar-aware patterns (day-of-week, time-of-day, holiday adjustments) reduced false positives by over 90 percent compared to static thresholds.
- Mean time to detection reduced from minutes to seconds – Sub-30-second streaming detection replaced batch detection cycles that previously required 1–2 minutes minimum.
- Self-service detection – Domain experts in Payments, Payroll, and Banking teams author their own AdaDSL rules without requiring platform engineering involvement.
- Unified platform – One system for anomaly detection, fraud detection, alert routing, and reconciliation across all business units.
Key learnings and best practices
Throughout the design and implementation of ADA, Razorpay identified several architectural principles that proved essential at scale:
1. Separate rule definition from execution
A declarative DSL lets domain experts define detection logic while the platform decides batch or streaming execution. This separation allowed Razorpay to scale the number of active detection rules from dozens to thousands without proportional engineering effort.
2. Use Amazon MSK as the unifying backbone
Kafka’s publish-subscribe model naturally decouples event producers from detection consumers. Beyond basic event transport, Amazon MSK serves as the distribution mechanism for rule updates (broadcast state), tenant isolation (topic partitioning), and fault tolerance (offset-based replay). Investing in the streaming backbone early benefited every subsequent design choice.
3. Combine Flink streaming with ClickHouse baselines
Flink excels at sub-minute, stateful detection. ClickHouse excels at deterministic baseline computation and historical context. Rather than forcing one engine to do both, the hybrid architecture plays to each engine’s strengths.
4. Design for multi-tenancy from day one
Shared infrastructure with tenant isolation (row-level security in ClickHouse, scoped topics in Amazon MSK, tenant-partitioned Flink pipelines) keeps operational costs low while serving multiple business units with independent SLAs.
5. Build for extensibility
A plugin-compatible architecture allows ML models (ETS/Prophet for forecasting), CEP patterns (Flink CEP for sequence detection), and custom root cause analysis (RCA) strategies to be added without platform-level changes. Razorpay’s roadmap includes large language model (LLM)-assisted RCA and autonomous AdaDSL generation.
Conclusion
Razorpay transformed its anomaly detection from static-threshold monitoring on Pinot + ThirdEye to an adaptive, real-time system on Amazon MSK and Apache Flink.
This reflects a pattern increasingly common among high-scale FinTech platforms: a reliable, high-throughput streaming layer is not an optimization. It’s a prerequisite for operating payment infrastructure at scale.
Amazon MSK forms the backbone that allows Razorpay to ingest 5 billion events daily and distribute detection rules in real time. It also isolates multiple business units on shared infrastructure and provides exactly-once processing guarantees for financial transaction monitoring. Apache Flink transforms those raw event streams into sub-30-second anomaly detection with CEP-based fraud pattern matching.
For platform engineers building real-time monitoring for financial services, the takeaway is clear. Invest in the streaming backbone early, design for declarative extensibility, and let managed services absorb the operational complexity of distributed stream processing.
If you’re building real-time monitoring for a high-throughput transactional system, start by evaluating your current architecture against the four limitations described in this post. These are anomaly blind spots, detection latency for fraud, static threshold scalability, and cost at high cardinality. From there, consider whether a declarative detection layer (separating rule definition from execution) could accelerate your team’s ability to ship new alerts without infrastructure changes. For a hands-on starting point, explore the Amazon MSK Labs workshop.
To learn more about Amazon MSK, visit the documentation.
