OpenSearchCon India 2026

Large-scale ingestion can significantly impact OpenSearch cluster stability, query latency, and operational efficiency. Traditional bulk APIs and Spark/Hadoop connectors rely on online indexing, where clusters must handle indexing and queries simultaneously—creating resource contention and long ingestion windows.

This session introduces an Offline Batch Ingestion framework that builds OpenSearch-compatible Lucene segments outside the cluster using distributed Spark executors. The system leverages OpenSearch’s native indexing engine and snapshot/restore APIs to deploy indexes atomically with zero production impact.

We’ll cover the architecture (distributed indexing, shard merge, snapshot generation), technical implementation using OpenSearch Engine APIs, production learnings, scalability characteristics, and future extensions such as vector indexing. This approach decouples indexing compute from serving clusters, enabling faster ingestion, safer reindexing, and improved operational resilience.