Software Development Engineer — DynamoDB Indexing Amazon DynamoDB is a serverless, distributed NoSQL database that delivers single-digit millisecond performance at virtually any scale.
DynamoDB's indexing infrastructure powers Global Secondary Indexes (GSIs), enabling customers to efficiently query their data across alternate access patterns without managing additional infrastructure.
We are looking for a Software Development Engineer II to help design, build, and operate the distributed indexing infrastructure at the heart of DynamoDB.
You will work on problems at the intersection of distributed systems, stream processing, storage engines, and large-scale data replication — ensuring that secondary indexes remain consistent, performant, and highly available as the underlying table data changes continuously.
The role includes designing and evolving the index replication pipeline that propagates mutations from base tables to secondary indexes in near-real time;
building index lifecycle mechanisms that handle creation, backfill, compaction, rebalancing, and recovery across partitioned datasets at massive scale;
and optimizing the storage, query, and cost efficiency of secondary indexes across all DynamoDB partitions and regions.
You will help determine how indexes are created, maintained, scaled, and recovered while the underlying DynamoDB data continues to change at millions of requests per second.
You will also shape how indexing infrastructure integrates with DynamoDB's partitioning, replication, admission control, and query-processing systems — delivering predictable performance across latency, throughput, freshness, availability, and cost.
This is an opportunity to shape the future of DynamoDB's indexing capabilities — from foundational architecture through customer launch and operation at DynamoDB scale.
Key Job Responsibilities - Design and implement distributed index replication infrastructure that propagates base-table mutations to Global Secondary Indexes with low latency and high fidelity.
- Build and optimize the Log Propagator pipeline, including log streaming, checkpointing, session management, admission control, and failure recovery across thousands of partitions.
- Develop index lifecycle mechanisms covering index creation, backfill from existing table data, incremental updates, compaction, partition splits and merges, rebalancing, and deletion — all while the table continues to serve live traffic.
- Ensure that secondary indexes remain consistent with the base table under concurrent writes, partition movements, leader elections, and failure scenarios, with well-defined freshness and consistency semantics.
- Integrate indexing infrastructure with DynamoDB's distributed storage, partitioning, replication, and query-processing systems, including StorageNode, Partition Manager, and the control plane.
- Design and build mechanisms for index availability and recovery, including handling degraded partitions, stuck replication sessions, and automated remediation of index lag.
- Optimize tradeoffs among replication latency, write amplification, storage efficiency, throughput, memory usage, and operational cost across the indexing fleet.
- Build observability, alarming, and operational tooling for the indexing pipeline — enabling rapid diagnosis and resolution of replication lag, backfill progress, and partition-level health.
- Drive operational excellence across the indexing fleet, including patching, deployment safety, capacity planning, and on-call processes for a Tier-1 service.