OpenAI Habitat: Scaling Storage for 1B ChatGPT Users

Habitat is OpenAI's internal online storage platform built to serve ChatGPT and its other products. In short: Habitat started as a client library, hit scaling limits, was carved out into an independent service, and in 2026 was rewritten from Python to Rust — this article walks through that journey, based on Part 1 of OpenAI's engineering blog post.

The original post, "Rapidly scaling online storage to serve over 1 billion ChatGPT users," was published on September 11, 2026 by Jon Lee, Chaomin Yu, and Ben Ries. It's Part 1 of a two-part series on how the storage layer behind ChatGPT's 1-billion-plus weekly users hit its limits and was rebuilt. OpenAI has also been shipping developer-facing features like the OpenAI Agents API, but this piece is about the infrastructure behind the scenes. Below, we stick strictly to what the original post states.

Habitat sits between every client service and the actual storage resources. The structure is a three-layer stack: client services → Habitat → storage resources (such as Azure Cosmos DB). Client teams only need to work against Habitat's simple API; they don't need to know how or where data is actually distributed underneath. Primary source: OpenAI's official post.

Habitat began as a client library embedded into each service, but by mid-2025 that model had hit its limits. One incident captures why: during a migration to regionally distributed Cosmos DB accounts, rolling out a new client to dozens of services took days on its own, adding shadowing for safety took more days, and fixing bugs found along the way took still more days. In the middle of this careful rollout, one team — for reasons unrelated to the migration — rolled back to an old, buggy client version, and triggered exactly the kind of incident the whole process was designed to avoid.

That experience pushed OpenAI to pull Habitat out as an independent service rather than a library. As a service, deployment, observability, and improvements all funnel through a single control point instead of needing to be distributed and coordinated across dozens of teams. It also became a security chokepoint: ACLs, audit logging, and access restrictions to storage can now be enforced centrally, guarding against unauthorized access from external actors, internal actors, and agentic actors alike.

Habitat's service itself is written in Python — a choice OpenAI describes as deliberate technical debt. The team knew it wouldn't hold up at 100x the current scale, but bet that continued progress in Codex and GPT models would make a future rewrite easier, and chose to push forward in Python anyway. Reaching a peak of over 20 million requests per second took four specific tricks.

(a) Measuring asyncio event loop delay — asyncio gives you concurrency, not CPU parallelism, because of the GIL. But Habitat does plenty of CPU-heavy work: routing, compression, encryption, checksums, health checks, shadowing, and hedging. In traces of slow (p99+) requests, the pattern was clear: the downstream database was responding quickly, but the coroutine was stuck waiting to be rescheduled. To quantify this, OpenAI scheduled a background task at a regular interval and measured the gap between the expected and actual execution time, directly measuring event loop delay. Under high load, jitter of hundreds of milliseconds to multiple seconds showed up. The fix was to cap concurrent requests per process at a small number and instead scale out horizontally with many more processes.

(b) Reducing feature-flag config parsing load — Statsig, the feature-flag system in use, polls by default every minute, with no jitter, pulling a huge config that includes production rules for every service. Combined with Habitat's design of up to 8 Python processes per pod, this meant every worker in a pod would wake up and parse JSON at roughly the same moment, every minute. The fix: scope the config down to only what's needed, lengthen the polling interval, and add jitter to background task scheduling so workers don't all fire at once.

(c) Switching connection pools from LIFO to FIFO — Client-side connection pooling tended to concentrate load onto a small number of server processes; tail processes sometimes carried 5-10x the average concurrent request count. Even after throttling overloaded clients, some server processes stayed degraded and kept getting worse until they were restarted — a metastable failure. The root cause was aiohttp's TCPConnector, which by default reuses connections LIFO (last-in, first-out): connections to slow, overloaded servers got returned to the pool later, which meant they were picked first for the next request, concentrating load even further in a positive feedback loop. Patching this to FIFO broke the loop and also improved load distribution under steady-state conditions. Today, Habitat relies mainly on Istio/Envoy's server-load-aware balancing instead.

(d) Fanning connections in through Envoy — With Habitat running vastly more processes than a typical service, downstream connection counts could balloon into a thundering herd; ordinary deployments already caused CPU churn from connection regeneration, and leaked connections could saturate NAT gateways. The fix was to use Envoy to upgrade Python's HTTP/1 connections to HTTP/2, enabling multiplexing, pooling, and longer-lived connections. Rate limiting and circuit breaking were also consolidated into Envoy.

Habitat doesn't allow arbitrary SQL queries. Instead, it deliberately constrains clients to a simple NoSQL API, trading query flexibility for requests that cost a predictable amount. In the earlier Postgres era, the team could review every query and schema change, but as the service grew, that stopped being feasible, and expensive hot-path queries repeatedly took down the database. SQL has an asymmetric cost profile — cheap to write, expensive to run — and Habitat avoids that trap by making potentially expensive operations visible in client code itself.

The data model is an object/edge model inspired by Meta's TAO, with types defined by clients; graph traversal beyond direct edge lookups isn't supported. Objects and their own edges are co-located in the same storage partition, but Habitat doesn't try to co-locate an object with the objects its edges point to — that keeps horizontal partitioning simple, though traversals can be inefficient and may even cross into a different Cosmos DB account in a different region. For complex queries, teams instead use CDC to stream data into offline secondary views in Rockset; each team scales its own Rockset, which adds friction, but OpenAI considers "simple queries by default, complex queries as an escape hatch" the right trade-off, and it keeps analytics and search load isolated from online storage.

Deferring the rewrite for a year let OpenAI stay focused on urgent needs during a period of rapid growth. Habitat is now the second-largest service inside OpenAI by core count, and fourth-largest by Envoy footprint. The Python implementation handled a peak of more than 20 million requests per second.

In Q2 2026, two engineers rewrote Habitat from scratch in Rust, using Codex and GPT-5.5. According to OpenAI, the Rust version now handles 95% of production requests, with Python set to be fully retired within weeks. OpenAI's own numbers put the gains at 6x CPU efficiency and 15x memory efficiency, with substantial improvements to both average and tail latency. The internal design of the Rust rewrite hasn't been published in this post — OpenAI says it will cover that in a future post — so we won't speculate here. For contrast, Prisma ORM's current direction went the other way, moving from Rust toward TypeScript; which language makes sense depends on your scale and team, not on a universal answer.

- Shrink your API surface — disallowing arbitrary queries makes cost predictable by design
- Question your defaults — feature-flag polling intervals, connection pool LIFO/FIFO behavior, and other library defaults can quietly become bottlenecks
- Measure the real bottleneck before fixing it — hidden delays like event loop lag need to be measured directly, not guessed at
- A rewrite can be deliberately deferred as technical debt — just set a deadline and exit criteria up front for when you'll actually do it

Habitat evolved in a deliberate sequence: first turning a client library into a service, then deliberately constraining its API, and finally paying down Python as intentional technical debt with a planned Rust rewrite. Part 2 is expected to cover multi-tenant reliability, tiered read performance, and collaboration with the Cosmos DB team — worth watching for.