DoltLite just keeps getting better. Six weeks after Beta, DoltLite is on the DoltHub homepage, in Homebrew, and somehow 18 patch releases deep. This article explains the progress and what’s next.
What is DoltLite#
DoltLite started as a lark. I needed a pet project to test with Steve Yegge’s innovative agent orchestrator, Gas Town. I wanted a real problem, not a toy. We’d wanted an embedded version of Dolt for years but rewriting Dolt’s storage engine in C or Rust was a bridge too far. SQLite seemed like a logical host for that engine. Could a team of agents pull it off? Yes. Yes, they could. DoltLite is currently in Beta and getting better one agent-engineered PR at a time.
DoltLite is a fork of SQLite. Everything above the B-tree layer is the same. That means the SQL parser and analyzer, the file system interaction layer, and test harness are stock SQLite. With other Dolt products, we had to implement a SQL engine on top of version-controlled storage. Building a SQL engine is hard. With DoltLite, we got one for the cost of writing a version-controlled storage engine in C.
The B-tree layer is swapped out for a Prolly Tree backed by a single file chunk store. Prolly Trees are content-addressed B-trees. This magical data structure powers the version control functionality in all Dolt products.
That means you get all the version control features of Dolt and Git in a SQLite package. Have you ever wanted to branch, merge, and diff SQLite? DoltLite is for you. Or maybe even more importantly, have you ever wanted a conflict-aware SQLite sync engine powered by Git-style push, pull, clone, and fetch? DoltLite is for you. You can even use DoltHub as your sync backend.
So Many Patches…#
DoltLite improves by cranking the open source flywheel.
Agents generate the vast majority of issues but we do have user issues mixed in. When there are no issues to work on, I will have an agent (currently Claude Opus 5.5) do a deep code review to identify reproducible, actionable issues to fix. I review the issues and have agents fix as many as I can. This process can be repeated two to four times per week with each cycle generating between ten and fifty issues. When I fix a user-generated issue or complete a sufficient number of agent-generated issues, I cut a patch release. This process is fueling the patch release volume.
So what’s changed? Here are the highlights.
Homebrew#
To get accepted into Homebrew, you need 200 GitHub stars for a self-submission. DoltLite got there about a month ago. Last week, the Homebrew maintainers merged our PR, and now, Mac and Linux users have easy DoltLite access via Homebrew. Throw DoltLite a star if you haven’t already. It helps.
$ brew install doltlite
...
...
...
==> Downloading bottle manifests
✔︎ Bottle Manifest doltlite (0.50.17) Downloaded 15.7KB/ 15.7KB
==> Fetching downloads for: doltlite
✔︎ Bottle doltlite (0.50.17) Downloaded 7.4MB/ 7.4MB
==> Upgrading doltlite
0.50.15 -> 0.50.17
==> Pouring doltlite--0.50.17.arm64_sequoia.bottle.tar.gz
🍺 /opt/homebrew/Cellar/doltlite/0.50.17: 9 files, 16.2MB
==> Cleanup
Removing: /opt/homebrew/Cellar/doltlite/0.50.15... (9 files, 15.9MB)
Removing: /Users/timsehn/Library/Caches/Homebrew/doltlite--0.50.15... (7.2MB)
Removing: /Users/timsehn/Library/Caches/Homebrew/doltlite_bottle_manifest--0.50.15... (15.7KB)New Bindings#
On top of Homebrew, DoltLite now ships with the following additional bindings:
| Binding | First release | Date |
|---|---|---|
.NET (DoltHub.Doltlite on NuGet) | 0.50.1 | September 1 |
Rust (doltlite on crates.io) | 0.50.2 | September 1 |
Go (doltlite-driver) | 0.50.3 | September 3 |
This is in addition to Node, WASM, Python, Ruby, and all the other language bindings that existed at Beta. You can use DoltLite in almost any language.
If the language binding you want isn’t available yet, just cut a GitHub issue.
Performance Hotspot Testing#
DoltLite performance has been tested since week one using a sysbench-style benchmark against DoltLite. These metrics have been heavily optimized, and there is not much room for improvement. See the checked-in nightly performance report for up-to-date benchmarks. DoltLite pays a Prolly Tree performance tax of 0-60% depending on the workload. Autocommit writes pay a 3X (300%) tax due to write amplification and hashing.
What else in DoltLite is slow? I’ve tackled this question since Beta. I set up a nightly performance hotspot detection job. Over the past month or so, the job has been identifying SQL workloads where DoltLite is more than 3X slower than SQLite. If the performance hotspot can be fixed, it is. Otherwise, it is added to the known performance hotspot list and checked for regression on every PR. I started with individual hotspots, but I have since grouped them into themes.
DoltLite is slower than SQLite on the following workload types:
- Add column with default
- Wide rows
- Uncached reads
- Primary key index rewrites
- Text key index fetches
- Extensive use of the pending edit map
Not all workloads in these themes are 3X slower than DoltLite, but it’s easy to design a workload that is.
A common workaround to many of the performance issues above is to give DoltLite a larger in-memory cache than you would SQLite. DoltLite chunks are checked for corruption using their content address on read from disk as an extra durability measure. Once in cache, chunks are not checked. Thus, in DoltLite, it pays more dividends to keep hot chunks in cache. You raise the cache size using PRAGMA cache_size, the same interface as SQLite. The default is 64Mb. Each chunk averages 4kb so that is approximately 16,000 chunks.
Version Control Edge Cases#
The other issues fixed in this cycle have mainly been on the version control surface.
The SQL surface is very clean, outside of performance concerns. SQLite ships with a battery of tests, and DoltLite passes 99.4% of them with the vast majority of the other 0.6% being known divergences. It is difficult to find SQL issues.
That leaves the version control surface. Version control fixes have mostly been edge case functionality. Version control concurrency and durability fixes are less numerous but present. This will be a continuing focus until DoltLite’s version control is perfect.
What’s Next?#
Great progress. What’s next?
DoltLite on DoltHub#
You can already push DoltLite databases to DoltHub and clone them once they are there. But all the additional database functionality is not implemented. This month, we will add that functionality including Pull Requests. This will enable DoltHub to act as your online DoltLite sync-engine, complete with conflicts and Pull Requests.
Getting DoltLite on DoltHub involves improving remotes, the least tested surface area of DoltLite. Large databases and remotes are a particular concern.
Version Control Under Concurrency#
Version control under concurrency will be the primary focus of my review agents. There have already been some major improvements, and these will continue until agents cannot come up with reasonable breaking edge cases.
Major Version Bump?#
Finally, since a couple months before Beta, I’ve been deferring fixes that require a storage change. So far, nothing has risen to the level of requiring a storage change. If there is something in the list you want, let me know.
Conclusion#
DoltLite keeps getting better at the pace of agents driven by a really motivated founder. Try DoltLite today for your embedded Dolt use cases. If you need any help, come by our Discord. Meet me in the #doltlite🪶 channel.