DumboDB is a MongoDB-compatible version-controlled database. Its origin story, honestly, was an experiment I didn’t expect to pan out. Using a pre-1.0 version of Gas Town, I pointed a bunch of coding agents at FerretDB, Dolt, and MongoDB’s documentation, and they built the first version of DumboDB. I was surprised it worked at all, and even more surprised that it worked well enough to continue.
DumboDB continues to be a completely AI-generated code base. I don’t read any of the application code, though I do occasionally read some tests. When discussing this fact with other engineers, I often get the question, “How do you know it works?” Today, I’ll try to answer that question.
Reference Implementation#
The most obvious entity that describes how DumboDB should behave is MongoDB itself. When you have a reference implementation, it greatly simplifies the process of determining if your implementation is correct. This is sometimes called a “golden master” or “oracle” in software testing.
DumboDB has tests stored in a separate repository called dumbodb-parity-testing. The parity tests run by executing a series of queries against MongoDB, capturing its results, then doing exactly the same queries against DumboDB and comparing the results. If the results are identical, then DumboDB is behaving correctly.
The current state of the tests at HEAD for DumboDB (not released yet):
| DumboDB Status | Count | Meaning |
|---|---|---|
| DumboDBFull | 1,935 | Run on both; DumboDB must match MongoDB — divergence fails CI |
| DumboDBMongoOnly | 38 | Run on MongoDB only; DumboDB skipped (unsupported feature) |
| DumboDBXFail | 10 | Run on both; DumboDB divergence recorded, not a CI failure |
| DumboDBDeviates | 6 | Run on both; DumboDB intentionally differs — each server asserted against its own expected outcome |
| Total | 1,989 |
To give a little history, the first version of DumboDB had ~1400 parity tests. There were about 1000 tests which were ported from FerretDB’s test suite and about 400 tests which were added to cover gaps we found along the way. When we first released, there was no DumboDBDeviates category, and the DumboDBXFail category was bumping up close to 100. We had no tests which were DumboDBMongoOnly.
Since then, we’ve had a lot more time to evaluate MongoDB and how DumboDB plays in the space. For example, almost all of the tests which are currently marked as DumboDBMongoOnly are tests which exercise features we never intend to support. Of the 38 tests, 29 are related to capped collections and insertion ordering using the $natural aggregation operator. Since we are building a database which supports branching and merging, we run into a challenge with capped collections right away. If you merge two branches and that results in overflowing your collection cap, how can we determine which documents to keep and which to discard? You could imagine ways to address this, but we haven’t worked out what that looks like, so for the time being, that’s a MongoDB-only feature.
Similarly, the DumboDBDeviates category is a collection of tests in which we intentionally diverge from MongoDB. A great example of this is that the admin.system.* tables allow arbitrary inserts. admin.system.users is a table which stores user credentials, and the documentation states that you should avoid modifying this table directly and instead use the createUser and updateUser commands. Inserting directly into the admin.system.users collection is allowed in MongoDB, for reasons I’m sure go back to the ancient history of the product. In DumboDB, we don’t allow this (because it’s insane) and instead require that you use the createUser and updateUser commands. This is a case where we intentionally diverge from MongoDB, and we have tests to ensure that we continue to diverge in the future.
We are adding new tests every time we expand the feature set. Recently, we added 298 new parity tests to cover all authentication and authorization features. Coding agents are pretty good at this kind of work. They look at all of the documentation for the reference implementation. Then they spin up a server instance and test some of their assumptions before generating a test matrix and a ton of tests. Finally, we can run those new tests against MongoDB as a sanity check that the tests are doing something. We look at the logs of the server, do some spot checks on the test code, etc. Ultimately we must recognize that coding agents aren’t trying to fool us. You are welcome to go and review the 5K lines of test code yourself, but I don’t recommend it. It defeats the purpose of coding agents, and if we are going to squeeze every ounce of productivity out of them, we need to learn how to trust them, at least to some extent. Claude isn’t going to just generate 300 no-op tests; he’s not a fraud.
Human Verification#
Comparing against the reference platform only gets you so far. That would be fine if we had no new features to add that MongoDB doesn’t have, but what would be the point of that?
For features that are unique to DumboDB, we use a process where we get our coding agent to generate a series of reproduction steps for a given feature. These reproduction steps are produced in two forms: 1) A markdown document that describes the steps in English, and 2) unit tests that are 1:1 with the reproduction steps.
As an example, the dumboDiff command has a human verification document and identical unit tests. The first document describes how I can copy commands into the mongo shell and what I should expect to see. Honestly, 25 years ago writing such a document and handing it off to a QA team was part of my job. What’s nice about coding agents is that the parallel form, as unit tests, come for “free.” You can read the tests and see that they do the same thing, but in a different language. The purpose of the unit tests is to avoid regressions. While I do manually go through the verification document when the feature is built, the automated tests are run in continuous integration to ensure that the feature continues to work as expected.
Currently, there are ~200 scenarios which are covered by ~1800 unit test assertions. All of those 200 scenarios have been verified by a human (me). It’s not uncommon for interface improvements and tweaks to application behavior to happen during this verification. This is the human-in-the-loop step which verifies the product is cohesive. This number will continue to grow as we add new features.
Load Testing#
We run nightly load tests against DumboDB. These tests, like everything we ship, are in public code, but you need an AWS instance properly configured to run them. Every morning, I start my day looking at a report like the following:
I was curious about the errors listed there, so I dug in and found a memory leak when using the $group aggregation operator. With that fixed, the next day the report looked like this:
The errors not only went away, but the memory utilization flattened out. Two birds with one stone!
There are also benchmarks that compare MongoDB and DumboDB query performance. I wrote about this before, and honestly I need to automate running these benchmarks. Totally doable, just a matter of getting to it. In the meantime, I run them manually on each release candidate. These tests have been expanded since the 0.1 release, and I’m sure we will add more as real-world usage of DumboDB uncovers new bottlenecks.
Third-Party Apps#
There are several third-party applications that have been built on top of MongoDB. These applications all use the MongoDB interface, and therefore, DumboDB should be a drop-in replacement. Each time we test out one of these applications, we discover new ways in which MongoDB behaves. For example, when testing Compass, we discovered support for the $$ROOT operator, which isn’t heavily documented but is required for Compass to work. There were also silent errors uncovered by enabling Compass debug logging. We tested against mongo-express and discovered that the --auto-commit flag was not quite correct. So we fixed that. Currently, I’m testing against parse-server. The parse-server project has a test suite with more than 1000 tests, all of which fail because DumboDB doesn’t have collation support. So I’m working on that now. There are more than a dozen more third party applications which we will be testing against, and I expect to find more bugs and divergences along the way.
In the ideal case, these third-party applications have test suites. A common pattern is to run a MongoDB server with these tests, so we just run a DumboDB server instead and see what goes sideways. We haven’t automated any such test harnesses yet, but I suspect we will when good candidates come along. Parse Server is certainly a good candidate.
In the long haul, third-party applications will probably be the most fruitful way to validate that DumboDB is working correctly. The MongoDB interface is large, and there are many ways to use it. We can’t possibly test all of them manually, but third-party applications will be a good proxy for how the interface is used in the wild. We had a similar “Works On Dolt” campaign for Dolt, and it was very successful in uncovering bugs and divergences. I expect the same for DumboDB.
Conclusion#
I have high confidence that DumboDB works well, at least for the features we have built. There are certainly gaps in behavior which we haven’t tackled yet, some known and some unknown. But the combination of parity testing, human verification documents, load testing, and third-party application testing gives me confidence that DumboDB is a solid implementation of the MongoDB interface.
If I’m being honest, this all looks like regular software engineering work. Dolt has hundreds of thousands of tests, all added over time and all run on every PR. It doesn’t matter that people wrote all of them - they are the codification of Dolt’s behavior. DumboDB is no different. The only difference is that coding agents wrote all of the code.
The next question I get after “How do you know it works?” is usually “Is it maintainable?” I honestly don’t know if I can answer that question until we hit the one-year mark. Adding new features has not gotten any harder over the last three months. Maybe we’ll hit a point where agents can’t make sense of the codebase they have written, and we will have to start over. Thankfully we have all of this behavioral testing, so a complete re-write of the application would at least have that to start with. I don’t see that happening any time soon though. Stay tuned, we’ll let you know when we hit the one-year mark and see how things are going.
Want to learn more about Dolt and Dumbo? Hop on our Discord to ask questions and nerd out about version-controlled databases!