As summer winds down and fall begins, back-to-school season is upon us, and the academic researchers are getting back to work. Here at DoltHub, that means I’ve got a lot of GitHub issues in the Dolt repo to sort through.
While researcher-filed issues get labeled as “customer issues”, they are immediately identifiable from typical customer issues since they tend to fall into the same pattern: reports that follow a somewhat formal structure, simple queries that allow a bug to be easily and minimally reproducible, clearly detailed actual and expected results. Since they are not true customer issues, they are not prioritized as highly and don’t always get our 24-hour bug fix guarantee. However, they are still important since we do deeply care about ensuring correctness on Dolt (and all our other databases).
Last year, we thought we were experiencing a bit of a bug swarm when the researchers working on UC Berkeley’s Argus filed 19 issues in the span of 2 months. But that was nothing, and I mean NOTHING, compared to the 100+ issues collectively filed by GitHub users wanteatfruit and yibo-dong in the past 1.5 months. yibo-dong alone filed 25 bugs on August 11. Here’s some actual footage of me opening the Dolt GitHub issues page that morning:
These issues may seem like Dolt has a lot of correctness bugs, but rest assured, we maintain 100% correctness on sqllogictest, and after fixing the bugs identified by Argus, we only failed one out of every 12 million of their test queries. We rarely get correctness issues filed by actual real-world users. So these issues are likely edge-cases. And given the sheer number of them, these queries were probably generated by AI, similar to those generated by Argus.
So who are wanteatfruit and yibo-dong? According to their GitHub profiles and websites, wanteatfruit is Junwen An, a computer science PhD student at the Trustworthy Engineering of Software Technologies (TEST) Lab at the National University of Singapore (NUS), and yibo-dong is Yibo Dong, who is also a PhD student at the TEST Lab at NUS. What a coincidence! But what’s the TEST Lab? What were they up to filing so many issues with us? And how were they generating so many queries in such a short amount of time?
So I did a bit of internet sleuthing and found that the TEST Lab is a computer science research lab at NUS led by Dr. Manuel Rigger. The lab is part of the Programming Languages and Software Engineering group in the School of Computing, and one of their projects, SQLancer, focuses on automatically finding bugs in database systems.
As I explored the lab’s website, I saw they had a section of their Found Bugs page dedicated to bugs found in Dolt. That’s when I recognized some of the bugs filed by our old friend theoristcoder – some might recall that theoristcoder filed an issue that exposed problems with Dolt’s implementation of zero time. In fact, the TEST Lab has been filing bugs with Dolt as far back as 2023. And when I checked the People page, I recognized Emily Ong, who had filed bugs with us last year, and Qiuyang Mang, who had worked on Argus at UC Berkeley. So it seems like the TEST Lab has been doing work with Dolt for quite some time.
This was exciting! But how have we not made contact before? I reached out to Manuel with my observations that the TEST Lab seemed to have taken an interest in testing Dolt in the past several years, and he responded! He told me that the TEST Lab took interest in Dolt due to its popularity on GitHub (we do take pride in our 24.5k stars) and its open source nature (which we do also proudly boast). They continue to file issues with us since we are “highly active” about fixing bugs and ensuring correctness, which is apparently rare even among popular databases.
Manuel linked to me Suyang Zhong’s paper “Scaling Automated Database Testing Systems” and Zhaokun Xiang’s paper “Detecting Join Bugs in Database Engines via Join Implication Reasoning”, both of which were published earlier this year in ACM journals. Suyang had been filing bugs with Dolt since 2023, and Zhaokun is theoristcoder! Yibo and Junwen are working on separate projects right now, but both projects involve testing SQL databases, and they have papers in progress.
I also had the chance to chat with Yibo on his project WINE, a testing approach that focuses on window functions, and got confirmation that he was indeed using AI to generate his queries, though he does manually review them before filing issues with us on GitHub. The number of queries WINE has been able to generate is impressive, and I’m eager to talk to him more about it.
This is the first blog post in a series on the work the TEST Lab has done with Dolt. In the upcoming posts, I will be discussing Yibo’s project WINE, Suyang’s paper on scaling automated database system testing, and Zhaokun’s paper on detecting join bugs, with possibly even more posts to come as more papers from the TEST Lab are getting accepted into journals.
I would like to give a huge thanks to all the researchers who have helped us improve correctness on Dolt over the years and a special shoutout to Yibo and Junwen, who have filed issues more recently. As Tim outlined in his recent blog post on the open source flywheel, we are driven by fixing user issues. The number of SQL queries out there is infinitely large, and as a small team of engineers, there is no way we could do this alone so we are so grateful for any help from others.
Are you an academic researcher working on a project with Dolt or another DoltHub database? Join our Discord community – we’d love to hear from you.