We make Dolt, the world's first version controlled SQL database. If you're making a database, you need to be fast, so performance has always been important to us. As part of our continuous integration, we run sysbench to monitor potential performance regressions. And we talk a fair bit about how we run these benchmarks and the results they give us.

Recently, we encountered an issue where the sysbench tests would no longer run locally on our some of our workstations. It still ran fine in CI, and Zach could run the tests locally without any issue, but whenever Dustin, Max or I tried, we got an error message:

FATAL: error 2026: TLS/SSL error: SSL is required, but the server does not support it

It's true that the Dolt server we were having sysbench connect to wasn't using SSL, but that hadn't been an issue for the benchmark before. Why did it suddenly care, and why did it only care for some of us?

Max, Dustin and I were both developing on MacBooks; Zach was using Windows. This pointed to something specific to the Mac environment. But what had changed, and why?

This error vexed us. Sysbench documentation suggested passing the --mysql-ssl=off flag to sysbench, but this had no observable effect on the output. The MySQL docs suggested that a client could be forced to not use ssl by passing the flag --ssl-mode=DISABLED, but modifying the sysbench scripts to pass this flag resulted in the error unknown variable 'ssl-mode=DISABLED'.

Clearly something was going wrong with the communication between the SQL client and the server that was preventing them from establishing a connection. But was the client responsible, or the server? And why did this only affect the team members on MacBooks?

We had a theory: Dolt implements a MySQL server, and implements MySQL's mysql_native_password plugin to handle authentication. This plugin was deprecated in MySQL client 9.0, but remains available in 8.4, the LTS version. The error message didn't mention anything about authentication or plugins, but it wouldn't be the first time an error message misrepresented the root cause. Had we all been somehow updated off of 8.4?

Dustin checked his environment:

dustin@MacBook-Pro % mysql --version
mysql  Ver 9.1.0 for macos14 on arm64 (MySQL Community Server - GPL)

This seemed promising. But before he could try downgrading, Max reported that he was already on 8.4. And as for myself:

nick@MacBook-Pro % mysql --version
mysql from 11.6.2-MariaDB, client 15.2 for osx10.19 (arm64) using  EditLine wrapper

MariaDB is an open-source implementation of the MySQL dialect and API. So not only were the three of all running different versions of the client software, we weren't even running the same software. This made it unlikely that the client version was causing the problem.

So was it a server problem? We tried running the benchmarks against an older version of Dolt that had worked in the past, but we got the same error.

So if it wasn't because of Dolt, and it wasn't because of the installed MySQL client, was it because of sysbench itself somehow? We all had used Homebrew to install sysbench. Had Homebrew pushed a new version of sysbench that we'd all upgraded to?

Brew offers a command line tool brew info that can list the available versions of a package:

% brew info --json sysbench | jq '.[].versioned_formulae'
[]
% brew info --json sysbench | jq '.[].versions'
{
  "stable": "1.0.20",
  "head": "HEAD",
  "bottle": true
}

Basically, this told us that Homebrew only has one supported stable version of sysbench, and it's version 1.0.20. This is the latest stable release of sysbench, and was released all the way back in 2020. Since this is the only version in Homebrew, there was no prior version to roll back to. This definitely suggested that Homebrew couldn't have upgraded us to a new version of sysbench.

But it turns out that version numbers are just part of the story. Because Homebrew is more than just a package manager.

What is Homebrew?

Homebrew is a bit of an odd duck among package managers. For starters, they like to give everything an on-theme name. Just check out this table of terminology from their documentation:

Cellars contain racks contain kegs which contain formulae... still following?

While you can get prebuilt binaries (bottles) through Homebrew, these "bottles" were "poured" from a "formula"... which is to say, the target was produced by following build instructions written in a domain specific language. Formulae are stored in git repos, with the canonical formulae repo hosted at https://github.com/Homebrew/homebrew-core.

I suspect that most of the time, this process doesn't actually doesn't matter: at the end of the day, you're still going to just be running brew install whatever, and maybe picking the version you want to install if you really care.

But the whole "formula" situation means there are actually two different notions of "version" in Homebrew: the version of the software being built, and the version of the formula itself (which Homebrew refers to as the formula's revision). In certain circumstances, Homebrew may update the build process for a target even when the target has no new version number. This might be done in order to update one of the target's dependencies, or to fix a security vulnerability in the formula itself, or other similar reasons. If this change bumps the formula's revision number, then brew update will detect the new revision and rebuild the target automatically, the same as if the target had a version bump.

Did this theory have legs? Did Homebrew do something to change how it built sysbench on Mac?

Taking a quick look at the sysbench GitHub repo, I noticed something in the build instructions:

If you have MySQL headers and libraries in non-standard locations (and no mysql_config can be found in the PATH), you can specify them explicitly with --with-mysql-includes and --with-mysql-libs options to ./configure.

To compile sysbench without MySQL support, use --without-mysql. If no database drivers are available database-related scripts will not work, but other benchmarks will be functional.

Sysbench depends the MySQL client in order to run its benchmarks, but it's a static dependency. In other words, it doesn't matter what version of the client you have installed on your system, because sysbench doesn't use your MySQL client, it links against its own copy of the client at compile time. If Homebrew issued a new revision that changed how this linking was done, it could cause a change in observed behavior, even though sysbench didn't release a new version.

Because Homebrew sources all its forumlae from a GitHub repo, we can not only look at the formula for sysbench, but track its changes to see if it issued a revision that changed how Sysbench was built from source. Homebrew even offers a command to view the Git history of a formula, brew log:

% brew log sysbench
commit 25146b19a08069ab8dd420520dc547e15e5267b0
Author: Michael Cho <michael@michaelcho.dev>
Date:   Sun Nov 24 16:35:10 2024 -0500

    sysbench: migrate to `pkgconf`

commit aee80d32858258b07f6d592235b5e980bb84a940
Author: BrewTestBot <1589480+BrewTestBot@users.noreply.github.com>
Date:   Sat Nov 9 22:19:14 2024 +0000

    sysbench: update 1.0.20_7 bottle.

commit 04094efcee6053dec166e60dde0ebd5e4f4c5eef
Author: Michael Cho <michael@michaelcho.dev>
Date:   Tue Nov 5 16:02:40 2024 -0500

    sysbench: switch to `mariadb-connector-c`

commit af264df35f07d3411119d0192ac5b4ba47e92b9b
Author: Michael Cho <20700669+cho-m@users.noreply.github.com>
Date:   Tue Sep 10 23:18:37 2024 +0000

    sysbench: update 1.0.20_6 bottle.

We can see here that while Homebrew only supports a single version of sysbench, there have been multiple revisions of the formula, with several changes made last November.

The commit that says switch to mariadb-connector-c looks promising. Here's the PR on GitHub. And here's the change that commit made to the formula in Sysbench.rb:

@@ -4,7 +4,7 @@ class Sysbench < Formula
  url "https://github.com/akopytov/sysbench/archive/refs/tags/1.0.20.tar.gz"
  sha256 "e8ee79b1f399b2d167e6a90de52ccc90e52408f7ade1b9b7135727efe181347f"
  license "GPL-2.0-or-later"
- revision 6
+ revision 7
  head "https://github.com/akopytov/sysbench.git", branch: "master"

  bottle do
@@ -25,19 +25,13 @@ class Sysbench < Formula
   depends_on "pkg-config" => :build
   depends_on "libpq"
   depends_on "luajit"
-  depends_on "mysql-client"
-  depends_on "openssl@3"
+  depends_on "mariadb-connector-c"

   uses_from_macos "vim" # needed for xxd

-  on_macos do
-    depends_on "zlib"
-    depends_on "zstd"
-  end

  def install
    system "./autogen.sh"
-   system "./configure", *std_configure_args, "--with-mysql", "--with-pgsql", "--with-system-luajit"
+   system "./configure", "--with-mysql", "--with-pgsql", "--with-system-luajit", *std_configure_args
    system "make", "install"
  end

Essentially, sysbench and several other formulae had been changed to depend on MariaDB's client library instead of MySQL's, as part of an experiment. While both libraries are intended to implement the same interface, they seem to differ when it comes to default settings and command line flags: hence why the sysbench scripts were now trying to use SSL by default and didn't respond to the flags that were supposed to disable it.

We considered how to proceed here. We could try to figure out how to disable SSL in the MariaDB client... but why bother? If the old formula worked, why not just build sysbench using the old revision and pin ourselves to that revision?

It turns out pinning to an old revision is possible, but requires jumping through some hoops. We can't build it using the canonical "tap", we have to make our own.

Here's the script we ended up using:

TAP="dolthub/homebrew"
MODULE="sysbench"
GIT_REV="af264df"
brew tap homebrew/core
brew tap-new $TAP
brew extract --git-revision $GIT_REV $MODULE $TAP
brew install $TAP/$MODULE

We'll go through this one line at a time.

• TAP="dolthub/homebrew"

A tap in Homebrew is an isolated environment that contains its own formulae. Every tap is backed by a Git repo. By default, Homebrew only has the canonical tap "homebrew/core". If we want to define our own formulae, we need to make our own local tap. We name it after ourselves, of course.

• MODULE="sysbench"

The name of the module that we want to make a custom formula for. Simple enough.

• GIT_REV="af264df"

If every tap is a Git repo, it has a history. This is the last commit in the history of Sysbench.rb before it switched to depending on MariaDB. At this revision, the homebrew/core tap contains the formula that we know works.

• brew tap homebrew/core

This command clones the core tap so that we have a local copy of it. This is important because it lets us access the repo directly instead of going through the homebrew API.

• brew tap-new $TAP

At this point, we probably could reset the core tap to an earlier revision, but that would affect lots of other formulae too. Instead of making changes to the core tap, it's safer to just create a new one that serves as an alternate installation source.

• brew extract --git-revision $GIT_REV $MODULE $TAP

This is where the magic happens. We query the Git history of the named module, find the version at the specified commit, and create a new formula in the specified tap. Because the name of the module is simply "sysbench" instead of a qualified name, Homebrew defaults to the core tap for the lookup.

When this command completes, our custom tap now has a single formula in it: a copy of the sysbench formula as it existed back in November.

• brew install $TAP/$MODULE

This is the final step. We install the formula using the fully qualified name. Since it's sourced from our custom tap, it's effectively pinned too unless we update the tap.

But did it work?

nick@Nicks-MBP scripts % /Users/nick/Documents/dolt/go/performance/scripts/local_sysbench.sh
benchmark oltp_point_select bootstrapping at /var/folders/vh/b5w64dds3_d5l65rjw0mxdp40000gn/T/tmp.ye11EpGfoa
Starting server with Config HP="0.0.0.0:3366"|T="28800000"|R="false"|L="info"
sysbench 1.0.20 (using system LuaJIT 2.1.1741730670)

Running the test with following options:
Number of threads: 1
Initializing random number generator from current time

Initializing worker threads...

Threads started!

SQL statistics:
    queries performed:
        read:                            562417
        write:                           0
        other:                           0
        total:                           562417
    transactions:                        562417 (18746.85 per sec.)
    queries:                             562417 (18746.85 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          30.0003s
    total number of events:              562417

Latency (ms):
         min:                                    0.04
         avg:                                    0.05
         max:                                    7.80
         95th percentile:                        0.00
         sum:                                29939.64

Threads fairness:
    events (avg/stddev):           562417.0000/0.00
    execution time (avg/stddev):   29.9396/0.00

benchmark oltp_point_select complete at /var/folders/vh/b5w64dds3_d5l65rjw0mxdp40000gn/T/tmp.ye11EpGfoa

Success!

Conclusion

I was surprised that once I deciphered Homebrew's creative terminology, it's really just a thin layer on top of Git. This makes Homebrew extremely flexible, albeit with a steeper learning curve than most package managers. But the fact that it's built on top battle-tested open standards like Git meant it was easy to learn if you already understand Git and version control.

And we know version control. We built our entire company around our belief that all data benefits from version control. And Homebrew seems to agree with us, since they built their whole package manager on top of it.

If you also think that everything benefits from version control, consider giving Dolt a try! If you prefer Postgres flavor over MySQL flavor, we just beta-launched Doltgres, our version of Dolt that speaks the Postgres dialect. Alternativelty, join our Discord and chat with us. We're always looking to learn more ways that people want to use version control to improve their lives.