Since its launch in 2008, GitHub has catalyzed the open source software world and accelerated the culture of software
collaboration. Source control was an old idea at
that point, but GitHub offered a centralized place to discover and collaborate on new projects. We created DoltHub
to foster that same collaboration around open data sets, and now DoltHub allows
exploration of those dataset via SQL queries on the web.
Let's use the baseball-databank repository to demonstrate.
The initial view shows the repository's tables on the left and the data in the center, but if we want to understand the
usefulness of this dataset we can do better than scrolling through its rows. Let's say we want to do a simple correlation
analyzing the relationship between player salary and batting average. We'll start by checking what years we have data for.
Next, we can do some sanity checks to ensure we trust our source.
It seems that everything is in order. At this point we can grab the data and perform our analysis.
If you haven't done so already, install Dolt
> sudo curl -L https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
and clone the repository
> dolt clone dolthub/baseball-databank
Rows and cells in isolation don't tell the full story. Queries on the web offer a lightweight way to explore the shape of the dataset,
and decide whether it's worth cloning locally.
Hosting a service that can query any repository on DoltHub presented unique challenges in development. Hosting a traditional
database instance per repository is prohibitively expensive. Cloning repositories on-demand would introduce server state
and would scale poorly. Thankfully Dolt's storage layer is immutable and content-addressable, meaning that the we can
fetch the correct subset of data on-demand from the remote S3 chunk store. Queries are automatically fetched from the HEAD
of each branch, so collaborator's changes will be reflected on DoltHub as soon as they
Currently, queries are limited to read-only select, describe and show statements. In the future we will extend this
functionality to querying diffs and generating pull requests from write queries. We've also placed limits on the compute
time and the number of rows returned by a single query. Once you know what data is available, you can easily get the full
dataset including history with