Grafana's metrics backend for go-mysql-server
We're building Dolt, the world's first and only version-controlled SQL database. Dolt's SQL engine, the part that parses SQL queries and executes a plan to get their results, is go-mysql-server.
go-mysql-server
is written in Go and is backend-agnostic, which means that any Go program can
implement a handful of interfaces to query any data source they want. The project ships with an
in-memory backend good for testing, and Dolt of course implements its own full
backend for its version-controlled table storage. But with those exceptions, we weren't aware of any
other custom backends in the wild. It was always our hope that other teams would build their own
backends that would contribute new features and find bugs in the SQL engine, but if anyone ever did,
they didn't tell us about it.
That is, until earlier this year, when Grafana launched support for SQL expressions in their dashboard panels.
How does it work?
go-mysql-server
defines a bunch of different interfaces that collectively implement a data source
backend. For example, here's how a table is defined:
// Table is a SQL table.
type Table interface {
Nameable
fmt.Stringer
// Schema returns the table's schema.
Schema() Schema
// Collation returns the table's collation.
Collation() CollationID
// Partitions returns the table's partitions in an iterator.
Partitions(*Context) (PartitionIter, error)
// PartitionRows returns the rows in the given partition, which was returned by Partitions.
PartitionRows(*Context, Partition) (RowIter, error)
}
So a sql.Table
is an object that knows how to return the columns in its schema, can iterate over
its rows, and has a couple different metadata properties. This is the most basic interface required
to implement a backend. But you also need to implement a sql.Database
to return the available
sql.Table
objects.
// Database represents the database. Its primary job is to provide access to all tables.
type Database interface {
Nameable
// GetTableInsensitive retrieves a table by its case-insensitive name. To be SQL compliant, databases should not
// allow two tables with the same case-insensitive name. Behavior is undefined when two tables have the same
// case-insensitive name.
GetTableInsensitive(ctx *Context, tblName string) (Table, bool, error)
// GetTableNames returns the table names of every table in the database. It does not return the names of temporary
// tables
GetTableNames(ctx *Context) ([]string, error)
}
With just these two interfaces, you can get a lot of functionality from the query engine. And this
is what Grafana did. For example, here's how they implement a
sql.Database
:
// NewFramesDBProvider creates a new FramesDBProvider with the given set of Frames.
func NewFramesDBProvider(frames data.Frames) mysql.DatabaseProvider {
fMap := make(map[string]mysql.Table, len(frames))
for _, frame := range frames {
fMap[frame.RefID] = &FrameTable{Frame: frame}
}
return &FramesDBProvider{
db: &framesDB{
frames: fMap,
},
}
}
// framesDB is a go-mysql-server Database that provides access to a set of Frames.
type framesDB struct {
frames map[string]mysql.Table
}
func (db *framesDB) GetTableInsensitive(_ *mysql.Context, tblName string) (mysql.Table, bool, error) {
tbl, ok := mysql.GetTableInsensitive(tblName, db.frames)
if !ok {
return nil, false, nil
}
return tbl, ok, nil
}
A data.Frames
object is Grafana's internal representation of a data series used by their
dashboard, so this code adapts those into named tables that can be used in queries. The sql.Table
implementation is pretty
straightforward as well.
// FrameTable fulfills the mysql.Table interface for a data.Frame.
type FrameTable struct {
Frame *data.Frame
schema mysql.Schema
}
// Name implements the sql.Nameable interface
func (ft *FrameTable) Name() string {
return ft.Frame.RefID
}
// String implements the fmt.Stringer interface
func (ft *FrameTable) String() string {
return ft.Name()
}
func schemaFromFrame(frame *data.Frame) mysql.Schema {
schema := make(mysql.Schema, len(frame.Fields))
for i, field := range frame.Fields {
schema[i] = &mysql.Column{
Name: field.Name,
Type: convertDataType(field.Type()),
Nullable: field.Type().Nullable(),
Source: strings.ToLower(frame.RefID),
}
}
return schema
}
// Schema implements the mysql.Table interface
func (ft *FrameTable) Schema() mysql.Schema {
if ft.schema == nil {
ft.schema = schemaFromFrame(ft.Frame)
}
return ft.schema
}
// PartitionRows implements the mysql.Table interface
func (ft *FrameTable) PartitionRows(ctx *mysql.Context, _ mysql.Partition) (mysql.RowIter, error) {
return &rowIter{ft: ft, row: 0}, nil
}
type rowIter struct {
ft *FrameTable
row int
}
func (ri *rowIter) Next(ctx *mysql.Context) (mysql.Row, error) {
// We assume each field in the Frame has the same number of rows.
numRows := 0
if len(ri.ft.Frame.Fields) > 0 {
numRows = ri.ft.Frame.Fields[0].Len()
}
// If we've already exhausted all rows, return EOF
if ri.row >= numRows {
return nil, io.EOF
}
// Construct a Row (which is []interface{} under the hood) by pulling
// the value from each column at the current row index.
row := make(mysql.Row, len(ri.ft.Frame.Fields))
for colIndex, field := range ri.ft.Frame.Fields {
if field.NilAt(ri.row) {
continue
}
val, _ := field.ConcreteAt(ri.row)
// If the field is JSON, convert json.RawMessage to types.JSONDocument
if raw, ok := val.(json.RawMessage); ok {
doc, inRange, err := types.JSON.Convert(ctx, raw)
if err != nil {
return nil, fmt.Errorf("failed to convert json.RawMessage to JSONDocument: %w", err)
}
if !inRange {
return nil, fmt.Errorf("invalid JSON value detected at row %d, column %s: value required type coercion", ri.row, ri.ft.Frame.Fields[colIndex].Name)
}
val = doc
}
row[colIndex] = val
}
ri.row++
return row, nil
}
As you can see, it doesn't take a whole lot of code to get useful SQL query functionality for a custom data source integrated into a product. Most of the challenge for Grafana wasn't in creating this custom backend, but designing and implementing how it would work in their excellent dashboard frontend. When it all comes together, it's really neat.
Advanced use cases
go-mysql-server
is designed to support partial backend implementations, ones that only support a
subset of SQL operations. A data source can implement only a subset of the available interfaces, and
in that case some queries won't work at runtime, will miss some query planner optimizations,
etc. For example, the Grafana data sources are defined read-only, so INSERT
statements don't work
on them. If they wanted to add support for INSERT
, they would implement sql.InsertableTable
:
// InsertableTable is a table that can process insertion of new rows.
type InsertableTable interface {
Table
// Inserter returns an Inserter for this table. The Inserter will get one call to Insert() for each row to be
// inserted, and will end with a call to Close() to finalize the insert operation.
Inserter(*Context) RowInserter
}
Other advanced use cases include support for index lookups, which are vital to support performant
JOIN
operations.
// IndexAddressable is a table that can be scanned through a primary index
type IndexAddressable interface {
// IndexedAccess returns a table that can perform scans constrained to
// an IndexLookup on the index given, or nil if the index cannot support
// the lookup expression.
IndexedAccess(ctx *Context, lookup IndexLookup) IndexedTable
// GetIndexes returns an array of this table's Indexes
GetIndexes(ctx *Context) ([]Index, error)
// PreciseMatch returns whether an indexed access can substitute for filters
PreciseMatch() bool
}
You can start small with simple read-only tables, then keep expanding your implementation to get the full set of functionality supported by the engine. If you have a data source you want to query with SQL in a Go program, you should read the full guide to backend development and let us know how it goes.
Conclusion
Have a question about Dolt? Or do you have a data source you want to query with SQL in your Go application? Come by our Discord and talk to our engineering team. We hope to see you there.