TLDR: The NVD is a lot more useful when you can simply clone it
and query it.
The National Vulnerability Database
(NVD) is the authoritative source for the publication of Common Vulnerabilities
and Exposures (CVE). The vulnerabilities cataloged in the NVD represent the most severe and most impactful cyber-security
events. The database exists as a collaboration between the
MITRE corporation and The National Institute of Standards and Technology (NIST).
Within the NVD you'll find entries for famous security bugs such as
But the NVD also contains information about more routine security issues, like the 2120 CVEs affecting
The trouble with the NVD is that it's just not that easy to use. Surely somewhere within NIST there is a database that
contains all of this data, but for everyone else there's a JSON api where you can download around twenty zip files
containing portions of the dataset. Cat all these files together and you still wouldn't have anything that would answer
a question like 'how many Redhat Linux bugs allowed root access?'. A lot of code has been written trying to make this
dataset available and useful, but having to add more software to your toolchain to run a query seems kind of broken right?
Using this data through Dolt
is an order of magnitude improvement over writing your own parsing code. Simply
dolt pull to download the latest data.
Then you can run a SQL query against it with
dolt sql -q '...'. If you come across an interesting query result,
you can save the query and distribute it along with the dataset. The NVD is constantly adding new data and updating
information for old entries, you can track all of these changes with
Within the time it took to write this blog,
was added! It's a pretty cool dataset, let's take a look.
Getting the Data
As I mentioned above, when it comes to doing anything useful with the data, you're on your own.
Sweet. Luckily our friends at Facebook maintain a Github Repo
with tools to download and parse the feed. The CVE feed API follows this JSON schema,
which is not all that useful without specific tools that know how to query it. Thankfully your friends at dolthub
mapped it to a SQL schema
so you didn't have to. After doing a little digging, it seemed like there were some not-so-useful fields in the data.
% dolt sql
NVD> SELECT DISTINCT data_format FROM CVE;
| data_format |
| MITRE |
NVD> SELECT DISTINCT data_type FROM CVE;
| data_type |
| CVE |
NVD> SELECT DISTINCT data_version FROM CVE;
| data_version |
| 4.0 |
These fields don't add any information to the CVE entries, so I deleted these columns and added this info to the
You can view the diff
of the schema change on Dolthub. Once we're all done normalizing our data, we get the following:
% dolt sql -q 'describe CVE;'
| Field | Type | Null | Key | Default | Extra |
| cve_id | VARCHAR(30) | NO | PRI | | |
| description | LONGTEXT | YES | | | |
| problem_type | LONGTEXT | YES | | | |
| expoitability_score | FLOAT | YES | | | |
| impact_score | FLOAT | YES | | | |
| date_published | DATETIME | YES | | | |
| date_last_modified | DATETIME | YES | | | |
The CVE table contains the date we're really interested in. Each row lists a single CVE along with the data about the nature and severity of the problem.
The rest of the tables contain links to references,
and data for scoring the CVE.
So what about Redhat Linux?
nvd> select count(distinct cvss2.cve_id) from cvss2
-> join affected_products on cvss2.cve_id = affected_products.cve_id
-> where cvss2.obtain_all_privilege = 1
-> and affected_products.vendor = 'redhat'
-> and affected_products.product = 'linux';
| COUNT(DISTINCT CVSS2.cve_id) |
| 92 |
Keeping up to date
The NVD is a constantly evolving dataset. The API feeds are updated every hour and import into this Dolt repository automatically.
Security vulnerabilities are extremely time sensitive, so it's vital to have up-to-date information. I've created some
in the Query Catalog to quickly access the most recently published vulnerabilities. If you clone a local copy of this
repo, you can quickly get the latest updates in your shell.
% dolt pull
% dolt sql -x "New CVEs Today"
Executing saved query 'New CVEs Today':
select * from cve where date_published > (now() - interval 24 hour) order by date_published desc;
| cve_id | description | ... |
| CVE-2019-20102 | The attachment-uploading feature in Atlassian Confluence Server from ... |
| CVE-2018-21145 | Certain NETGEAR devices are affected by a stack-based buffer overflow ... |
| CVE-2020-1757 | A flaw was found in all undertow-2.x.x SP1 versions prior to ... |
| CVE-2020-5268 | In Saml2 Authentication Services for ASP.NET before versions 2.7.0 ... |
There's a lot more to explore with the NVD repo, we hope you'll clone your own copy and see what you can find. We believe
that Dolt adds a lot of value in distributing and consuming dataset. The ability to
merge your data brings a new level of fluidity to data analysis. This form of collaboration is the norm in software
development, why shouldn't data collaboration be just as simple? Check out these dataset we're publishing on Dolthub
or create your own!