Hacking the Heartbeat Monitor of a Data Center - ScienceLogic SL1
Updated: Sep 19
June 29th, 2023 By Vincent Mentz
Update 9/19/23 - This vulnerability has been assigned CVE-2023-42266.
Digging deep, being thorough, and remaining persistent are attributes of all great pentesters. Throughout the course of my pentesting career, I've discovered that those three actions have paid off more times than any other efforts I've given. The thrill of compromising a host or network that you've laid many hours into understanding is unmatched. This blog post is a story about how I discovered and exploited a built-in functionality to gain code execution on a ScienceLogic SL1 appliance during a data center pentest, which ultimately led to the total compromise of an extremely large network.
Before I share about the techniques used, I must pretext with some messages sent to ScienceLogic regarding the issues that I discovered. My emails, support tickets, tweets, and phone calls were disregarded entirely which leads me to simply releasing this information along with SL1Pwn - a proof-of-concept to dump all stored (automation) credentials as well as gain code execution on SL1 devices. These actions are executed from an authenticated perspective.
Note: Currently there are ~600-700 SL1 devices internet exposed which can easily be found on Shodan. This may not seem like many but when you consider the purpose of this device and how it may be interconnected to every "tunnel" in a network, it's not good.
Debrief of SL1
SL1 was built to "enable intelligent automated IT operations (AIOps)" meaning there are thousands of functionalities built into the device which aggregate and analyze data as well as execute automated actions based on retrieved information. The only way to obtain this data is to be interconnected with systems from which analytics must be retrieved. SL1 appliances are special because the devices they communicate with are often incommunicable from the majority of the network. A very high level of trust is placed on SL1 appliances with the expectation that they are unlikely to be compromised but as we know, a device is only as secure as the administrator's efforts to secure it.
For an attacker, central and feature-rich devices like this are high-value targets. This is because compromising it could mean a great deal when trying to unlock doors in a network. Considering that SL1 is licensed (less attackers will have access), my guess is that there are many more issues to be found in the platform. A quick search for any CVE's relating to SL1 reveals 0 assignments. What does that say to you? (Leave a response as a comment please!)
Note: This device reminds me of the Offensive Security's OSCE (pre-OSCE3) HP OpenView exercise. OpenView is similar in many ways due to it's monitoring functionality. OffSec discovered a buffer overflow vulnerability in OpenView during a pentest ultimately resulting in remote code execution on the device!
Attackers Point-of-View 😈
Back to the story - the discovery of this device on the data center network was simple. SL1 runs on port 80 or 443 which reveals a login portal when visited. Any pentester's dream is to locate an obscure device with default credentials right? At least for me it is because many times these devices are quite interesting and often lead to greater compromise of the client's network.
The first step after discovering an unfamiliar portal is to search for default credentials! A Google search reveals that SL1 default credentials are em7admin:em7admin. Upon attempting this login, you may find that the credentials are valid and will be presented with a clunky appearing UI like so:
With SL1 access obtained, you will notice hundreds of buttons which do unpredictable things. After digging around, I eventually stumbled upon the "System" tab which at the left column shows a "Credentials" section. When I first visited the page I was very confused but quickly realized that there are many benign entries which are examples of how admins may store different types of credentials (SSH, SNMP, SMTP, AWS, etc). These stored credentials serve the purpose of enabling SL1 scripts to authenticate to remote services for data aggregation and more! The credentials are stored encrypted in the SL1 database however when accessed via the UI or API are unencrypted.
The classic change the "type='password'" to "type='text'" trick worked to expose "hidden" credentials pre-populated in the UI (thank you to that kid in high-school who taught me that trick) 😅. I eventually found out the API uses the same credentials as the front end (not an API key) so I could query all the credz at once! W00t.
Remote Code Execution
Gaining credentials would be great if all I had to do to pwn remote hosts was login. Remember how I can't access these normally "incommunicable hosts" though? Makes the creds not as good of a grab when you can't use them. My next goal after obtaining credentials was obvious: compromise the SL1 appliance and pwn all the things using the creds I just discovered.
My first instinct was to look for any vulnerabilities in SL1 that I might be able to leverage for RCE. I found a couple file upload content/restriction bypasses but nothing turned up RCE'able (yet). I resorted to reading a bunch of SL1 documentation which eventually led me to Run Books! These Run Books "allow you to specify actions you want SL1 to execute automatically when specific event conditions are met". That sounded interesting!
Luckily, SL1 Run Books are entirely Python3 based. Some built-in Run Books show examples of emails being sent, sending alerts to various platforms, pinging devices, logging into remote hosts to execute a command (if SSH credentials were stored for that device), pretty much anything you can think of. This was a very ripe functionality ready to be abused by a shell hungry pentester.
Next was to figure out how to create and execute a Run Book. This was very simple thanks to the instruction manual however I quickly recalled the Run Book description "when specific event conditions are met" when I couldn't seem to get my Run Book to execute. The conditional event system determines that Run Books will only execute when a monitored host triggers an alert. That doesn't help me because I don't know when the next alert will be triggered. From the operational security standpoint of an attacker, creating a condition which applies to every device in SL1 could be quite noisy and may cause adverse affects like Denial of Service if thousands of alerts are triggering rapidly.
After digging deeper, I managed to manually trigger a Run Book which just came down to some slight Run Book "schedule" configuration changes. Let's walk through the steps to create a manually trigger-able Run Book.
1. Navigate to the "Registry" tab then click "Actions"
2. Set the action type to "Run a Snippet" and set the action name. Set the python code to whatever you want the SL1 device to execute. I set mine to a reverse shell.
3. Next we want to create a schedule from the "Schedules" tab in "Registry". The key here is to place the "Start Time" to the past and do not set any other fields other than the schedule name.
4. Finally we want to create an "Action Policy" from the "Registry" tab. In the editor, select a policy type of "Scheduled". Set "System" or any other device in the available devices then click the arrow to move it over. Select your created schedule and your snippet as the action and move them over. It does not matter what "Organization" is set to so just leave it on the default.
5. It's go time! Make sure all your additions are saved. Navigate to "Actions" from "Registry" then find your action. Click the little lightning bolt to the far right of the action. Confirm the execution by clicking "Ok" on the prompt. Your action will run and you will gain code execution on the SL1 appliance!
With these steps executed in proper order, your snippet will run within ten seconds (results may vary) of executing the Run Book! If your listener is setup prior to execution, you will catch your shell in the context of the "s-em7-core" user account. This account is a low privilege user (thank goodness) however if you're lucky you may be able to trivially privilege escalate using the same default credentials as before by running "su em7admin" and entering the password 😊.
SL1 appliances run a Fedora operating system which means that all your typical scripts should run normally as expected on any other Linux system.
If for some reason the default password is changed, try visiting the same appliance on port "7700" and you may be able to use the default em7admin:em7admin combo here. This configuration interface exposes clear-text credentials for the back-end MySQL database in the "silo.conf" tab at the bottom of the "Device Settings" page.
If you're incredibly lucky and can access SL1's database port 3306 you may be able to login with root:em7admin. From there you can reset the em7admin's front-end password by running the command: SET PASSWORD FOR 'em7admin'@'localhost' = PASSWORD('thanks_sm00v');. You may have just asked yourself if the database stores the LDAP integrated passwords for every user account connected to SL1 for which the answer is yes. Dump the MySQL database hashes as usual and get to cracking 🤠.
Protect this device from internet exposure.
Change the em7admin password and disable the em7admin account after setup.
Ensure port 7700 is also protected as this is the the database configuration utility.
Change the default credentials for the configuration utility and API as these differ from the main login.
Turn on MFA (built-in) on this device.
Monitor logins routinely.
Keep your appliance up-to-date. At the time of this blog, the most recent version is 11.3.1.
Consider running an anti-virus on the underlying OS.
Email ScienceLogic and ask them to create a vulnerability disclosure program.
Vincent Mentz - @sm00v