Decrypting FortiGate passwords (CVE-2019–6693)

10 min readJan 12, 2020

Did you ever wonder about the following type of passwords in FortiGates?

config wireless-controller vap
    edit "dummy-decrypt"
        set vdom root
        set passphrase ENC umGOJVCWhGhoiuY/EjTZcZKjuuIkusDNkvdvUkU3awr5TGudxfmidR2bOyoBlQgHho0DuORJafh1WiCzaoBpRNv/gHCFC5mlPVcjjpHXTUvG47/qlBusgELO1ctsLt/4RVjov2S5R7+6DdkU/PbSZVoNkeINDQBsP3TTmxEz9+YyPleLzBZh4RKU2OKTsqe6TF/uHA==
    next
end

The very first time we saw a FortiGate configuration file (2014), we knew it had to be reversible encryption and we therefore knew we could reverse it, too! This story is all about “the proof of the pudding is in eating”.

Reversible encryption versus one way encryption

A common misconception is that the passwords are hashed, like in https://forum.fortinet.com/tm.aspx?m=46528. User rwpatterson writes: “ The password is hashed, and I don’ t believe there is any way to reverse engineer it.”

If the FortiGate cannot decrypt the password, then how can it show the password in the GUI? Remember that restoring a configuration file, well, restores the configuration, even on a different FortiGate unit. The configuration file must have all details. If a password is hashed (one way encryption) then the FortiGate cannot decrypt it. Period.

Perhaps we have to convince you a bit more: what about PPPoE passwords? A FortiGate has to provide the actual password to the Internet provider. If the password was hashed in the configuration file, then the FortiGate cannot decrypt it.

So, the password is stored in an encrypted form and every FortiGate knows how to decrypt it. We just have to find out how a FortiGate does this…

FortiVM

A physical FortiGate is far better shielded from tampering than a FortiVM. We never tried to open a physical one, but we bet there is no JTAG connector nor other debugging features. A FortiVM, however, runs within our Linux environment, with various tool chains at our disposal. Fiddling with a FortiVM was the logical choice.

Building on prior work

On https://www.cnblogs.com/studyskill/p/6524672.html (“Backdooring a FortiOS VM”), you can find commands to fiddle with the FortiVM image. (FortiGate fixed this approach in CVE 2019–5587.) The tl;dr is:

We can install any binary we want.
We overwrite smartctl with busybox for a proper shell (ash).
We activate the shell via the CLI with diagnose hardware smartctl (or shorter: d h smartctl)

https://cookbook.fortinet.com/encryption-hash-used-by-fortios-for-local-pwdpsk/ also helped, by saying “The encoding consists of encrypting it with a fixed key using AES”. A fixed key? That’s good to know! And it is bound to be a very reliable source: Fortinet wrote it.

https://docs.fortinet.com/uploaded/files/3624/fortigate-hardening-your-fortigate-56.pdf confirms the fixed key: “The encoding consists of encrypting the password with a fixed key using DES (AES in FIPS mode) and then Base64 encoding the result.”

Finally, the US government (FIPS 140) also chips in some intel in https://csrc.nist.gov/csrc/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp2765.pdf: “IKE Pre-Shared Key (…) AES encrypted”

Toolchain

Since we were free to choose our own toolbox, we just used what we are familiar with:

strace
gdb
objdump

objdump: Obtaining the disassembly

objdump was our friend for obtaining the disassembly of all binaries. Since FortiGate basically has just one binary (init), we disassembled that binary:

objdump\
  --disassemble-all\
  --reloc\
  --dynamic-reloc\
  --syms\
  --dynamic-syms\
  --wide\
  init > init.asm

That produced roughly 700 MiB of disassembly. We used it for searching and finding interesting functions and strings (but that road failed). We used it as a reference during the debugging with gdb: always helpful to view the disassembly in another window than gdb.

Symmetric cipher

The online documentation told us we are dealing with AES encryption. AES is a symmetric cipher, meaning that the same key is used for both encrypting as decrypting. We are not sure when a FortiGate decrypts a password, but we do know when it encrypts one: during a configuration change! Every clear passwords needs to be encrypted before writing.

gdb: Sniffing a configuration change (1)

Every configuration change you make in the GUI or CLI, gets saved immediately. If we could only listen to what the FortiGate does when saving… Well, we can with gdb. To “listen” to every function call, you can use rbreak .: break on regexp ‘.’, i.e., break on every function call.

We first have to know which process governs the saving. That turns out to be cmdbsvr (rather obvious with 20/20 hindsight). We attached gdb to this process, issued the rbreak . and got some 20K breakpoints. We are definitely not using the sharpest tool in the shed (understatement). It hits breakpoints very often of course, so we weed out every false positive with delete num. Tedious work, yet we got there. We reached functions starting with EVP_.

OpenSSL

Just use your favourite search engine and search for “evp linux”. You will get results about “high level cryptographic functions” and “EVP Symmetric Encryption and Decryption — OpenSSLWiki”.

So, functions starting with EVP_ are from the OpenSSL library. Sweet. Which functions are used to encrypt AES? We found examples on https://wiki.openssl.org/index.php/EVP_Symmetric_Encryption_and_Decryption. The function that needs the AES key as argument, is EVP_EncryptInit_ex.

Sniffing a configuration change (2)

We already found EVP_functions with gdb. We can either continue with our blunt “ rbreak .” or start a new one with a break on EVP_EncryptInit_ex. We opt for the latter:

(gdb) break EVP_EncryptInit_ex
Breakpoint 1 at 0x7f23a78c2c70
(gdb) continue
Continuing.Breakpoint 1, 0x00007f23a78c2c70 in EVP_EncryptInit_ex ()
   from /fortidev4-x86_64/lib/libcrypto.so.1.1
(gdb) bt
#0  0x00007f23a78c2c70 in EVP_EncryptInit_ex ()
   from /fortidev4-x86_64/lib/libcrypto.so.1.1
#1  0x0000000001babe48 in ?? ()
#2  0x00000000017fd6ac in ?? ()
(...)

The call to EVP_EncryptInit_ex was made by the instruction before 0x0000000001babe48. The offset starts with 1babe, we must be getting close if we find babes ;-). Let’s switch to the calling frame and view the code up to the call to EVP_EncryptInit_ex:

(gdb) frame 1
#1  0x0000000001babe48 in ?? ()(gdb) disassemble $pc-41, $pc
Dump of assembler code from 0x1babe1f to 0x1babe48:
   0x0000000001babe1f: callq  0x4256a0 <EVP_CIPHER_CTX_new@plt>
   0x0000000001babe24: test   %rax,%rax
   0x0000000001babe27: mov    %rax,%r12
   0x0000000001babe2a: je     0x1babf70
   0x0000000001babe30: callq  0x424060 <EVP_aes_128_cbc@plt>
   0x0000000001babe35: xor    %edx,%edx
   0x0000000001babe37: mov    %rsp,%r8
   0x0000000001babe3a: mov    %r14,%rcx
   0x0000000001babe3d: mov    %rax,%rsi
   0x0000000001babe40: mov    %r12,%rdi
   0x0000000001babe43: callq  0x422910 <EVP_EncryptInit_ex@plt>
End of assembler dump.
(gdb)

There is no data pushed to the stack, so arguments are probably provided via registers. Not having profound assembly knowledge, we simply try all the registers that were loaded prior to the call to EVP_EncryptInit_ex:

(gdb) x/16bx $r8
0x7fffde603050: 0x0d 0xb4 0x82 0xbb 0x00 0x00 0x00 0x00
0x7fffde603058: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
(gdb)
(gdb) x/16bx $rcx
0x7fffde603060: 0x52 0x65 0x64 0x61 0x63 0x74 0x65 0x64
0x7fffde603068: 0x50 0x61 0x73 0x73 0x77 0x6f 0x72 0x64
(gdb)
(gdb) x/16bx $rsi
0x7f23a7bcf060: 0xa3 0x01 0x00 0x00 0x10 0x00 0x00 0x00
0x7f23a7bcf068: 0x10 0x00 0x00 0x00 0x10 0x00 0x00 0x00
(gdb)

Observe that all but one register shows null bytes. Having many null bytes doesn’t seem random and doesn’t seem human too; not likely for a very secure AES key. So, register rcx might hold the one key. Let’s investigate:

(gdb) x/1s $rcx
0x7fffde603060: "RedactedPassword\340\066`\336\377\177"
(gdb)

Fortinet asked us kindly to not reveal the actual password: “(…) we would appreciate it if you don’t mention the actual key in your post; that’s for the safety of people who are not going to upgrade, for some reason (unaware of the issue, too old hardware, not willing to touch the production environment, etc…”. Therefore “RedactedPassword” is not the actual password. (The hex version isn’t the actual password too).

Nevertheless, you now know how we figured out the one key.

Where is this password in the binary?

If you simply “grep” on the fixed key in the binary or in the disassembly, you will notice that the password is not in it. Well, not in plain text form. Fortinet must have disguised it somewhere… Let’s find it!

Recap: EVP_EncryptInit_ex() uses register rcx for the AES key. Corresponding disassembly:

   0x0000000001babe3a: mov    %r14,%rcx ;; rcx = r14
   0x0000000001babe3d: mov    %rax,%rsi
   0x0000000001babe40: mov    %r12,%rdi
   0x0000000001babe43: callq  0x422910 <EVP_EncryptInit_ex@plt>

On offset 0x1babe3a, register rcx is filled with the contents of register r14. But when is register r14 filled with the one key? Let’s try something obvious: check when register r14 is explicitly changed in the disassembly just before 0x1babe3a. That turns out to be:

   0x0000000001babdcc: mov    %r14,%rdi ;; r14 = rdi

So, we break’d gdb after this instruction and asked for the contents of where register r14 points to:

break *0x1babdcf
x/16bx $r14

Too bad, jibberish, but it did narrow our search tremendously. The difference (offset wise) between 0x1babdccand 0x1babe3a is only 110. The only call and jump commands in between are:

1babdd6: e8 f5 fe ff ff callq 1babcd0 <conf_end@@Base+0x3dad40>
1babe00: e8 6b 6b 87 fe callq 422970 <memcpy@plt>
1babe0d: e8 8e 7d 87 fe callq 423ba0 <memset@plt>
1babe1a: e8 f1 ee 15 00 callq 1d0ad10 <wl_ah_getChipPowerLimits@@Base+0x3ea10>
1babe1f: e8 7c 98 87 fe callq 4256a0 <EVP_CIPHER_CTX_new@plt>
1babe2a: 0f 84 40 01 00 00 je 1babf70 <conf_end@@Base+0x3dafe0>

One of these instructions must have changed register r14 to point to the AES key. With 20/20 hindsight, there is one very obvious candidate, yet we used:

break *0x1babdcf
display/3i $pc
cont
stepi
x/16bx $r14

This means we start at the last known wrong value in register r14, display the current instruction and the next two, wait for the break, step to the next machine instruction (even into function calls) and show the contents of where register r14 points to.

We continued issuing

stepi
x/16bx $r14

And this is where the hex version of “RedactedPassword” showed up:

(gdb) stepi      
0x00007f23a7f30d6e in __memcpy_ssse3_back ()
   from /fortidev4-x86_64/lib/libc.so.6
1: x/3i $pc
=> 0x7f23a7f30d6e <__memcpy_ssse3_back+9262>: retq   
   0x7f23a7f30d6f <__memcpy_ssse3_back+9263>: nop
   0x7f23a7f30d70 <__memcpy_ssse3_back+9264>: lddqu 0x7f(%rsi),%xmm0
(gdb) x/16bx $r14
0x7fffde601d40: 0x52 0x65 0x64 0x61 0x63 0x74 0x65 0x65
0x7fffde601d48: 0x50 0x61 0x73 0x73 0x77 0x6f 0x72 0x64
(gdb)

retq is about to be executed, but the instruction before retqchanged r14. So we want to see what happened before retq:

(gdb) x/19i $pc-87
0x7f23a7f30d17 <__memcpy_ssse3_back+9175>: retq   
0x7f23a7f30d18 <__memcpy_ssse3_back+9176:> nopl   0x0(%rax,%rax,1)
0x7f23a7f30d20 <__memcpy_ssse3_back+9184>: lddqu  0x70(%rsi),%xmm0
0x7f23a7f30d25 <__memcpy_ssse3_back+9189>: movdqu %xmm0,0x70(%rdi)
0x7f23a7f30d2a <__memcpy_ssse3_back+9194>: lddqu  0x60(%rsi),%xmm0
0x7f23a7f30d2f <__memcpy_ssse3_back+9199>: movdqu %xmm0,0x60(%rdi)
0x7f23a7f30d34 <__memcpy_ssse3_back+9204>: lddqu  0x50(%rsi),%xmm0
0x7f23a7f30d39 <__memcpy_ssse3_back+9209>: movdqu %xmm0,0x50(%rdi)
0x7f23a7f30d3e <__memcpy_ssse3_back+9214>: lddqu  0x40(%rsi),%xmm0
0x7f23a7f30d43 <__memcpy_ssse3_back+9219>: movdqu %xmm0,0x40(%rdi)
0x7f23a7f30d48 <__memcpy_ssse3_back+9224>: lddqu  0x30(%rsi),%xmm0
0x7f23a7f30d4d <__memcpy_ssse3_back+9229>: movdqu %xmm0,0x30(%rdi)
0x7f23a7f30d52 <__memcpy_ssse3_back+9234>: lddqu  0x20(%rsi),%xmm0
0x7f23a7f30d57 <__memcpy_ssse3_back+9239>: movdqu %xmm0,0x20(%rdi)
0x7f23a7f30d5c <__memcpy_ssse3_back+9244>: lddqu  0x10(%rsi),%xmm0
0x7f23a7f30d61 <__memcpy_ssse3_back+9249>: movdqu %xmm0,0x10(%rdi)
0x7f23a7f30d66 <__memcpy_ssse3_back+9254>: lddqu  (%rsi),%xmm0
0x7f23a7f30d6a <__memcpy_ssse3_back+9258>: movdqu %xmm0,(%rdi)
=> 0x7f23a7f30d6e <__memcpy_ssse3_back+9262>: retq   
(gdb)

So the magic happened during this command:

0x7f23a7f30d6a <__memcpy_ssse3_back+9258>: movdqu %xmm0,(%rdi)

This is a double quad (dq) move: it copies 128 bit from register xmm0 to where register rdi points to. (Apparently, register rdi and r14 point to the same memory address.) But how did register xmm0 get filled? Well, the instruction prior to the movdqu is:

0x7f23a7f30d66 <__memcpy_ssse3_back+9254>: lddqu  (%rsi),%xmm0

This loads register xmm0 with where register rsi points to. Let’s check register rsi:

(gdb) x/16bx $rsi
0x32dd100: 0x52 0x65 0x64 0x61 0x63 0x74 0x65 0x65
0x32dd108: 0x50 0x61 0x73 0x73 0x77 0x6f 0x72 0x64

Yes, still the one key. But the address is the interesting part here: 0x32dd100. That value is hard coded(!); we found this in the disassembly:

 1babd25: be 00 d1 2d 03        mov    $0x32dd100,%esi
 1babd2a: e8 41 6c 87 fe        callq  422970 <memcpy@plt>

We decide to see what the content of 0x32dd100 right after a reboot (i.e., prior to a configuration save):

/ # gdb -p 104
GNU gdb (GDB) 7.10.1
(...)
0x00007f41d93c2993 in __select_nocancel () from /fortidev4-x86_64/lib/libc.so.6
(gdb) x/1s 0x32dd100 
0x32dd100:	"RedactedPassword"
(gdb)

This is strong evidence that the one key is loaded just once, right at the start of process cmdbsvr. Let’s look for more disassembly with magic value 0x32dd100:

 1babce6:       b9 6b d1 2d 03          mov    $0x32dd16b,%ecx
 ;; ecx = 0x32dd16b;
 ;; This is the last character of the AES key.
 
 1babceb:       0f 1f 44 00 00          nopl   0x0(%rax,%rax,1)     
 ;; No-op;
 ;; Might be to align to a 16 byte boundary.
 
 1babcf0:       44 0f b6 41 ff          movzbl -0x1(%rcx),%r8d
 ;; r8d = *(rcx - 1);
 ;; r8d contains the character just before the character rcx
 ;; points to.
 
 1babcf5:       44 30 01                xor    %r8b,(%rcx)
 ;; *rcx ^= r8b;
 ;; XOR the current character with the preceding character.
 
 1babcf8:       48 83 e9 01             sub    $0x1,%rcx
 ;; --rcx;
 ;; Go to preceding character. (Decrease loop variable.)
 
 1babcfc:       48 81 f9 00 d1 2d 03    cmp    $0x32dd100,%rcx   
 ;; if (rcx == 0x32dd100) { ... };
 ;; If we are at the start of the AES key, then ...

This is why we couldn’t find the fixed AES key with grep: the key is encrypted with “rolling XOR”…

You may have guessed it already: the rolling XOR encrypted form of the AES key is indeed in the .data section of the binary.

The fix

Fortinet published their fix in https://fortiguard.com/psirt/FG-IR-19-007:

config system global
  set private-data-encryption enable
end

The FortiGate then asks for a “data encryption key”. Fortinet notes:

This CLI option is disabled by default.

This implies that the one key is still used ifset private-data-encryption enable is not used. Since the one key has been in FortiOS for so long without admins asking to remove it, we think it is unlikely that the new command will be used much. And therefore, the fix wouldn’t be a true fix as 99,9% of the world will continue to use the one key…

The workaround

Common sense is always good. Just knowing that encrypted passwords can be decrypted is enough to keep you safe on forums: just remove any encrypted password before posting, as you would remove any other sensitive data.

Common sense also dictates that configuration files should be made available on a need-to-know basis; keep configuration back-ups in a safe, protected environment.

Responsible disclosure

We used responsible disclosure to get this vulnerability fixed. We would like to thank Fortinet for believing in responsible disclosure: their communication was always to the point, friendly, thankful and prompt. Thanks Nesrine.

Acknowledgement

The authors would like to thank Bas Wijnen for his help in reading and understanding (dis)assembly.

Timeline

[2018–12–13] The quest to find the one key begins.
[2019–01–06] We found the one key.
[2019–01–09] We reported our findings to psirt@fortinet.com
[2019–01–09] Fortinet confirms it is an actual vulnerability: “The security of AES algorithm relies on the security of the symmetric key so of course this is a vulnerability.”
[2019–01–10] We provided a short version of our discovery
[2019–01–21] Fortinet asked us politely: “How were you able to modify FortiVM to include other binaries ?”
[2019–01–22] We provided our entire shell script.
[2019–04–02] Fortinet let us know about spin-off CVE 2019–5587 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-5587)
[2019–05–22] Fortinet let us know about https://fortiguard.com/psirt/FG-IR-19-017 — “FortiOS VM images lack an integrity check of the file system at boot time” (the spin-off CVE)
[2019–06–28] Fortinet let us know that the actual problem is known as CVE 2019–6693 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=2019-6693) and that the problem is fixed in versions 5.6.10, 6.0.6 and 6.2.1.
[2019–08–19] Fortinet informed us about a delay: “This issue is now fixed in 5.6.11, 6.0.7 and 6.2.1 (6.0.7 instead of 6.0.6)
Versions 5.6.11 and 6.2.1 have been released but not version 6.0.7 which is scheduled to be released around Oct 07, 2019 — Oct 11, 2019.”
[2019–11–19] FortiGuard Labs published advisory https://fortiguard.com/psirt/FG-IR-19-007.
[2020–01–12] Publication of this article.