authbind vs iptables on AWS

Here is a short description of the scenario I was working on. I am using a standard AWS AMI to run tomcat (tomcat7, to be specific.) The default configuration of AWS AMIs (and many other off-the-shelf unix based servers) is such that tomcat (or any other program that runs with a non-superuser credentials) can’t bind to privileged ports. However, tomcat needs to use these privileged ports (443 for TLS and 80 for standard HTTP) to serve public facing pages.

Making tomcat run as superuser is really a bad idea (the why question is beyond this article.) So there are a few tricks to make tomcat work on privileged ports.

authbind

There is lot of mindshare around authbind when it comes to hosted environments. The manpage of authbind describes how authbind can be used to make a program bind to sockets on privileged ports. However, if you are using a standard AWS AMI, you may have some challenges using authbind. Also, for automated environments (read Chef) in AWS, I felt authbind is more complicated to work with.

iptables

Port redirection using NAT features of iptables is very simple and straight forward. However, it requires an additional configuration on tomcat to use proxy mode on privileged ports.

Here is the NAT configuration using iptables.

sudo /sbin/iptables -t nat -I PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080
sudo /sbin/iptables -t nat -I PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 8443
sudo service iptables save

Once this is done, all inbound traffic on 80 will be redirected to 8080. The same is the case with port pair 8443 and 443. This way, tomcat can still bind to port 8080 for HTTP and 8443 for TLS while serving incoming connections on 80 and 443 respectively.

When a client program queries the port information from tomcat, it should respond with port 80 and 443 instead of 8080 and 8443. To ensure that, one can use the proxy support feature of tomcat. Here is the additional configuration in tomcat connector settings in server.xml

<Connector port="8443" proxyPort="443" .../>
<Connector port="8080" proxyPort="80" .../>

Other Considerations

There are better ways to handle this port redirection when you have front-ending loadbalancers and/or proxy servers in place. Having proxy/loadbalancers solves helps mitigate more issues than just solving the redirection problems. However, the iptables approach is better than authbind approach when you are using a single server on AWS without lot of additional infrastructure and configurations in place.

Shellshock bug and the risks

Bash, the quarter century old shell utility on almost all popular unix based systems, is found to be vulnerable. The exploit works by injecting specially crafted values into an environment variable and using it to invoke a shell command. Once the exploit gets to that level, there is hardly any limit on what can be executed as part of the shell command.

The problem gets worse for the fact that many of the day to day usages of the network facing services have potential to use bash internally. For example, CGI scripts on web servers, convenience utilities offered by network routers and any other limited command execution tools might be the key vulnerability public and guest access private networks. Mitre warns that sshd with ForceCommand is a potential attack vector.

The bug is being termed as Shellshock bug or bash bug. RedHat’s security blog article is one of the earliest articles that discussed the Shellshock bug in detail. Robert Graham of Errata Security is the best known tracker of the issue and has ongoing observations and comments on his blog/twitter account.

Here is how you can check if the current bash is vulnerable on your system. If it prints vulnerable on the first line, then patch your bash package.

$ env x='() { :;}; echo vulnerable' bash -c "echo test completed"
 vulnerable
 test completed

For web servers, here is the test suggested:

$ curl -i -X HEAD "http://sometestdomainhere.com/" -A '() { :;}; echo "Warning: Server Vulnerable"'

The output looks somewhat like the following listing. If it contains “Warning” text, then it is highly likely that the web server’s bash is (and cgi’s based on bash are) vulnerable. This test doesn’t assure that the system is not vulnerable. You may still have other CGIs run with bash that are vulnerable.

HTTP/1.1 200 OK
Date: Fri, 26 Sep 2014 02:51:52 GMT
Server: Apache
X-Powered-By: PHP/5.4.32
X-Pingback: http://sometestdomainhere.com/xmlrpc.php
Link: <http://sometestdomainhere.com/?p=14>; rel=shortlink
Content-Type: text/html; charset=UTF-8

Since the Shellshock bug existed for quite a while, all versions of bash that are currently out there in active usage are likely to be vulnerable. Patching some of these devices might be trivial, but there still might be several other devices that are hard to patch.

  • Servers that run services like web/ftp might be vulnerable if the CGI scripts end up using bash. Invoking bash from PHP code is considered not vulnerable, unless there are ways to circumvent input parameter validations of the PHP code. The RedHat article mentioned above has links to instructions on how to fix this on RedHat variants of linux. For Ubuntu, this is a good thread to follow.
  • Desktops that use network facing services like DHCP over wireless and sshd are vulnerable as long as these services internally use bash commands or bash as the shell for the session. There are still discussions on whether Mac OS X DHCP is vulnerable or not, because Apple modified its DHCP and claims that the DHCP utilities don’t use bash internally. Mac OS X branched version 3 of bash and does its own updates to the shell. There are instructions on how to patch OS X, tailored more for unix admins (and requires xcode) than normal users.
  • There are some suggestions on renaming bash to a different name, but that might break more things than fixing them. Use this technique with utmost caution.
  •  Beyond Desktops and Servers, devices like internet routers may have vulnerabilities due to utilities and services they offer. For these devices, waiting for vendor released patches is the best option, but explore the possibility of turning off these convenience utilities.

Errata Security also has notes on wormable nature of the Shellshock bug. So patch your bash package as early as you can.

Upcoming AWS / EC2 instance reboot

If you are using AWS and EC2 instances, a reboot of most those instances is on the horizon. Amazon’s AWS informed of this reboot that is scheduled between 02:00 GMT on September 26th and 23:59 GMT on September 30th.

Read more about this reboot on Gigaom and Rightscale. Technical Forums on AWS and other sites are already buzzing with lot of traffic, discussing the potential impact and how to ensure that the services are not impacted.

Given the urgency and magnitude of the instances that are impacted, it looks like the patch is going potentially going to address a security vulnerability. The actual details of the patch and the issues that are fixed by it will be known around October 01st.

Summarizing various discussions on related forums, here is a quick summary of what to watch out for during this AWS / EC2 instance reboot

  • The reboot is not limited to any single availability zone. It spawns across all the availability zones
  • Good news is that the EC2 instances on all availability zones are not rebooted at the same time. So if your instances spawn across multiple availability zones, you are on a relatively safer side.
  • The reboot does not impact instances of the type T1, T2, M2, R3, and HS1. However, if the patch fixes issues on these instance types too, then you might be on your own. We will know more around October 1st.

Here are a few quick checks for those who are getting impacted.

  • Check your mailbox for a notice from AWS and it is likely to give more details about the reboots, impact and schedules
  • Ensure that the key services on your instances are configured for auto restart when the system boots up. It looks silly, but I have seen code that takes good care of newly spawned instances but doesn’t address reboots that well.
  • Ensure that your network paths (non-Elastic IPs, Route 53 entries, S3 buckets) survive reboot of the instances.
  • For those whose instances are NOT rebooted by AWS, watch out for the issues fixed by AWS during this reboot and evaluate their impact on your instances. Take corrective measures as soon as possible.

For those who can afford to be heroic enough – why wait till AWS reboots your instances? Reboot these on your own in each availability zone and test the resilience.