IPv6 on AWS

IPv6 is finally gaining some momentum, thanks to support from several public cloud vendors and data center players in the recent months. Beyond the infrastructure players, the slow migration of several ISPs and corporations towards IPv6 is evident in Google’s IPv6 traffic statistics:

Raju Alluri: IPv6 adoption over years, courtesy Google

Hovering around 13%, the adoption rate is impressive in the recent months and support from players like AWS is going to improve this metric over the next few months.

It is also interesting to see that India is reasonably ahead in this initiative, as seen in Google’s stats:

Raju Alluri: IPv6 adoption by country, courtesy Google

AWS published its support for IPv6 in EC2 around 1st December 2016 and by January this year, the support is extended to 15 of its global regions. During those weeks, the dual-stack support is also extended to ELBs (load balancing), Route 53 (DNS), public VIFs, S3, IoT and individual CloundFront distributions.

One key thing to note is that the IPv6 addresses assigned are internet-routable and it needs an Egress-only Internet Gateway if any if the VIFs don’t need to get exposed to internet and have to remain in a private network.

Simple yet strong steps towards IPv6 adoption!

Shellshock bug and the risks

Bash, the quarter century old shell utility on almost all popular unix based systems, is found to be vulnerable. The exploit works by injecting specially crafted values into an environment variable and using it to invoke a shell command. Once the exploit gets to that level, there is hardly any limit on what can be executed as part of the shell command.

The problem gets worse for the fact that many of the day to day usages of the network facing services have potential to use bash internally. For example, CGI scripts on web servers, convenience utilities offered by network routers and any other limited command execution tools might be the key vulnerability public and guest access private networks. Mitre warns that sshd with ForceCommand is a potential attack vector.

The bug is being termed as Shellshock bug or bash bug. RedHat’s security blog article is one of the earliest articles that discussed the Shellshock bug in detail. Robert Graham of Errata Security is the best known tracker of the issue and has ongoing observations and comments on his blog/twitter account.

Here is how you can check if the current bash is vulnerable on your system. If it prints vulnerable on the first line, then patch your bash package.

$ env x='() { :;}; echo vulnerable' bash -c "echo test completed"
 vulnerable
 test completed

For web servers, here is the test suggested:

$ curl -i -X HEAD "http://sometestdomainhere.com/" -A '() { :;}; echo "Warning: Server Vulnerable"'

The output looks somewhat like the following listing. If it contains “Warning” text, then it is highly likely that the web server’s bash is (and cgi’s based on bash are) vulnerable. This test doesn’t assure that the system is not vulnerable. You may still have other CGIs run with bash that are vulnerable.

HTTP/1.1 200 OK
Date: Fri, 26 Sep 2014 02:51:52 GMT
Server: Apache
X-Powered-By: PHP/5.4.32
X-Pingback: http://sometestdomainhere.com/xmlrpc.php
Link: <http://sometestdomainhere.com/?p=14>; rel=shortlink
Content-Type: text/html; charset=UTF-8

Since the Shellshock bug existed for quite a while, all versions of bash that are currently out there in active usage are likely to be vulnerable. Patching some of these devices might be trivial, but there still might be several other devices that are hard to patch.

  • Servers that run services like web/ftp might be vulnerable if the CGI scripts end up using bash. Invoking bash from PHP code is considered not vulnerable, unless there are ways to circumvent input parameter validations of the PHP code. The RedHat article mentioned above has links to instructions on how to fix this on RedHat variants of linux. For Ubuntu, this is a good thread to follow.
  • Desktops that use network facing services like DHCP over wireless and sshd are vulnerable as long as these services internally use bash commands or bash as the shell for the session. There are still discussions on whether Mac OS X DHCP is vulnerable or not, because Apple modified its DHCP and claims that the DHCP utilities don’t use bash internally. Mac OS X branched version 3 of bash and does its own updates to the shell. There are instructions on how to patch OS X, tailored more for unix admins (and requires xcode) than normal users.
  • There are some suggestions on renaming bash to a different name, but that might break more things than fixing them. Use this technique with utmost caution.
  • ¬†Beyond Desktops and Servers, devices like internet routers may have vulnerabilities due to utilities and services they offer. For these devices, waiting for vendor released patches is the best option, but explore the possibility of turning off these convenience utilities.

Errata Security also has notes on wormable nature of the Shellshock bug. So patch your bash package as early as you can.

Upcoming AWS / EC2 instance reboot

If you are using AWS and EC2 instances, a reboot of most those instances is on the horizon. Amazon’s AWS informed of this reboot that is scheduled between 02:00 GMT on September 26th and 23:59 GMT on September 30th.

Read more about this reboot on Gigaom and Rightscale. Technical Forums on AWS and other sites are already buzzing with lot of traffic, discussing the potential impact and how to ensure that the services are not impacted.

Given the urgency and magnitude of the instances that are impacted, it looks like the patch is going potentially going to address a security vulnerability. The actual details of the patch and the issues that are fixed by it will be known around October 01st.

Summarizing various discussions on related forums, here is a quick summary of what to watch out for during this AWS / EC2 instance reboot

  • The reboot is not limited to any single availability zone. It spawns across all the availability zones
  • Good news is that the EC2 instances on all availability zones are not rebooted at the same time. So if your instances spawn across multiple availability zones, you are on a relatively safer side.
  • The reboot does not impact instances of the type T1, T2, M2, R3, and HS1. However, if the patch fixes issues on these instance types too, then you might be on your own. We will know more around October 1st.

Here are a few quick checks for those who are getting impacted.

  • Check your mailbox for a notice from AWS and it is likely to give more details about the reboots, impact and schedules
  • Ensure that the key services on your instances are configured for auto restart when the system boots up. It looks silly, but I have seen code that takes good care of newly spawned instances but doesn’t address reboots that well.
  • Ensure that your network paths (non-Elastic IPs, Route 53 entries, S3 buckets) survive reboot of the instances.
  • For those whose instances are NOT rebooted by AWS, watch out for the issues fixed by AWS during this reboot and evaluate their impact on your instances. Take corrective measures as soon as possible.

For those who can afford to be heroic enough – why wait till AWS reboots your instances? Reboot these on your own in each availability zone and test the resilience.

Costs of Data Center Outages

This post at VirtualHosting.com discusses several interesting metrics and forecasts. They include

  • Data transfer metrics from/to Data Centers
  • Top causes for Data Center Downtimes
  • Top Cloud Outages due to Data Center Downtimes
  • Projected outage costs

As cloud based services become more and more prominent, the reliance on data centers is pushed from end customers to cloud service providers. That way, data center outages are going to get more visibility while they are hosting cloud service providers than while hosting end customers.

In other words, while consumers appear to be moving more towards cloud based services, the cloud based services themselves become more reliant on the data center services that cut across multiple geographic regions.

Here is the graphic for quick access, from VirtualHosting.com

Data Centers Downtime
Data Centers Downtime

Netflix Open Source – Inauguration

Today, Netflix conducted the inaugural open house of its opensource applications.This is the first ever public event of Netflix in its opensource efforts that started almost three years ago.

 

Netflix OSS Inauguration

There are several opensource projects that Netflix lined up for this event. Here are the ones I personally liked

  • Asgard: Web application for application deployment and cloud management. I like the simplicity and focus of this web application. Primarily speaking the AWS language, this is a great tool.
  • Denominator: This project doesn’t feature yet on Netflix’s github page (at least at the time of the writing) but this is something very close to the challenge I face regularly – How to ensure the name services integrity and availability in business continuity situations.

Visit Netflix’s opensource page at http://netflix.github.com/. This site presents you with a familiar Netflix streaming interface to the company’s new baby.

Netflix OSS

The next meetup is scheduled for March 13th.