VPCs, Peering, Routing, and other AWS networking concepts

My topics today include:

  • VPC design and peering
  • Transient routing
  • Multicast, unicast, and broadcasting
  • Common networking questions

Everything here is pretty basic, but that doesn't mean even long time users can't be caught by simple slip ups. I recommend CloudFormation for building and managing your network stack.

VPC Design

Generally a VPC includes the following components:

  • VPC metadata
  • Subnets
  • Routing Tables
  • Internet Gateways
  • NAT Gateways
  • Security Groups

Optionally you might configure:

  • Elastic IPs
  • Network ACLs
  • VPC Endpoints
  • DHCP Options
  • VPN (Customer Gateways, Virtual Private Gateways, and VPN Connections)

Typically to configure a VPC you'll start by creating the VPC and enabling things like DNS resolution and DNS hostnames. Then you'll create a couple subnets, with the intent of having some public subnets and some private subnets. You'll create two routing tables for the VPC, one for the private subnets and one for the public subnets. Then create an Internet Gateway and attach it to the VPC. You'll need to configure the public subnet's routing table to have a default route ( pointing at the Internet Gateway. You can then create a NAT Gateway in a public subnet with an Elastic IP and configure the private subnet's routing table to have it's default route ( pointing at the NAT Gateway. For the public subnet's you'll probably want to consider enabling the auto assign public IP option.

Here's a table summarizing what each component is and what it does:

Table - VPC Components

Components Description
VPC Metadata The VPC metadata includes options such as ClassicLink, which DHCP Option set you've attached to your VPC, DNS resolution, DNS hostnames, and Flow Logs.
Subnets A subnet is a logical slice of your VPC. You can configure a subnet to be either public or private, to have Internet or not, depending on your needs.
Routing Tables By default a VPC can route in between it's subnets. In order to route to the Internet or to another VPC you'll need to add an appropriate route to your VPC's routing table.
Internet Gateways Creating an Internet Gateway and attaching it to a VPC enables you to configure Internet access for your instances by setting up a default route.
NAT Gateways You can create a NAT Gateway in a public subnet, and route traffic for private subnets through it.
Security Groups By default, you should use Security Groups to restrict traffic between your instances. Think of security groups like host level firewalls that you can make consistent for a cluster relatively easily. Security groups are stateful.
Elastic IPs Provisioning an Elastic IP allows you to retain an IP address across instance migrations, stops/starts, etc. Typically good for things that resolve by DNS to an IP address.
Network ACLs Similar to security groups, Network ACLs controls traffic at the VPC or network level. Think of an ACL like the border firewall and security groups are the host firewall.
VPC Endpoints Currently just used to provide private subnets access to S3 without needing Internet access.
DHCP Options DHCP provides a standard for passing configuration information to hosts on a TCP/IP network. The options field of a DHCP message contains configuration parameters. We can use DHCP to configure things like NetBIOS, NTP, DNS, and search domains.
Customer Gateway A device on premises, represented by a name, routing mode, and IP address. This is used to establish a VPN connection between your VPC and your data center.
Virtual Private Gateway Essentially a virtual endpoint that can connect to a customer gateway. A virtual private gateway is the router on the Amazon side of the VPN tunnel.
VPN Connection You can connect a Virtual Private Gateway and a Customer Gateway to form a VPN connection between the two devices. Once created, you can download a (probably) working configuration file directly.

VPC Peering

Configuring VPC peering is fairly simple, you can submit a request to peer VPCs on your account within a region and cross account within a region. The owner of the other VPC must accept the request. You can then set up routing in between the peered VPCs.

The most common mistake I see is when a user sets up peering and either forgets to configure the route tables or only configures the route tables on one of the two peered VPCs.

Note: You cannot peer inter-region. So VPC Ay on us-east-1 cannot be peered to VPC Az on us-west-1. Until Amazon releases an inter-region VPN service anyways.

Transient Routing

I actually have people ask transient routing questions quite frequently, often without them realizing what transient routing is.

Let's say we have VPC A, VPC B, and VPC C. VPC A is peered to VPC B. VPC B is peered to VPC C. Can VPC A send traffic to VPC C? No. VPC A can send traffic to VPC B, and VPC B can send traffic to VPC C but VPC A cannot send traffic to VPC C nor can VPC C send traffic to VPC A.

I want this to be really, really clear so we'll illustrate it with a table.

Table - VPC can send / receive traffic

VPC A Yes Yes No
VPC B Yes Yes Yes
VPC C No Yes Yes

Routing Protocols

If you're used to an on premises environment or are using software made prior to the huge cloud computing push, you might encounter limitations in regards to support for multicast, unicast, and broadcast traffic. If you're using software that usually relies on multicast of broadcast to discover other worker instances or similar, then you'll need to figure out a work around since those protocols aren't supported on AWS. Unicast does work, and if supported you can use some sort of EC2 discovery service and mime multicast / broadcast functionality and discover hosts that way.

Here's a quite summary:

Table - Routing Protocols

Routing Protocol Supported On Premises Supported on AWS Supported on Cloud Providers
Multicast Yes No No
Broadcast Yes No No
Unicast Yes Yes Yes

As stated above, you can use the EC2 API in order to mock the functionality of a protocol if you need it. Elasticsearch is a good example of this, they have a module for EC2 discovery.

Frequently Asked Questions

This list is a work in progress, and I might even append to it at times.

  1. Why isn't my VPC assigning my instances a public ip address?
    In the VPC console, navigate to your subnets. Select a subnet and click actions. There's an option to auto assign public ips to your instances. Ensure this is checked if you want public ips to be assigned by default.

  2. Why isn't my VPC assigning DNS names to my instances?
    In the VPC console, navigate to the VPCs page. Select a VPC and click actions. There are two options of interest here - "Edit DNS Resolution" and "Edit DNS Hostnames". You likely want to enable both of these options in most common scenarios.

  3. Why can't I access the Internet from my instance?
    Depending on whether you've created a public or private subnet for your instance, you'll need to confirm whether you're instance has a public IP, an Internet Gateway, and a route to the Internet Gateway in the subnet's routing table. Alternatively you'll have to ensure that you have a NAT Gateway or NAT instance with an appropriate route table entry. Missing or misconfigured components are typically the reason you can't access the Internet from an instance when you expected to.

  4. Why can't I find my Internet/NAT gateway?
    Assuming you created an Internet/NAT gateway, in the top left corner of the VPC console is a "filter by VPC" dropdown. If you have filtered by a VPC, when you create an Internet or NAT gateway they will immediately be filtered before you can assign it to a VPC. Switch the filter to "None" and your unattached gateway should be visible. This is when tagging your resources with names can make a huge difference!

  5. I added an Internet/NAT gateway but it disappeared?
    In the top left corner of the VPC console is a "filter by VPC" dropdown. If you have filtered by a VPC, when you create an Internet or NAT gateway they will immediately be filtered before you can assign it to a VPC. Switch the filter to "None" and your unattached gateway should be visible. This is when tagging your resources with names can make a huge difference!

  6. I added an Internet/NAT gateway but can't seem to ping the internet?
    You need to ensure a valid route for is correctly configured to route traffic to the gateway in your subnet's route table.

  7. I created a peering connection put can't ping instances?
    You probably either forgot to allow access through your instances security groups or you forgot to configure routing in the VPC routing tables in both of the peered VPCs.

  8. Is there a way to make transient networking possible?
    If you have an instance set up to receive and proxy your traffic it becomes possible to have traffic from from VPC A through a proxy on VPC B into VPC C.