I was recently playing around with the Traffic Mirroring feature in AWS. As a network geek, this is right up my alley because as some colleagues and I used to say, "the wire never lies!". Being able to pick packets off the wire for detailed inspection has saved the day many a time. Until Traffic Mirroring came along, it wasn't possible to do that in an Amazon VPC. Below are my notes and considerations for using this feature.

Facts

First, the stone cold truth.

  • Traffic Mirroring (TM) takes traffic seen on a given Elastic Network Interface (ENI) and mirrors it to a destination ENI (which should be attached to a compute instance with sniffer or security appliance software installed).
  • The two ENIs can be in different accounts, in the same account but different VPCs, or in the same VPC.
  • TM can mirror DNS traffic to/from the VPC+2 resolver but cannot mirror ARP, DHCP, or traffic to the link-local (169.254.x.x) metadata or NTP services.
  • Mirrored traffic is encapsulated in Virtual eXtensible Local Area Network (VXLAN) packets and routed to the target ENI.
    • VXLAN is only used as a dataplane transport; there's no VXLAN control plane involved.
    • The mirrored traffic arrives at the target ENI still encapsulated in VXLAN so the sniffer/appliance must know how to decapsulate such packets.
    • Because the VXLAN packet is routed to the target, there are implications for how Security Groups and routing tables must be configured (this is part of my notes below).
  • TM consumes network bandwidth on the target. If there's a mirror setup from a 10G ENI to a 1G ENI, that's just asking for trouble.
  • TM is only supported on Nitro-based EC2 instances. Update: TM is now supported on select non-Nitro instances.

Define interesting traffic

Traffic Mirroring has three basic constructs:

  1. A filter
  2. A target
  3. A session

A filter is mandatory as it defines the type(s) of traffic that Traffic Mirroring should grab and send to the target. Without a filter, TM will ignore all traffic.

A filter defines exactly what traffic type(s) to pay attention to or to ignore by specifying a set of rules. Rules have the following parameters:

  • Action: accept will cause TM to mirror matching packets; reject explicitly causes TM to ignore matching packets.
  • Protocol: The protocol ID of the Layer 4 protocol inside the IP packet.
  • Source/destination port range (optional).
  • Source/destination IP prefix (IPv4 or IPv6).

Filter rules are created for inbound and outbound traffic separately (where "in" and "out" are from the perspective of the source ENI).

I defined my filter in the AWS Console under VPC, Mirror Filters.

Define a target

The target is an ENI attached to whatever instance is supposed to receive the mirrored traffic. It could be an instance running a sniffer, a monitoring tool, or a security appliance.

It's important to understand the target is an ENI and not an instance. This provides choice. If the instance type supports it, a secondary ENI can be add to it just for use with TM. Secondary ENIs can also be detached from instances and attached to other instances which allows moving the target around. Or the primary ENI can be used as the target and because the mirrored traffic is encapsulated in VXLAN, there's no need to worry the mirrored traffic will co-mingle with non-mirrored traffic that's actually for the target instance.

Find the Interface ID of the ENI that is the target. I have a Windows instance acting as my sniffer. I go into EC2 in the AWS Console, select that instance, and click on eth0 in the instance detail pane. That brings up a modal with a bunch of information about the primary network interface including the Interface ID.

Copy the ID (eni-0c99 in the example) to a notepad.

In the AWS Console, I navigate to VPC then Mirror Targets where I choose Create traffic mirror target.

  • I entered a name and description for the target
  • I chose the network interface target by looking for the Interface ID in the drop-down list

Create a session

Lastly, I create a TM Session to start mirroring traffic.

I follow the same process as I did above for the target ENI in order to find the Interface ID of the source ENI (EC2, select instance, click eth0, copy the Interface ID).

In the VPC Console, I click Mirror Sessions, Create traffic mirror session and I fill in the form.

  • I name the session so I can refer to it later on by name.
  • I give it a description (this might be a good field for a ticket/incident number).
  • I scroll through the list in the Mirror source and find the Interface ID for my source ENI.
  • I select my previously defined target in the Mirror target drop down.

Under Additional settings, I choose the following:

  • Session number I leave at the default of 1 since this is the only session I'm creating.
    • If I was creating multiple sessions for the same source interface, I would pay closer attention to this value as it defines the order of evaluation of the sessions. A packet will only be mirrored one time. Priorities gives control over which of multiple sessions should govern for a given packet. More details in the documentation.
  • The VNI (VXLAN Network ID) is an identifier carried in the VXLAN header. If there are multiple sessions setup with the same target, setting a different VNI in each session allows the target to identify which session a mirrored packet belongs to.
  • Lastly, I select the Filter that I defined previously.

Breathe deep

With that complete, packets should now be mirrored to the target. My target ENI is attached to a Windows Server instance that I've installed Wireshark on.

Note a couple of things in the screenshot:

  • I've applied a display filter in Wireshark of vxlan.vni == 64. This lines up with the VNI I set in the TM session. If I had multiple sessions all using the same target, this is one example of how I could zero in on mirrored traffic related to a specific session.
  • The layers of packet headers shown in the bottom pane shows that inside the VXLAN packet is the original IP packet and Ethernet frame. TM grabs the Layer 2 Ethernet frame as well as the IP packet at the source ENI.
  • The timeline pane (in pink) shows the encapsulated packets (which are ICMP requests/replies) and not the outer VXLAN transport packets.
  • The source IP address of the VXLAN packet is the primary IP address on the source ENI (172.17.0.4); the destination IP address is the primary IP of the target ENI (172.17.0.81). 172.17.0.223 is not involved in the mirroring process, it just happens to be sending ping requests to the source.

Troubleshooting

There are two likely reasons why traffic wouldn't show up in my sniffer:

  1. A missing route. The VXLAN packets are routed from the source to destination ENI just like regular IP packets within the VPC. The routing table that's bound to the subnet where the source ENI is attached must have a route towards the TM target. (This would really only be a consideration when the source and target are in different VPCs.)
  2. The Security Group(s) attached to the target ENI do not allow inbound VXLAN traffic. Since the VXLAN traffic is routed in the VPC just like regular traffic, it is subject to inbound Security Group rules on the target. Either allow UDP/4789, or all traffic from the source ENI's primary IP address, or some combination thereof. (TM traffic is not subject to outbound Security Group rules at the source ENI so there is no special consideration needed there.)

Reference


Disclaimer: The opinions and information expressed in this blog article are my own and not necessarily those of Amazon Web Services or Amazon, Inc.