Peerd: an AWS VPC peering connection management tool

by bmcalary_atl • 6 years ago

30 26 6 years ago

Be kind, this is one of my first python projects. :)

peerd is an AWS VPC Peering Connection management tool. It manages the full lifecycle of creation, deletion and route table updates needed to make VPC peerings useful.

Capabilities

    Capable of creating and accepting cross-account VPC peerings.
    Capable of creating and accepting cross-region VPC peerings.
    Capable of creating full-meshes of VPC peerings.
    Injects, repairs and removes routes as needed from VPC routing tables.
    Capable of managing overlapping meshes with the use of an environment name which is used to tag peerings.
    Overlapping meshes supported through the use of different environment names in configuration file.

Yes, Transit Gateway is a valid replacement for this tool. :)

JshWright• 6 years ago

Haven't had an opportunity to look any deeper than the project description, but this is definitely a thing I've wanted in the past...

saber62• 6 years ago

Interesting concept - is this designed to be used independent of some kind of infrastructure as code tooling?

I do all of my VPC peering and route-table manipulation via terraform repos today, so I am curious what the benefit of using this would be over what I have today (if there is).

Thanks.

bmcalary_atl4• 6 years ago

Terraform is definitely an option, but we found (the story might be different since 0.12) that creating large full meshes with many VPCs across many accounts with many route tables resulted in Terraform modules and files that were quite complex, long and hard to overlap with other meshes.

This tool can be used alongside terraform, ansible or whatever infra-as-code orchestration tool you like. I encourage you to do so! :)

kesor• 6 years ago

Can you explain the "Patent Pending" bit? Why would you need to create a patent for a short Apache 2.0 licensed API calling Python script?

KenanSulayman• 6 years ago

In addition, the apache license grants patent rights (§3 Grant of Patent License). Since it seems to be a research project, I assume they're just very happy to be able to put their name on it with Atlassian funding the patent :-)

apengwin• 6 years ago

Why doesn't Atlassian-labs use Atlassian's own remote git hosting service, bitbucket?

yjftsjthsd-h• 6 years ago

Network effects are nothing to sneeze at.

slovenlyrobot• 6 years ago

BitBucket still has users?

I assume at least one, the person who thought the last redesign was a good idea.

outworlder• 6 years ago

The enterprise version has plenty of users.

meritt• 6 years ago

I still use bitbucket because I also use mercurial, but they've deprecated that and will only offer git beginning later this year.

So, no, I don't know why anyone would use bitbucket at this point.

tjalpin• 6 years ago

I had this exact use case in my job, trying to connect an app hosted on an ec2 in one account to a redshift instance on another, vpc peering to the rescue. will checkout your project when i find time.

code4tee• 6 years ago

Cool tool and the problem it’s trying to solve is a real one. However on the complex mesh of connections across accounts isn’t the new way to do this now not the Transit Gateway?

A tool that could manage complex TGW setups across many accounts would also be cool. Terraform can do that but it’s not the most elegant solution.

jcrites2• 6 years ago

In addition to Transit Gateway, there's also AWS PrivateLink [1], which is my preferred approach for scenarios where it works. As an alternative to peering entire networks together, PrivateLink allows you to establish minimal connectivity between specific clients and services that need to communicate. When a service supports PrivateLink, any authorized client can create a private endpoint for that service in their VPC, and the endpoint receives a local IP address (in that VPC).

This has a couple of benefits. The first is that you don't need to coordinate IP address ranges in different VPCs, which is necessary if you're peering them together (unless you're using NAT). IPv4 address management in large networks can be challenging.

The second benefit is that although clients can communicate with the service, there isn't an actual route between the two networks. This better achieves the principle of least privilege at the network level; it avoids the security risks of connecting networks together such that any machine can connect to any other. When you design an application network to use PrivateLink from the ground up, you can start with the expectation that the VPC doesn't need any connections to any other networks, not even the Internet; it can potentially be an isolated network bubble, with only PrivateLinks allowing traffic in and out (using e.g. Session Manager for administration).

There are also practical limits on how large a federated network of peered VPCs can grow (probably helped by Transit Gateway), but there isn't any limit I'm aware of on the number of client/server connections that can be established with PrivateLink.

For these reasons I prefer to use PrivateLink when building systems that communicate across VPCs, falling back to network peering or Internet-facing endpoints in scenarios where it doesn't work.

However, one downside compared to network peering is that you need to set up a PrivateLink connection for each service that each client needs to call, though this process can be automated during infrastructure provisioning. Services need to have specifically onboarded with PrivateLink; you can't just deploy a service on the network and have it work.

[1] https://aws.amazon.com/privatelink/

bmcalary_atl4• 6 years ago

We love PrivateLink but there are limits and it needs to be provisioned on a service basis, or fronting a proxy for usage across thousands of services. :) And as you can imagine, it too would need some complex orchestration if used across thousands of VPCs or hundreds/thousands of accounts.

But definitely a great tool in the toolbox.

gregwebs• 6 years ago

How do you deal with the connections being to only one availability zone?

bmcalary_atl4• 6 years ago

Yup, Transit Gateway is definitely a solution!

therockspush• 6 years ago

Terraform gets unruly at scale, and makes it hard to pass it off to other teams or take a vacation if you're responsible when it breaks.

Aviatrix popped up on my radar recently trying to solve the operations side of this. Handles cross account TGW attachments and simplifies the routing.

highprofittrade• 6 years ago

Vpc peering will be replaced by transit gateway

Peerd: an AWS VPC peering connection management tool

Related Stories