NAT between identical networks using VRF

https://blog.oddbit.com/post/2023-02-19-vrf-and-nat/

by ytch • 3 years ago

92 62 3 years ago

blog.oddbit.com

tarasglek2• 3 years ago

This is a very real problem when using a VPN into another network with same ip-space.

Though I'm not sure the complexity is worth it. Suppose it would be nice if tailscale implemented this type of substitution-routing-NAT this for users.

dilyevsky2• 3 years ago

I mean they kinda do by assigning each node a cgnat space IP?

tarasglek2• 3 years ago

They have NAT routing support for exposing nodes that don't/can't run tailscale. Meant that ip space

zamadatix3• 3 years ago

Another possible option for the VPN use case, depending on what you're trying to do and how you're doing it, is to just put the VPN into the VRF (or a netns) and only bind connections you intend to run through the VPN to that context. It's not particularly fantastic either but it's a little more straightforward.

bob1029• 3 years ago

This is a very real problem when using a VPN

I've been really careful about selecting virtual network IP spaces throughout due to this concern.

The hardest part is accounting for the point-to-site VPNs on top of everything else (i.e. employees connecting from home/Starbucks/airport).

For internal corporate networks, I typically will provision smaller targeted networks in order to get around this. Lopping-off all of 10.0/16 or 192.168/16 for your internal LAN will put you at a fairly high statistical likelihood of overlapping domestic ISP address spaces. Smaller, odd networks like 192.168.111/24 are much easier to plan around.

m463• 3 years ago

One network I used allocated from 172.16.0.0 – 172.31.255.255 but then docker wouldn't work if you were in 172.17

zamadatix3• 3 years ago

For virtual network IP space you can go with the 0.0.0.0/8 space (sometimes called "zeronet") as long as you are using kernel 5.4+ everywhere you want that network. Obviously 0.0.0.0/32 is still excluded.

Windows (and probably macOS?) still block that as of today so it can be an extra safe (and large) space if your only worry is "I don't want my virtual network space to collide with outside or client sourcing networks".

justsomehnguy• 3 years ago

There is an extremely low chance what you find a collision with 30/8.

linsomniac• 3 years ago

Tailscale does do something along these lines, but it's kind of surprising... If there is a tailscale route announced, it will take precedence over other routes, even if they are more specific. The primary use case for this seems to be if you are at a coffee shop, you can access a tailscale routed network that has the same IP range as the coffee shop.

The down side of this is that if you have, say, a wider announced network routed over your tailscale (say 10.0.0.0/16), and then you have machines within that network also connect to the tailscale (say a machine with 10.0.1.1/24), it now will try to reach machines on its local network interfaces via routing over the tailnet. Pretty unexpected.

https://github.com/tailscale/tailscale/issues/6231

WirelessGigabit3• 3 years ago

We had this issue with Docker, which uses 172.19.0.0/16 for the default bridge. You can change it but it was weird figuring that out the first 2 months.

tecleandor• 3 years ago

Tailscale does a fun thing that's mapping overlapping ipv4 networks into different ipv6 networks:

https://tailscale.com/kb/1201/4via6-subnets/

staringback• 3 years ago

Can we let IPv4 and NAT die already?

greyface-7• 3 years ago

No. Even in IPv6-only-land, multihomed networks without PI space will have good reason to NAT.

ArchOversight3• 3 years ago

If you generate a ULA address using the algorithm that is recommended the likelihood of a collision in IPv6 addressing space in networks is absolutely miniscule.

groestl• 3 years ago

This will never happen, I'm afraid :/

ok123456• 3 years ago

Just give it another 25 years.

yjftsjthsd-h• 3 years ago

IPv6 started rolling out... let's call it in 2000, which is probably a touch late but close enough, and https://www.google.com/intl/en/ipv6/statistics.html puts it at 43% of traffic, but that's positive biased (because it's only traffic on the public internet). So... I'm going with no, as a society we can't and/or won't.

sgjohnson4• 3 years ago

I'm going with no, as a society we can't and/or won't.

But we definitely should. The number of IPv6-only networks is growing by the day.

The next generation of tinkerers and internet engineers will have no IPv4 address space at all, because it makes no sense whatsoever paying $14k (at current prices) for /24 of legacy IP address space.

Unfortunately, the best we can hope for is for giant corporations like Cloudflare, Microsoft and Amazon to buy up most of the IPv4 space.

dilyevsky2• 3 years ago

Tl;dr: remap each side to a different /24 using iptables and vrf device. If you dont want to mess around with iptables this can also be trivially achieved using OVS

d-z-m• 3 years ago

ryanschneider• 3 years ago

If each network also had unique dns services, then I think in theory you could run a split horizon DNS server on the middleman that rewrites results from the other network as the “alt” address.

E.g. in the example middleman could rewrite a query for `outernode0` that comes from the inner node network to resolve to 192.169.3.10 (instead of .2.10).

I’m not sure which dns servers allow rewriting logic like that though.

zamadatix3• 3 years ago

The usual suspect, BIND, has a feature called Response Policy Zones that can serve different responses to different ACL matches (such as source IP). It can even autogenerate the modified records if you ask it politely.

Of course a screwdriver also has a feature called Quick Eyeball Removal which may be preferable. That or I'm just bitter about having done this :).

3np• 3 years ago

You can also achieve similar goals in dnsmasq via --localise-queries and --dynamic-host. It's a bit clunky but not complicated.

Already__Taken• 3 years ago

I run coredns as internal dns and the advice for this solution from them is to run another coredns and solve it in config management. While not a pleasing answer its probably the easiest thing to do. Someone did write a plugin but it requires a fork to get some hooks and that was declined its PR.

supriyo-biswas• 3 years ago

I am piggybacking my question on this discussion: as a beginner to Linux networking, are there good resources one can learn from (such as the use of ip link for route management as discussed in the article?)

faragon• 3 years ago

Check the FRR project.

https://frrouting.org/

andix2• 3 years ago

Who are this powerful sorcerers that implement those network features? It’s one thing to figure out how to do that, but there must’ve been somebody who crafted all those components first. Truly impressed.

suprjami• 3 years ago

Network providers have been using VRF for decades. I was doing Cisco admin of VRF networks in the 2000s.

andix2• 3 years ago

True, i didn’t think about that. The Linux implementation probably came afterwards and was based on the experience with those dedicated network devices.

KaiserPro• 3 years ago

So, whilst possible to do, and this blog is a brilliant guide to doing it, but there are better ways.

if this is a network you own either end, then if you really do have the same subnets issued to different sites, then you would be better served having virtual interfaces/VLANs to have a second IP address.

Or, just use ipv6, and you have enough IPs to do what you want. You can use 6to4 or nat64 to get IPv4 connectivity. most modern networks terminated services on the edge with loadbalancers, so you can use them as 6->4 bridges.

yrro2• 3 years ago

Very cool. Surprised nftables doesn't support NETMAP yet though!

thedougd• 3 years ago

If you want to do VRF on Linux and don’t have a philosophical objection to systemd, I strongly recommend using systemd-networkd. VRF, wireguard, and many other complicated things are easy to configure out of the box and don’t require extra scripts or custom units.

My home router is a Raspberry Pi running Ubuntu server. VRF, Wireguard, VLANs, bonding, llrp, route advertisement, prefix delegation, dhcp server and leases, all configured only with systemd-networkd.

pm2222• 3 years ago

Network overlap alright. If the hosts do not overlap I bet you can enable end to end comm with just /32 host routes

solotronics• 3 years ago

All of this just because people didn't want to figure out IPv6!

bogantech• 3 years ago

Any system connecting to the internet will need IPv4, if you go down the IPv6 route you're going to have to maintain a dual-stack infrastructure (with host, firewall, rotuer configs etc) and get at least 2x the amount of headaches whenever anything goes wrong.

What is the business case for all that extra pain?

yrro2• 3 years ago

I was looking for some information on how VRFs compare to Linux network namespaces & found https://www.toddpigram.com/2017/03/vrf-for-linux-contributio... which does a good job of explaining why it makes sense to use VRFs even though there is some functionality overlap with network namespaces.

I also found some information about using VRFs in the documentation for RHEL 8: https://access.redhat.com/documentation/en-us/red_hat_enterp...

NAT between identical networks using VRF

Related Stories