"BGP at home": getting a DIA circuit installed at home

https://aaka.sh/posts/20230129-bgp-at-home.html

by olivernyc • 3 years ago

183 97 3 years ago

aaka.sh

c3534l• 3 years ago

But why?

cobertos2• 3 years ago

Why not? Sounds fun

They mention at the top because they didn't want to colocate it. So I imagine they did it because they could?

dboreham5• 3 years ago

Probably cheaper than bringing up a second colo if you don't need to host many machines.

skissane4• 3 years ago

If I had more money than sense, I’d do the same thing.

If I was filthy rich (which I’m not and probably never will be), I’d build a data centre in my mansion and buy a brand new IBM mainframe and stick it in there. Because I can. For the same reason other rich people build some huge garage with dozens of cars in it.

remram3• 3 years ago

It sounds like they wanted 2 separate network paths but they still end up with only 1 where they can use BGP. Maybe they can still use their residential connection as failover, with a VPN to a VPS where they can use BGP?

HyperSane4• 3 years ago

I would buy one of every high end enterprise storage array to play with. My former life as a VMWare admin made me appreciate storage.

sbierwagen• 3 years ago

Besides the stated reason, there's also this:

(as of this writing, VZ was pricing 50Mbps commited @ $455/mo; 100Mbps @ $661/mo; 1Gbps @ $999/mo; 5Gbps @ $2,099/mo; and 10Gbps @ $3,099/mo).

Ten gigabit is not yet a common residential internet service in North America. If that's what you want, this would be the only way to get it.

I've occasionally daydreamed about buying a condominium near one of the Seattle Internet Exchange's PoPs, buying a 400 gigabit port, paying to run a fiber pair, arranging for transit, etc etc. https://www.seattleix.net/join As expensive and pointless as buying a yacht, but at least getting it online would require a lot of tedious work!

(Like OP, I also jumped through the hoops for a Comcast fiber drop years ago (but to a business location) for the same eyewatering price but dramatically fewer megabits per second. The only upside is that the pair traveled directly to a CO in the same office park, so we would routinely see 1-2ms pings to anycast IPs like 4.2.2.2, 8.8.8.8, etc)

Shank• 3 years ago

If that's what you want, this would be the only way to get it.

10Gbps PON is available from AT&T Fiber in some markets, and lower tiers (5Gbps) are coming online from Google Fiber in other markets. In SoCal, I've seen billboards for 10Gbps AT&T Fiber, for example. Obviously you don't get an SLA with a residential connection, though, and I highly doubt that many customers operating at full capacity will be problem-free.

techsupporter3• 3 years ago

I've occasionally daydreamed about buying a condominium near one of the Seattle Internet Exchange's PoPs, buying a 400 gigabit port, paying to run a fiber pair, arranging for transit, etc etc

For what it's worth, this isn't the same as what the person who wrote the post did. They bought a DIA circuit to an ISP.

What you're seeking is a point-to-point fiber link or MPLS to a panel in KOMO Plaza or the Westin building. You don't need to be near either of those locations to do it and CenturyLink (Lumen, these days) will sell you the circuit but you're not going to like what it costs. We have three of these circuits where I work and they are juuuuust a bit higher than what the article's author says they're paying for the DIA.

(This, along with the eyewatering cross-connect fees that the operators of those two facilities charge, is why getting a SIX port is so expensive on retail colo. You're either paying for the privilege of running some strands of glass or you're paying to use someone's extension switch and the connection they're paying for to get back to the SIX core.)

exikyut• 3 years ago

Move to Chatanooga (in Tennessee) instead. Business class (pretty sure with SLA) 10Gbps from EPB for $300/mo, introduced ~2015.

pdpi• 3 years ago

The only upside is that the pair traveled directly to a CO in the same office park, so we would routinely see 1-2ms pings to anycast IPs like 4.2.2.2, 8.8.8.8, etc

Those are the pings I see on my residential connection, though?

dboreham5• 3 years ago

This is how any larger business (e.g. a hospital, university, anyone doing any serious networking) gets their connectivity these days. Basically the modern equivalent of a T-1.

You get proper service: no random dropouts, and a clueful human on the phone and you could complain about QoS and they'd not laugh in your face.

The different thing here is that the service was provided to a residential building, but that's not entirely unheard of. I've done it myself for example.

gertrunde3• 3 years ago

Primarily for resilience.

In the event that there are issues at the primary location, internet routing protocols will re-route the relevant network prefixes to the secondary location.

The public /24 that was previously routed to their NYC datacenter now goes to that Connecticut house.

lukeh• 3 years ago

Why two NIDs?

rektide2• 3 years ago

This is one of the most literal Conway's Law instances I've seen...

organizations design systems that mirror their own communication structure.

An org for the physical layer has a box, then the org offering the connectivity layer plugs their box into the first box. Like OSI layers stacking up.

https://en.m.wikipedia.org/wiki/Conway%27s_law

dboreham5• 3 years ago

It's more likely to reflect a hierarchy in the network (e.g. monomode span dropping down to multimode for the demarc). The telco just colocated their box on the subscriber's premises.

VoidWhisperer• 3 years ago

I'm curious about this too.. it seems like it might be some intricacy because of Verizon business and Verizon telecom being two different entities, but I am curious if there is a technical reason for it too.

Spooky23• 3 years ago

The guys on the ground and their dispatch are CWA unionized employees. The businessy guys are not.

Verizon is like a state or federal government agency. Massive complex bureaucracy that nobody really understands. As one of the ultimate legacy businesses they have all sorts of crazy settlements, agreements, tariffs, etc and segment their businesses around them.

gertrunde3• 3 years ago

It's quite common tbh.

Two different companies, one provides the last mile, one provides the customer service layered on top.

It adds a lot of flexibility, for instance in this case the person was on-net for Verizon, but what if they were not? Then it wouldn't change at all, just the first NID would be a different provider than Verizon.

bogomipz5• 3 years ago

First "NID" is a Telco term of art, it's a contractual concern more than anything else.

The first device terminates the Fiber. It has an SPF transceiver in at for the wavelength the customer is getting the service on which is 1310nm. This is pure layer 1. A technician could/would use this "NID" to get a light reading during provisioning and/or troubleshooting. This is the end of the "local loop."

The second device is the one that provides Carrier Ethernet. Carrier Ethernet is like regular Ethernet but with some extensions that allow the provider "OAM"(operation, administration and management) capabilities. This is a layer 2 device.

NoZebra120vClip• 3 years ago

I'm curious to know details like what sort of house or dwelling this person lives in; anyplace with enough capacity for all that (4U rack space, power and cooling, etc.) seems spacious.

The article says it's right off a major street. So is it urban? Suburban? Single-family home? Inquiring minds want to know.

cmeacham984• 3 years ago

Just so you know, 4U of rack space is a whopping 7 inches of height. One rack holds many Us, with most "full size" racks clocking in at 42U.

Open-air racks are typically less than 2 feet across and 3-4 feet deep (example [1]), hardly a significant stretch unless you live in a very small space.

1: https://www.amazon.com/StarTech-com-Open-Frame-Server-Rack/d...

zamadatix3• 3 years ago

4u of space is less volume than many PC cases. I’ve got a 22u “half rack” in my apartment (half for testing gear for work, half because I like to tinker). Bigger concern would be noise. 1u network gear tends to have high pitched 20mm fans which can drive you nuts if you don’t have it in a separate room. Noctua makes a great kit I use in my own gear but Verizon probably wouldn’t be very happy with you if you tried to do that :).

I went the route of “find a dirt cheap gigabit colo” ($150/m for quarter rack with gigabit and bgp peering) and just tunnel the IP space back home. Upside is you don’t have to pay to have the circuit put to where you currently live and you don’t have to worry about space/noise. Downside is if you want to use it locally at home you get extra latency due to the tunnel.

cmeacham984• 3 years ago

Downside is if you want to use it locally at home you get extra latency due to the tunnel.

I guess it depends on how your tunnel is set up, but shouldn't you be able to add a static route to your rack on the "home side"?

icedchai3• 3 years ago

Currently I'm doing BGP over a wireguard VPN back to a couple of VPS providers. Actually, I only run OSPF to my home, the BGP part only happens on the VPS side. The tunnel does add about ~8 ms of latency worst case.

jordanbeiber• 3 years ago

I have a couple of 2U racks vertically mounted behind a door. Takes up zero usable space!

It took me way too long to realize this option - many builds using miniture casings with cooling issues later, when buying large 2U cases with 120mm fan walls is the ticket.

ilyt3• 3 years ago

You need like, a closet

cobertos2• 3 years ago

Curious what they're running just on the other side of the Verizon Business NIB, and if there was any hiccups in the peering process with Verizon

remram3• 3 years ago

BGP peering! I would be happy just to get IPv6 connectivity...

onphonenow2• 3 years ago

The pricing is eye watering. 2,100 for 5gbps? Att fiber is $200 per month where I am w a static block.

cmeacham984• 3 years ago

Residential internet service is oversubscribed and doesn't have an SLA.

digitalsushi• 3 years ago

Oversubscribed means different things to different groups of people. (I'm replying to people in general, and not to the commenter above me)

To an ISP, it means that a customer could exeperience a shortage of service, but that a group of customers would have to simultaneously utilize the infrastructure. ISPs try to keep a ratio of customers to service where this isn't likely to happen, or for discount providers, that this happens only during peak times. Ultimately it is a balance of customer attrition.

To a customer, it sounds like they are being deprived of what they are paying for. (Residental customers agree to a best-effort service typically - the opposite of agreeing to a specific service level). Even with ample service capacity, it is often the first factor considered when any component in the aggregate network is failing or underperforming.

I sure do not miss being a residential ISP. But I carry with me just enough sympathy for service providers that I am "always a blast at a party".

fragmede• 3 years ago

don't worry, there's no way the techs at Comcast could figure out how to prioritize speedtest.net in their QoS tables.

onphonenow2• 3 years ago

I absolutely understand- but is still amazingly cheap. The install was not simple. Despite being oversubscribed I get good speeds both ways. Most devices on my network can only handle 1 gig though my vmware/ synology stuff is 10g. I have business service from att at a different location - Even ordering process was kind of annoying- you get a rep and going around on price / term etc takes a bit vs clicking buy now.

At home I pay for 5 but just need 2. I have failover to comcast. Works well . And yes I do get reported outages on the att side but because failover works I’ve not chased things down.

zamadatix3• 3 years ago

ATT home fiber is typically 32 ONT sharing 1 OLT, doesn't come with the SLA, and doesn't let you assign your own IP space to the connection (along with some other knobs BGP lets you turn).

Of all of that the bulk of the price increase is them running the dedicated fiber so you can go from 32:1 to 1:1. It costs a lot to not have the same physical setup as the rest of your neighbors.

gertrunde3• 3 years ago

Bear in mind these are entirely different classes of products... it's not just about bandwidth.

The monitoring and SLAs alone are probably the most significant differentiators, with these kinds of services you would usually expect an engineer on site within 4 hours to fix problems, and service credits for anything breaching SLA.

I once accidentally snagged a single mode fibre pigtail while moving stuff in a rack. Everything back up after the provider engineers had re-spliced a new pigtail on in less than two hours.

And that's before you consider the dynamic routing side of things.

(And the 3 x /24's that AS54316 have are also not cheap these days, that would cost something like $30-40k these days).

MPSimmons• 3 years ago

General reminder that an SLA is a contractual agreement, and not a guarantee of availability. It literally describes the renumeration due when the service doesn't meet the promised availability.

cardy31• 3 years ago

I often hear people use SLA and SLO interchangeably. Very annoying having to clarify.

markonen3• 3 years ago

I rent a single ~600 meter fiber that connects my home to the network infrastructure of the company I run. The rent is $75/mo or so. I have passive CWDM muxes on the line to run two 10G connections to two separate edge routers.

My home router is a MikroTik CRS309—it’s about $250, fanless, has 8 SFP+ ports and advertises my home network blocks over BGP to the two routers (for HA).

The setup works great. The best part is how it’s small and fanless and fits inside the very small wall box the fiber terminates in.

techsupporter3• 3 years ago

My home router is a MikroTik CRS309 ... advertises my home network blocks over BGP

How do you fit two full BGP tables into 512MB of RAM? I've looked into MikroTik boxes before and maybe they're doing something I'm not understanding. On the routers I have manage, two IPv4 feeds take up about 1.1GB and three IPv6 adds another 450MB.

no_carrier• 3 years ago

If by 'full BGP tables' you mean the entire internet routing table, you don't need that to advertise a network on the internet. You can receive just a default route if you wish and still be able to advertise to any carriers you have a connection with. You won't lose out on any functionality in dual homing or anything like that.

markonen3• 3 years ago

I’m just getting the default route from both upstream routers. They’re not diverse in terms of the internet routes they have, this is just for HA.

ilyt3• 3 years ago

Use BIRD:

    bird> show memory 
    BIRD memory usage
    Routing tables:    262 MB
    Route attributes:  120 MB
    ROA tables:        192  B
    Protocols:         171 kB
    Total:             382 MB


    bird> show route count table r1
    906312 of 906312 routes for 906312 networks
    bird> show route count table r2
    903532 of 903532 routes for 903532 networks
    bird>

routes in kernel will take less as you're only getting best one exported to kernel and no route attributes to hold

tiernano• 3 years ago

I have their RB5009 with 1Gb RAM and I got 4 full feeds and have around 200Mb free. I do need to tweak stuff to get IPv6 working (again... only been sorting this out in the last couple weeks)

Aachen• 3 years ago

Huh, so per the last paragraph, they're paying a ton to have an SLA, all the ISP has to provide is basic connectivity because BGP and stuff is being done by the customer themselves, and yet they have more outages in a few months than I have on a consumer connection in years?

The whole thing has a lot of hack value, it's cool and also worth something to be in control of your own networking, even if it's more expensive. Like buying apple: overpriced for the specs you're getting, but you're assured it'll be good quality (that's the idea anyway) and it looks cool (to most). Except... apparently it's still got issues, regularly? Now I'm really wondering what the point was, at least with hindsight

rfoo• 3 years ago

The post said that consumer connection does not let you do BGP peering with the ISP.

There are real value in doing so in addition to the hack value, by doing so you can steer the Internet traffic to whatever IP blocks you owned to your house, dynamically. For example OP mentioned "add some resilience" as motivation, i.e. anytime their services running in the DC failed he can reroute the traffic to ... their house.

Karrot_Kream• 3 years ago

You're absolutely feeling the same problems on consumer connections. With DIA, you're the only subscriber on that circuit, so if upstream is having a problem then your neighboring consumers will have them too.

Lower bandwidth dedicated circuits can often "feel" faster than a higher bandwidth consumer connection.

scottgg• 3 years ago

There’s also DN42 [1] if you want to mess around with a bigish BGP network for free

1. https://dn42.eu/Home

jonatron• 3 years ago

See also a guy who used a microwave wireless link to a local data centre from his apartment: https://www.ispreview.co.uk/index.php/2020/12/video-using-a-... / https://www.youtube.com/watch?v=_AAOVVmaFEo

traceroute663• 3 years ago

While the SLA says 100%, don't expect perfection

I've always considered Verizon's (and prior to that, WorldCom and/or MCI) 100% SLA to be pure marketing BS.

Anyone who's ever worked with telecoms knows it's simply unrealistic. Fibres will get cut, carrier's routers will need software updates etc. etc. etc.

I reckon the reason you pay a Verizon-tax is so they have spare cash floating around to pay you the inevitable SLA penalties.

Personally, I prefer dealing with carriers who have more real-world SLAs.

bcrl• 3 years ago

The only thing SLAs do is tell you how much you will be compensated for downtime. If a fibre cut happens, you can end up with a 16 hour outage regardless of what the SLA guarantees. It all depends on the nature, location and extent of the damage. A couple of summers ago one of my wavelengths was down for more than 16 hours. 3 x 432 count cables had to be replaced through 3 manholes because a fibre seeking backhoe ignored the locates. Splicing high count cables takes a lot of time (even with ribbon fibre).

Habgdnv• 3 years ago

I've been running BGP at home for the last 10 years, and in my area, most ISPs prefer to use Dedicated Internet Access (DIA) for BGP sessions. Fortunately, I live in a place where there are a few smaller ISPs with around 2000-3000 customers that are willing to run BGP with me as a home user. They're impressed that I have BGP, ASN, and IPs as a home user, and I think it's pretty cool too. Additionally, running one more BGP session makes their network appear larger, so they benefit as well.

I currently have a BGP session with one of my ISPs on the 15 euro plan (1 Gbps), and another local ISP is giving me 10 Gbps for free because I'm a test user. Both of these ISPs support BGP, and it's amazing that I can have BGP sessions as a home user. However, it can be challenging to speak with regular sales reps who may not be familiar with BGP. It takes persistence and some effort to get to the network administrator.

It's frustrating that some ISPs make it artificially difficult for home users to obtain a BGP session, even when they're not asking for SLAs or dedicated connections. Technically, you can even run a BGP session over dial-up. It seems like these restrictions are in place to squeeze more money from customers.

It's no secret that we've run out of IPv4 addresses, and I happen to have a /24 subnet just for myself. Even with all my usage, I barely use half of it. There's no rule that says ISPs should only accept /24 or shorter prefixes, and this is contributing to the depletion of IPv4 addresses. I would happily announce a /25 and return the other /25 to the world, but I must have a /24 to do it. This practice of requiring /24 subnets may have made sense in the past to preserve router memory, but with today's technology, it's nonsensical and only serves to artificially deplete available IPv4 addresses.

traceroute663• 3 years ago

It's frustrating that some ISPs make it artificially difficult for home users to obtain a BGP session

Don't worry, it can be frustrating for businesses too !

Cogent famously nickle-and-dime their customers and consider BGP to be a chargeable extra, even on their IP Transit product which is somewhere where you would expect BGP to be a given.

Of course whether you should do business with Cogent is another matter, especially given their spammy cold-calling tactics, seemingly once you're on their list you can never get off it.

throwaway894345• 3 years ago

I have a super rudimentary understanding of what BGP is (it allows you to advertise addresses something something), but I don't really understand why it's useful and thus I'm finding TFA to be a little inaccessible. Could you provide some context about what you're using BGP for (what is your use case; what does it enable for you; etc) such that it might fill in some gaps for me?

linsomniac2• 3 years ago

BGP is what "the Internet" uses to say what IP addresses can be reached over the various network links that connect different providers. If you had links to two different providers, you would talk BGP to them and say "You can reach this set of IP addresses via me." They then propagate that announcement on to other providers that they connect to.

This is done via ASNs (Autonomous Systems Numbers), like 32150 (my old ASN). Each hop along the way gets their ASN tacked on. It's like a flood-fill to all the BGP routers in the world.

When you "traceroute", you see IP hops along the way, but the ISPs routers choose links based on this list of autonomous paths (among other things). Basically, a AS-level traceroute is built into the BGP routing information. At it's most basic, a path is selected based on the shortest AS path (traceroute, but at the organization level rather than IP level); the fewest number of networks the traffic has to traverse.

If a link goes down (either because of a network failing, or because of administrative reasons), traffic will eventually switch over to another way of reaching the destination. You can also do things like pad the AS path with multiple copies of your AS to cause traffic to switch to another link more frequently, as a kind of primitive load balancing.

traceroute663• 3 years ago

I have a super rudimentary understanding of what BGP is (it allows you to advertise addresses something something)

The TL;DR:

Yes, BGP allows you to advertise addresses.

Ergo, it is allows you to build resilience, i.e. you can connect to multiple independent ISPs and advertise the same addresses. Hence traffic will always be able to reach you even if one ISP goes FUBAR.

By the same token, it makes pretty much zero sense to have BGP on a home connection (or, in technical speak, a "single-homed connection"). Since really BGP is a zero-sum game on single-homed connections (no real benefit, much added complexity).

ctime• 3 years ago

The internet is a decentralized collection of autonomous systems (AS) and the primary benefit of this setup isn’t exactly realized afaict, which is multiple paths via different carriers. Your BGP AS is advertised via N paths, so your IP networks are thus reachable via multiple paths. Systems will take the shortest/best path adding to redundancy and throughput. Compared that to “business” service without routing options, when your “dedicated” connection drops, your IP addresses are no longer reachable from the greater internet.

notyourday• 3 years ago

This practice of requiring /24 subnets may have made sense in the past to preserve router memory, but with today's technology, it's nonsensical and only serves to artificially deplete available IPv4 addresses.

You should be surprised about how much of the internet still runs on gear from late 1990s

icedchai3• 3 years ago

You're lucky. There are basically no small ISPs around here anymore. It's the cable company (DOCSIS) or telco (fiber, or DSL if you're unlucky.) Without paying for an enterprise grade connection, it would be impossible to get a BGP session through either.

I also have a /24 for myself. I've had it since the mid-90's, from the InterNIC days, predating ARIN. I've gone years without using any of the addresses. Currently it's routed to my home lab. I'm doing BGP through a couple VPSes, then tunneling it back.

hattmall• 3 years ago

I have a /24 as well, can you point me to any links or tutorials of how I can actually use it with my home connections

bogomipz5• 3 years ago

There seems to be a lot of misconceptions in your post.

"Additionally, running one more BGP session makes their network appear larger, so they benefit as well."

This doesn't make any sense. The number of "BPG sessions" is not a metric any ISP thinks about. A BPG session is just TCP connection. The number of prefixes an ISP is advertising might be something they care about however but for smaller ISPs that doesn't really matter as they are just customers of the Tier 1 ISPs and that just requires paying the Tier 1 ISP. It has nothing to do with the number of sessions or prefixes. I could have a single /22 prefix for my company and the Tier 1 ISP will still sell me the same service.

"There's no rule that says ISPs should only accept /24 or shorter prefixes, and this is contributing to the depletion of IPv4 addresses. I would happily announce a /25 and return the other /25 to the world, but I must have a /24 to do it."

Well yes there is. Many ISPs will generally not accept routes smaller than a /24. They configure filter-lists that prevent those routes from being accepted. The reason for this is to reduce the size of the global routing table. Each prefix has to be stored in TCAM(memory) on a router. There have been many incidents where older routers couldn't handle the number of routes in the global routing table and caused them outages. See:

https://arstechnica.com/information-technology/2014/08/inter... and https://www.inap.com/blog/growing-pains-internet-global-rout...

Further if you have a /24 you can certainly advertise 2 /25s to the same upstream ISP if you have more than one transit connection with them. In fact this is exactly how do traffic engineering.

"It's frustrating that some ISPs make it artificially difficult for home users to obtain a BGP session, even when they're not asking for SLAs or dedicated connections."

This because home users absolutely don't need to run BGP! You just take a default route from your ISP and you're done with it. There is not point in running BGP if you are single-homed! You can't do any traffic engineering on it or influence the way upstream ISP route traffic to you. From the hobbyist point view it would be extremely boring. What can you actually do with it? View the global routing table? You can do this without ever running BGP by using a public looking glass server[1]. There is no incentive for ISP whose business is selling internet connectivity to cultivate a hobbyist community.

[1] https://www.bgp4.as/looking-glasses

tgsovlerkhgsel• 3 years ago

The main issue he wanted to avoid (trees falling on lines) still seems to be there.

Looking at past outages, I feel like given a certain budget, you would get much better uptime with two or more diverse (i.e. not "two fibers going exactly the same path") residential providers than with one commercial one, and for anything but the most critical projects, that's the way to go IMO.

Most importantly though, anything really really important that is so important to use custom fibers should be designed so it also works over the public Internet. Unless degraded service is worse than no service, it doesn't matter that the Internet is theoretically not reliable enough/doesn't provide enough guarantees - at least keep it as a backup so that when a backhoe takes out your custom fiber, you can switch over to the VPN instead of shutting down air traffic at one of Europe's largest airports.

Seems like another major reason for the decision here were SLO/SLAs. Those seem rather meaningless. The residential ISP will not try to max out their 95% SLO - an ISP that's down 1.5 days every month or leaves you offline for 18 days wouldn't be very popular. The commercial SLOs, on the other hand, sound great but they're not magic. If the issue can't be fixed in that time, they will be violated, and you'll typically get a credit for a few months of service - money that you wouldn't have spent in the first place if you just went with residential.

In the end, he still has countless single points of failure, e.g. the above-ground fiber line waiting for a tree, or a non-redundant power supply that he can't easily replace himself. Sure, he has an SLO, but with a dual uplink residential (which is cheaper), a CPE ("modem"/router) failure wouldn't be an outage in a first place.

tssva• 3 years ago

They state at the beginning of the article that the main service location is a data center in NYC. This is a redundant service location. An outage would require failure of connectivity to his home and to the NYC data center.

matthewmorgan• 3 years ago

When did it fall out of fashion to spell out any acronyms/abbreviations you're using the first time they're used?

"BGP at home": getting a DIA circuit installed at home

Related Stories