VyOS From Scratch – Edition 1 (set up your own router)

https://blog.kroy.io/2020/05/04/vyos-from-scratch-edition-1/

by elisaado • 3 years ago

121 83 3 years ago

blog.kroy.io

jstanley• 3 years ago

Might be worth adding a paragraph about what VyOS is, maybe not for the target audience but certainly for HN.

https://vyos.io/

The VyOS website says:

Democratizing how we access networks through a universal Router and Open source software.
Our vision at VyOS is to dramatically change how we access networks so that we can all build the solutions we always dreamed of, without restrictions, limitations, or prohibitive costs.

But I'm still kind of none the wiser. Does this thing use Linux or something else?

minimaul• 3 years ago

Yes - it's a Debian Linux based router distro that is at least partially modelled after Juniper's configuration style.

edit: used to be a maintainer for a short while :)

rnhmjoj6• 3 years ago

It's a fork vyatta[1], the same system used by Ubiquiti for EdgeOS. Yes, it's Debian with a declarative configuration system. It works more or less similarly to NixOS, if you know about that: basically it uses a bunch of perl scripts to install and set up software from a single unified configuration file, hiding all the implementation details.

[1] https://en.wikipedia.org/wiki/Vyatta

sebazzz• 3 years ago

And it works great because your router starts from a read-only image, then executes the scripts and applies the configuration. You can easily roll back and forward configuration and firmware versions.

LeBit9• 3 years ago

I don't think they use Perl anymore. They have been moving to Python.

nunez3• 3 years ago

Had no idea vyatta is what Ubiquiti uses. Very cool.

yrro• 3 years ago

It's a Network Operating System for configuring routers & switches (implemented on top of Debian). This means:

* You can SSH in and configure it like you would a managed switch or router. There's a single object that models all of the device's configuration, when you commit it a bunch of scripts activate and actaully apply the configuration to the running system.

* Deployments are image based, you can roll back to older images etc.

* You don't need to look at any of the system's underlying configuration files or use any of the normal Linux commands to examine and manipulate the state of the system (the commands are still there for convenience of course). You don't even need to be aware that you're really using a bunch of custom bash functions to examine and manipulate the state of the system.

Arnavion• 3 years ago

It's a Debian 8 base with a lot of its own custom packages. The homepage has an FAQ that tries to reassure you that being based on Debian 8 is not a problem. Whether that convinces you or not is up to you, of course.

I personally use regular Debian 12 on my router without problems. It also has "declarative config" since all the configuration, firewall rules, etc are a bunch of config files that I can scp / ansible over any time.

PreInternet01• 3 years ago

The question 'what is this thing' is probably best answered by the Github project page: https://github.com/vyos

It's a decent-ish option if you need advanced routing functionality; one thing to keep in mind, though, is that unless you're OK with running unstable 'nightly' code, you'll be spending USD 8K+ on an annual basis.

awesomeMilou2• 3 years ago

https://support.vyos.io/en/support/solutions/103000152091

They have an LTS release, no?

They seem to follow the RedHat strategy though, only subscribers can download prebuilt images, but you can build the LTS ones yourself:

https://blog.vyos.io/vyos-1.3.2-lts-release

blinkingled• 3 years ago

If you are an individual, you can get the generic ISO by donating on Open Collective. And if you are contributing to VyOS, whether you are writing code, improving the docs, or promoting VyOS publicly, we are happy to share pre-built images with you through contributor subscriptions. Finally, you can always build your own images — just follow these instructions.

Sounds fair to me. Truth is there's no good alternative other than pfSense but if you want Linux (hw support etc) I don't know if you can do better than vyos for routers.

laurowyn4• 3 years ago

If you want a web GUI, then pfSense or OPNSense are the general go tos.

However, if you're comfortable with CLI and modifying configs in /etc/ then just running a bare metal Alpine Linux box is perfectly doable on a tiny box. iptables/nftables for firewall/NAT, dnsmasq/bind9 for dns, dnsmasq/isc-dhcp for DHCP. I've got a handful of these boxes all interlinked via wireguard, sharing routes via BGP using bird.

Sure, you miss the config verification that VyOS provides, but does mean you learn the underlying tools themselves and that knowledge is portable to any other box running those systems.

Personally, I don't quite understand why VyOS is a standalone distro when it could just be a config generator/checker package. Could even support multiple different underlying tools so if you want to use dnsmasq over bind9, or vice versa, it can provide a unified config interface for them.

the_third_wave• 3 years ago

Truth is there's no good alternative other than pfSense but if you want Linux (hw support etc) I don't know if you can do better than vyos for routers.

OpenWRT comes to mind, I've been using it for decades on first dedicated hardware, the last 6 years running in a container on a ProxMox box (DL380 G7). It has no problems whatsoever routing at (gigabit) line speed using a few megabytes of RAM and a few cores. Configuration is mostly declarative using UCI although it also offers the freedom (which comes with responsibility) to use scripts. I use the latter to deal with edge cases which lie outside of the purview of normal routing operations, e.g. triggered actions related to the use of Timelimit [1] on my daughter's phone, IoShit things with special needs, etc.

[1] https://codeberg.org/timelimit/timelimit-server

synergy203• 3 years ago

openwry,ipfire,ipcop,all do the job well

sp0ck5• 3 years ago

You have very narrow definition of what software router/service router is. Firewall and two interfaces and VPN server is not the best scenario :) All those systems (pfSense etc) are for private/soho use. Big networks need stuff that is not avaiable on mentioned platforms like i.e BFD (Bidirectional Forwarding Detecion), MPLS (MultiProtocol Label Switching), VXLAN (Virtual Extensible LAN), IS-IS routing protocol or Segment Routing.

If anyone knows other Opensource routing software that support all of this - let me know. To my knowledge vOS is the only one.

LeBit9• 3 years ago

I have created some automation to build the LTS ISO every time a new commit is made on the 1.3 LTS branch.

awesomeMilou2• 3 years ago

VyOS is unfortunately completely useless for larger applications, since it's difficult to impossible to automate due to it's unique way of applying configurations. Don't get me wrong, for manual administration it's great, there's a lot of missed automation potential given that it's just Linux underneath.

As an example, the Ansible modules for VyOS are basicially just variations of an adapted ansible.builtin.shell, instead of offering to manage state in a more first class manner (via attributes and values):

https://docs.ansible.com/ansible/latest/collections/vyos/vyo...

LeBit9• 3 years ago

You can ssh into the router, copy a new config (state) and load that config.

It is not very elegant though.

Do you know of an open source router that does what you are looking for?

chomp• 3 years ago

That doesn’t sound that far off from what unifi does, it uses a daemon to do those things. (Receive configuration and then do a local apply)

jon-wood• 3 years ago

From what I've seen of VyOS using a configuration file that is then used to generate the actual system configuration I'm not really sure its so hard to automate. Take your target state, generate a configuration file in the right format, then send it over and apply.

To be honest this feels more like a limitation in Ansible, which has always felt like a bit of a hacky config management system to me in that the way it functions is generally to run a bunch of commands that gradually mutate the system's state, rather than atomically applying the target state, but then I've been spoiled by NixOS on my personal infrastructure recently.

icehawk• 3 years ago

There's higher level configuration for a number of major things like the interfaces, bgp, ospf and the firewall:

https://docs.ansible.com/ansible/latest/collections/vyos/vyo...

Sylamore• 3 years ago

AT&T had bought Vyatta before selling it to it's current owner, but I know they used a REST API internally when deploying it for 5G Edge use cases. It looks like VyOS gained an API in 2019.

nunez3• 3 years ago

idk I've found VyOS fairly easy to automate. It doesn't have an HTTP API and everything needs to be configured through vbash afaik.

numpad02• 3 years ago

But what'd you do with Ansible on a router? Looks like VyOS has REST API and OpenFlow support, btw.

da768• 3 years ago

1.4 has config sync and a REST API coming in

amar0c• 3 years ago

Unfortunately they went full "Vyatta way" or "RedHat Way" by basically giving rolling release for free only. I remember times when Vyatta went behind paywalls Vyos was completely free.

Not sure who would want something rolling on device like router.

Nowdays everyone wanting something good and free go OpnSense way.

rnhmjoj6• 3 years ago

The biggest problem I have is that they only support (or at least that's what they release publicly) x86_64. I am forced to use openwrt because the vast majority of consumer low-power hardware is using ARM or exotic architectures. VyOS interface is vastly superior, though.

sp0ck5• 3 years ago

VyOS is not targeted for such usage scenarios. It is ISP who are paying for support. I have VyOS systems with BGP, MPLS, PPPoE termination, etc that are pushing gigabits of traffic. This is VyOS usecase. Not 5W router handling home or small office.

thepill• 3 years ago

You can build your own LTS image relatively easy and free of charge

LeBit9• 3 years ago

The process is not straightforward, especially if you want to customize the image.

And I have been unable to build a cloud-init image.

But the ISO is buildable from the LTS branch and it works well.

pshirshov• 3 years ago

From my experience, Nix makes lot more reliable and easier to configure router. ymmv

LeBit9• 3 years ago

You are comparing bananas and windows.

rnhmjoj6• 3 years ago

I just finished building a router based on NixOS and I must disagree. NixOS modules mostly targets desktop and servers, but the network specific configuration is still very lacking. A few examples:

- up until a couple of weeks ago the hostapd module was basically a toy: could only manage a single SSID, no way to configure the radios, hardcoded to WPA2-PSK;

- the NixOS firewall is still based on iptables and conflicts with nftables, so you must disable and manually write rules;

- the `networking.nat` module (NAT44) doesn't do NAT reflection;

- I had to write a module for Jool (NAT64, SIIT);

- I had to write a module for libreswan (IPsec);

- I had to write a module for automatic rollbacks, otherwise you can loose access if you make a mistake.

Vyatta and VyOS also provide a much higher level abstraction over the software that is being configured (e.g. you don't have to deal with a specific IPsec implementation). Finally, once you do `nixos-rebuild switch` you're on your own, while with vyatta you have a clean command line interface to inspect the state of the router and manage it.

gbraad4• 3 years ago

does vyatta/vyos store the configuration in a single file or easy to back up config? I do not remember anymore how this was done, but I believe that is the appeal what 'nix' was mentioned.

solarkraft5• 3 years ago

Thanks for the warning; sounds like my mileage would have varied quite a bit.

Do you think that, had your use case been simpler, you'd have enjoyed the experience?

sp0ck5• 3 years ago

And this is just basic routing and firewall with NAT. Now try to setup 20 MPLS L3-VPN's + dozen VPLS services on top of IS-IS routed network with that ;-)

gbraad4• 3 years ago

do you have an example of a router setup? curious what this would look like.

lifeisstillgood• 3 years ago

A definite tangent: About three house moves ago I had OpenWRT on a cable router and knew what was going on. But with family and work and house moves I am now just staring at a flashing BT Home Hub and wondering how to tackle the inevitable "of course that router firmware wont allow that and your ISP wont give out its password and ..."

Is there a uptodate reliable guide (possibly including how to persuade your wife it's a good idea to drill holes in the living room ceiling to run cat6)

trustingtrust2• 3 years ago

What I find easy(ier?) is to run (x)sense on a dedicated firewall and either a mesh with cheap openwrt routers or get something like Deco mesh and run it in AP mode if you don’t have cat6 at home. I think this combination can be under 300$ for a 3-pack of mesh Deco x20 + an intel card on a refurb dell optiplex.

dgroshev3• 3 years ago

IME (x)sense is quite problematic when you start doing anything other than the bare minimum. I spent multiple days trying to figure out why it silently stopped accepting IPv6 delegation, or why does it spike latency for no visible reason under load. The underlying reality is that FreeBSD's network stack is much more conservative and has less resources than Linux's, which shows up in articles like this one [1].

On the same Celeron J4125/i226 box VyOS was absolutely perfect, not a single issue, significantly low (and always low) latency with higher throughput.

On the hardware side, I think the /r/homelab hivemind doesn't get challenged enough. Dell optiplexes cost very similarly to Aliexpress Protectli alternatives (such as [2]), while being larger, having a fan, and being overall more hassle. TP Link/Ubiquiti WiFi APs seem to be overall inferior to Aruba Instant On, which is exactly the same hardware HP sells in their Aruba line, but for the same SOHO price.

[1] https://teklager.se/en/knowledge-base/opnsense-performance-o...

[2] https://www.aliexpress.com/item/1005004272231167.html

didntcheck• 3 years ago

Same here. And AFAICT those "Hubs" have no bridge mode, so the best you can do is double-NAT yourself. Even if you can replace it with your own device, I've just received a letter informing me that they're migrating our landlines to be VOIP, delivered through the phone socket on the back of the router, so if you want to keep landline service then you may need to keep their hardware too

gbraad4• 3 years ago

I was wondering if this was Vyatta... it seems it is the community continuation after Broccade ceased development. Used to run this over a decade ago on a 'router' (ThinkCentre Tiny), but eventually went to a Fedora installation a few years back. Might have a look again.

gbraad4• 3 years ago

... have to say that the offering is confusing; it is a subscription or a rolling release.

Wondering how much can be automated of the installation/state; would it be possible to use version control? If not, I can see the appeal to suggest Nix over this.

kmbfjr• 3 years ago

Both. Pay for a subscription or use the RR.

You can also build your own images with docker using the annotated releases.

dgroshev3• 3 years ago

I was pretty confused what it is too and then I loved it.

It's debian plus some shell trickery and CLI tools that let you configure debian and debian packages as a router from one large config tree using neat CLI tools (that support commit/rollback).

Normally you'd need iptables, a separate DNS package, DHCP server, etc etc to set up a router, with VyOS you just change VyOS config and it configures normal debian packages for you.

Plus everything is exhaustively tested and configs are reverse compatible, hiding all breaking changes underneath.

It's super neat and it works perfectly on a £100 fanless Celeron J4125 box from Aliexpress as a home router, routing and shaping 1gbit without breaking a sweat and with deeply sub-ms delay.

solarkraft5• 3 years ago

This is probably the best explanation I've seen.

Do you have an idea why the CLI tools aren't distributed independently? Why shouldn't I be able to run it on a Debian system I already have (and understand)?

Running an entire new distro just seems like overkill for what it actually does over a normal Linux system. It's just a configuration manager!

dgroshev3• 3 years ago

I don't know, but I imagine it would make testing substantially more difficult (right now they have fully tested images available), and it would be impossible to implement their distro update thing (you can update the entire .iso backwards and forwards, the config will be reapplied).

solarkraft5• 3 years ago

I like the idea of VyOS. Networking software tends to have so much hidden, hard to control state for reasons I don't understand. As a programmer this melts my brain.

VyOS finally lets you have all your configuration in one, easily controlled place. Nice!

For some reason I ended not actually trying it out too actively. I think I was weirded out by the distribution model and concerned by the small community.

solarkraft5• 3 years ago

The theory of what VyOS does is (per my understanding) really simple: Configure all the networking components of a Linux system from a single place.

Why isn't doing this much more popular? All the systems are already there, after all! Why aren't there (that I know of) dozens of projects to accomplish this relatively easy, but relatively useful task?

I think it's a pretty big deal to be able to configure that stuff from a single place. Commercial router manufacturers all do it. Why does (as far as I know) only VyOS do it on the open source side of things?

sofixa2• 3 years ago

IMO, Open Source routers are a niche thing, and open source declarative CLI routers even more of a niche thing.

Most enterprises prefer buying something with a support contract from a known name vendor (Cisco, Juniper, etc.). Most home users just use what their ISP provides them with, and of those that want something more, they either pick a SOHO vendor like Ubiquiti/Mikrotik, or if DIYing the hardware, choose pfSense / OPNSense / DD-WRT for the clickOps options, because networking really isn't trivial. For those for whom networking is trivial, Debian is fine router OS if you know your way around iptables and friends.

That leaves all those who want to use DIY hardware, and a enterprise-like declarative CLI. That's really not a whole lot of people in the end.

whalesalad2• 3 years ago

OPNsense and pfSense exist, plus OpenWrt.

nunez3• 3 years ago

I thought this would be about building VyOS from source, but fun article nonetheless.

I use VyOS whenever I need layer 3 routing in vSphere for test environments. NSX-T is (way) faster by dint of being deeply integrated into VMkernel, but VyOS is pretty performant for what it is and is easier to install to boot.

rhn_mk1• 3 years ago

The idea of VyOS is awesome: have your configuration all in one place, apply it atomically.

After using it for several years, the implementation is clearly lacking. It seems that the maintainers are overloaded, because contributions to fix minor issues, or add config options get ignored in my experience. As a result, the configuration is missing some nice options in the IPv6 space (so-called tethering), and it's still using iptables for packet filtering. It's also rather hard to roll your own, with your own modules: the module system is rather hard to use.

Not to mention the lack of interest to roll out a version for ARM.

My next router will be based on NixOS, and will attempt to recreate the awesome UI of VyOS.

js2• 3 years ago

There's incorrect comments in this thread re: VyOS, Vyatta, EdgeOS.

Vyatta is the original OS, based on Debian, dating back to 2005. Its history is detailed here:

https://en.wikipedia.org/wiki/Vyatta

In 2011, Ubiquiti launched their EdgeMax products with EdgeOS which was a fork of Vyatta Core 6.3 ported from x86 to Cavium.

In 2012, Vyatta was acquired by Brocade.

IN 2013, Vyatta Core 6.6 was forked as VyOS.

That's the rough origin of these three OSes.

I used Vyatta Core on a PC at a startup from 2009-2013 as our office router. I haven't paid attention to it or VyOS since then.

I've been running various EdgeOS routers at my home since 2014 or so, first an EdgeRouter Lite and today an EdgeRouter 4.

EdgeOS has been updated quite a bit over the years from its Vyatta Core origins, but the original developers are no longer with Ubiquiti. EdgeOS hasn't seen updates in quite some time now.

Also, not all Ubiquiti run EdgeOS. Only the EdgeRouters do. The rest of their products run a completely different OS, generally either UbiquitiOS or UnifiOS.

Sources besides my own memory:

https://blog.vyos.io/versions-mystery-revealed

https://old.reddit.com/r/Ubiquiti/comments/scqlg3/what_happe...

whalesalad2• 3 years ago

My ER4 has been a very solid piece of kit.

k_roy• 3 years ago

hey... I know this post....

I need to write a follow-up so bad.

VyOS has evolved a BUNCH since I wrote this, but the same basic ideas apply. Mostly some configuration nodes have moved around.

grawlinson• 3 years ago

Hey! Thanks for writing this, this really helped me setup VyOS as my main router/firewall for my homelab.

Do you have any plans to expand into IPv6 functionality?

elisaado• 3 years ago

Thank you for writing the blog, the way it is written was very engaging :)

VyOS From Scratch – Edition 1 (set up your own router)

Related Stories