HN user
Sure, you can. But on advantage of IPv6 is you addresses are globally unique. If you ever need to connect two networks that were created independently (like different companies), unique addresses, even local are very helpful.
I've given a try to IPv6 in a company with few tens on servers in a 2 DCs, an office + additional location, 3 ISPs in total. For me the real challenge is not just different way to write an IP address or doing NAT. The challenge is that IPv6 changes a lot of unexpected things:
- Our ISPs support IPv6 but routing quality is way worse than IPv4 including occasional inability to connect to some networks or greater latency than IPv4. I had to create tickets with such issues understood that most probably they just don't have IPv6 BGP sessions to all their upstream providers they connect.
- How the VPN (an employee / road warrior setup) should be configured since from the routing perspective you don't need a VPN to connect from your home to the office? Assuming both have proper IPv6 connection and all devices in the office and your laptop have a globally addressable IP address. Employee can have IPv4 or dual stack at his home, where is dual stack in the office. Very confusing. Looks like Fortigate also don't have an idea and decided to not support such case.
- You have to be careful with site-to-site VPN since even your internal services like database are now globally addressable. You really need proper firewall rules / routing policies to not leak unencrypted packets over internet.
- SLAAC is cool but doesn't provide DNS configuration. (there is RFC8106 but is it supported by all OSes?). You need DHCPv6 for that. You have to choose: use only DHCPv6 or SLAAC + DHCPv6 or just relay on the vast that DNS will be proviedd by DHCP IPv4 in a dual stack setup.
- The way of providing high availability gateway address in a network is different. You need router advertisement where you can provide priorities. That actually is much better than any other VIP mechanisms (no issue with MAC table updates, etc.) but you need to know that.
- OSPF works a bit differently. For example: there is no authentication in router communication in OSPF itself, you are supposed to use IPSec.
The list is longer unfortunately...
With BGP (especially anycast) you don't have direct influence where the requests lands. You can steer traffic by techniques like AS prepend, some priorities per BGP session (I'm not an expert in BGP) but eventually it's not CDN that decides where the request will be routed. It's decided by routers of the client's ISP and backbone networks, each taking decision themself (shorted AS path from his point of view + BGP policies set by network operators), not by you. You can't for example split traffic between DCs with specific proportion (like 30% here and 70% there). You can't split by something else than network properties like forward request to the region that has better chance to have this content in the cache.
With DNS and dynamic responses you are directing request to specific DC, even server, almost on every request. It may be dedicated for this traffic type (live stream different than static images etc). Your DNS server can take the hostname ("www.google.com") into consideration - BGP doesn't even know the hostname in the URL. If you wanted to do it with BGP you would need to place specific content to a dedicated /24 subnet and that is impossible considering how many IPv4 addresses are available.
BGP doesn't even consider network latency, current network load. CDN knows load on their machines, on their network link, where given content is placed. The bottleneck may be storage, network or CPU processing, different for different sites and content type. They need to direct traffic on request basis considering this and at least the hostname from the URL. That's why DNS is used first.
Varnish (HTTP caching only) also uses one thread per client. I believe worker threads are used to handle reuqests while a dedicated thread handles all the idle connections between requests using epoll(). Also per-threads stack size is lowered so thousands of threads don't occupy massive amount of memory.
Single threaded HTTP servers have their own issues. If the bottleneck is the storage then lack of async open()/stat() and some other calls is problematic. We feel that serving hundreds of millions of files (long tail content) from slow storage using nginx. For that reason you can configure nginx to spawn multiple processes.
Can I ask if your whole content can be stored on a single server so content is simply replicated everywhere or there is some layer above that that directs requests to the specific group of servers storing the requested content? I assume the described machine is not just part of tiered cache setup since I don't think nginx capable for complex caching scenarios.
AFAIK no. The point of DRM is to prevent recording / playing the media on a device without decryption key (authorization). So the goal is different than TLS that is used by the client to ensure the content is authentic, unaltered during transmission and not readable by a man-in-the-middle.
But do we really need such protection for a TV show?
"Metadata" in HLS / DASH is a separate HTTP request which can be served over HTTPS if you wish. Then it can refer to media segments served over HTTP (unless your browser / client doesn't like "mixed content").
I have some experience serving static content and working with CDNs. Here is what I find interesting / unique here:
- They are not using OS page cache or any memory caching for that, every request is served directly from disks. This seems possible only when requests are spread between may NVMe disks since single high-end NVMe like Micron 9300 PRO has max 3.5GB/s read speed (or 28Gbps) - far less than 800Gbps. Looks like it works ok for long-tail content but what about new hot content everybody wants to watch at the day of release? Do they spread the same content over multiple disks for this purpose?
- Async I/O resolves issues with nginx process stalling because of disk read operation but only after you've already opened the file. Depending on FS / number of files / other FS activities, directory structure opening the file can block for significant time and there is no async open() AFAIK. How they resolve that? Are we assuming i-node cache contains all i-nodes and open() time is insignificant? Or are they configuring nginx() with large open file cache?
- TLS for streamed media was necessary because browsers started to complain about non-TLS content. But that makes things sooo complicated as we see in the presentation (kTLS is 50% of CPU usage before moving to encryption offloaded by NIC). One has to remember that the content is most probably already encrypted (DRM), we just add another layer of encryption / authentication. TLS for media segments make so little sens IMO.
- When you relay on encryption or TCP offloading by NIC you are stuck with that is possible with your NIC. I guess no HTTP/3 over UDP or fancy congestion control optimization in TCP until the vendor somehow implements it in the hardware.
I had brief experience managing such IT team after which I've gained big respect for them. I've never seen such low budget to amount of work ratio in my IT career. Everybody expects them to fix the "trivial" problems immediately but reality in my case was:
- 2 "IT guys" for > 500 headcount company - If you look at IT guy instant messaging app you will see many unread notifications all the time. Without tickets it's impossible to properly prioritize and divide work. "Tickets" for everything" was one of my first change. - They are responsible for the broadest set of technologies / vendors I've ever seen. That includes: networking (including cabling, switched, wireless routing, VPN, firewalls), laptops, office apps, conferencing hardware (and software), IT security, accounting and warehouse software (including their MS SQL databases,) windows servers, virtualization, managing email accounts and maaaany more. - without tickets they can't show real amount of work they are doing so justifying adjustment budget and staffing for the need is a big challenge. On the other hand everybody expects to fix trivial problems without forms (so most of their work!). - Upper management don't perceive the as value but only as a cost - trivial problems could be fixed faster but that would require supplies for spare parts. This is bigger than you may think: few generations of laptops in use, office hardware, spare monitors, docking stations etc. That require office space for a small "warehouse" which is not free. How to justify this need considering the previous points? - many people fix "trivial problems" themselves: replacing cable, "borrowing" cable/mouse/monitor from a different desks. That creates more work for the IT guys: more issues because of incomplete office hardware.
Note that they are using 10GbE network when even a single NVMe disk used in the test has few times more bandwidth so the bandwidth results are constrained by the network hardware.
Creating on-premise infrastructure a year ago we went for 2x25Gbps network. This or even 100Gbps seems to be current "mainstream" and 10GbE is definitely not enough for NVMe speeds available today.
I categorize this as bugs caused by data inconsistency because od data duplication. That includes:
- Using asynchronous database replication and reading data from database slaves - Duplicating same data over multiple database tables (possibly for performance reasons) - Having additional system that duplicates some data. For example: in the middle of rewriting some legacy system - a process that was split into phases so functionality between new and old systems overlap for some period of time.
Based on my experience I always assume that inconsistency is unavoidable when the same information is stored in more than one place.
If you have your own domain, then move it to one of the listed DNS providers and use DNS challenge with ACME:
We are using using certbot + cloudflare whis way. There is no HTTP request, certbot makes a temporary DNS record using the Cloudflare API to satisfy the challenge so you can run the script anywhere. Then copy the cert to the device that needs it.
DNS providers supported bycertbot:
https://community.letsencrypt.org/t/dns-providers-who-easily...
From the described mistakes two come from lack of understanding how exactly DNS works. But I agree it's in fact hard, see [1]).
1. "This strict DNS spec enforcement will reject a CNAME record at the apex of a zone (as per RFC-2181), including the APEX of a sub-delegated subdomain. This was the reason that customers using VPN providers were disproportionately" - This is non intuitive and maay people are surprised by that. You cannot create any subdomain (even www.domain.tld) if you created "domain.tld CNAME something...". Looks like not every server/resolver enforces that restriction.
2. "based on expert advice, our understanding at the time was that DS records at the .com zone were never cached, so pulling it from the registrar would cause resolvers to immediately stop performing DNSSEC validation." - like any other record, they can be cached. DNS has also negative caching (caching of "not found responses". Moreover there are resolvers that allow configuring minimum TTL that can be higher that what your NS servers returns (like unbound - "cache-min-ttl" option) or can be configured to serve stale responses in case of resolution failures after the cached data expires [2]. That means returning TTL of "1s" will not work as you expect.
[1] https://blog.powerdns.com/2020/11/27/goodbye-dns-goodbye-pow... [2] https://www.isc.org/blogs/2020-serve-stale/
Generally agree, for basic configuration it just works. But I was planning to use IPv6 in some network segments in my company. Providing proper VPN (road warrior style) for employees to access such network isn't simple with IPv6 where every address is routable over internet. Morver Fortigate devices we have don't have such functionality.
As I understand my favorite resolver - unbound - has separate "infra" cache that also caches RTT times, and "down" status of name servers:
https://nlnetlabs.nl/documentation/unbound/unbound.conf/#inf...
Our security team complained that we have some services like monitoring or SSH access to some Jump Hosts accessible without a VPN because VPN should be mandatory to access all internal services. I'm afraid once comply we could be in similar situation where Facebook is now...
I think the described machines were designed for a very specific workloads even in CDNs world. As I guess they serve exclusivelly fragments of on-demand videos, no dynamic content. So high bytes to number of requests ratio, nearly 100% hit ratio, very long cache TTL (probably without revalidation, almost no purge requests), small / easy to interpret requests, very effective HTTP keep alive, relatively small number of new TLS sessions (TLS handshake is not accelerated in hardware), low number of objects due to their size (much fewer stat()/open() calls that would block nginx process). Not talking about other CDN functionality like workers, WAF, page rules, rewrites, custom cache keys, no or very little logging of requests etc. That really simplifies things a lot compared to Cloudflare or Akamai.
Same here. mSBC coded that started working for me around Fedora 33/pipewire 3.2x is a night/day change for me. I can use my Jabra bluetooth headset with high quality audio instead of really crappy sound. I think neither Windows nor OSX can do that with regular bluetooth adapter built in laptops.
After some searching I've found that Supermicro is selling ATX power supplies with SMBus (hopefully PMBus really) connection in reasonable price but never had one and I'm not able to find any review.
Example:
Server PSUs support such communication (looks like this is PMBus protocol, variation of SMBus, which is variation of I²C). In servers it's connected to the BCM and usually allows to read input and output power, FAN speed, temperatures and status (good/bad). The BCM often records historical data, including peak power (in last 1h, 24h), etc.
After quick searching is looks there are people reading this info with a MCU [1]. But I think it would be hard to find such PSU in ATX form factor.
There are advantages of using HTTP cache on application side even if you use CDN:
- Varnish can guarantee that a given resource will be fetched at most once every TTL expiration since you can set up one instance to do so (+ some HA solution like hot standby). This complements a large CDN that is highly scalable but at a cost that there is no hard "synchronization" between servers. You will see many requests for the same resource, even when the CDN uses cache hierarchy because they can't risk such bottleneck not knowing in advance your traffic pattern. You can intentionally do that with Varnish and benefit from that.
- Complex caching rules, even including what to do if application is not available/slow. That includes serving stale content, how long to serve stale after TTL expires, guarantee that fetching a new version is performed in background. Example: https://varnish-cache.org/docs/trunk/users-guide/vcl-grace.h...
We've used varnish + some other magic in addition to Cloudflare to withstand Black Friday when marketing insisted the promo has to start at specified time, anuonced much earlier to all customers. The landing page had up-to date info on available products updated almost instantly, thanks to serving stale (in practice < 1s) content and guaranteed prefresh in background. Cache key was properly set up to exclude tracking elements from URLs (from FB, adwords etc) and only include what is necessary.
You're right. I'm wasn't really serious. Since I'm in the middle of calculating costs of own servers in rented racks in Poland (you're right labor is more difficult than hardware) let me imagine the rest of the infrastructure (probably not all) for this "projects", just for fun:
- network switch Juniper EX4600 (10Gbps ports) + 3rd party optics ~$11k
- cheap 1Gbps switch for management access <$1k
- some router for VPN for management network - $500
- 1Gbps (not guaranteed) internet access with few IPs ~$350 / month
- 100Mbps low traffic internet access for the management/OOB network.
Time to get the hardware - 2 months. Time to rent and install hardware in rack - about 1 month. I don't count configuring the software.
This setup is full of single points of failure so I would consider it one "region" and use something like CEPH + some spare servers in each "region". That way you don't need to react immediately to hardware failures. Just send a box of hardware from time to time to the DC and use ~$20-40h/h remote hands service to replace the failed drives or whole servers. You could also buy on-site service from the hardware vendor for 1-3 years adding some cost.
I think the most important thing would be have a cleaver person who design a fault tolerant system, automatic failover, good monitoring and alerting so that any on-call and maintenance job is easy and based on procedures. That way you could outsource it. Only then it might have some sense.
10x Supermicro SSG-6049P-E1CR60H servers (60 x 3,5" HDD in 4U enclosure) - $5k each
600x WESTERN DIGITAL Ultrastar DC HC550 18TB (10800PB in total) - $500 each
~$350k in hardware, up to 20kW energy consumption, should fit in two rack towers. You can host it for about $1.5k somewhere. All assuming no redundancy :)
IPv6: curl http://ipv6.whatismyip.akamai.com curl http://ipv6.whatismyip.akamai.com/advanced
IPv4: curl http://whatismyip.akamai.com curl http://whatismyip.akamai.com/advanced
Both hosts are distributed around the world using the Akamai platform and should be fast for everyone.
Some providers are doing that. For example you can rent such bare metal servers from OVH's cheap brand So You Start. But that doesn't make sense if you need something more that one cheapest server. Consider the total cost of buying and hosting such servers compared to the resources they provide.
You are paying more but mainly when the capabilities are better. Example for SSD: Samsung 860 PRO 1TB and "for Data Center" Samsung PM883 1TB both have about 1200 TBW lifespan, both cost about $180.
On the other hand 8-core Xeon 4215 is almost 2 times more expensive than 8-core consumer CPU but it has 3x the memory bandwidth (6 channel controller vs 2 channel), 1TB of RAM per processor vs 128GB, 48 PCIe lanes vs 20 and supports 2-way SMP. That means you can put few times more capacity (cores, memory, in 1U rack enclosure than with consumer hardware. That additional capacity can also share 1 PSU, 1 network card.
I'm not even talking about ECC memory and remote management capabilities available only in server grade hardware.
The way I look at the major improvements in the two recent HTTP versions is:
HTTP/2 was needed to better utilize the bandwidth by keeping the TCP connection busy without silence period. That means instead of "request - RTT delay - response - RTT delay - request - RTT delay- response ..." we have "request - request - request - response - response - response ..." without pauses between responses. These pauses in HTTP/1.1 cause the congestion windows drop to initial value and slow start after each response. For HTTP/1.1 the workaround was to establish multiple TCP connections to fetch resources in parallel. This + using multiple domains for assets was best practice with HTTP/1.1 and is not needed (or even discouraged) with HTTP/2.
The above change has a drawback: for links with high latency and lower reliability a single packet loss is causing this whole pipline to stall. The cause is how TCP retransmission + congestion control works. This is in contrast to multiple HTTP/1.1 connections where just single connection would stall and take time to recover to full speed. This has bigger impact on performance in wireless connections and content like live video streaming so something that the world move to.
To fix that HTTP/2 moves congestion control to upper layer that understand what is transmitted. Loss of single packet no longer causes the whole pipeline stall, it affects only the resource for which the loss occurred.
This all is needed especially in mobile networks. In older times the a good CDN was used to put content close to the user (or at least terminate the TCP connection close to the user) lowering the round trip time, TLS handshake time. With mobile connections becoming more popular this doesn't work because the big delay is in the "last mile" - in mobile link. You just can't move the CDN's server closer to cut this latency.
Of course there are also other improvements like faster connection establishment (with TLS).
For web serving it's possible to have IPv6-only server accessible to anyone. Just use Cloudflare. I'm not sure it's documented but I started using IPv6-only origin server behind Cloudflare and it seems to work OK for people connecting both IPv4 and IPv6. The client IP is in CF-Connecting-IP, even for IPv4 connections. For me it just works.
One of nice options in recent versions of OpenSSH is the "expiry-time" key option you can put in authorized_keys. There are also other options handy to restrict the usage of the key, for example you can limit the key to be used only as a jump host. One can combine them so the users on the jump host will not be able to execute any command and even not able to edit authorizd_keys file and remove the "expiry-time" option.
That is a step forward. But I'm still missing few features. Most annoying is that there is no way to make multiple changes as one atomic change. From time to time I need to change an A record to a CNAME record. It's not possible without first deleting the A record and then creating the CNAME record. This creates a short window when the record is missing. The NXDOMAIN response can be cashed by recursive resolvers creating temporary outage for some users / whole networks. Unfortunately I'm also unable to change negative TTL for the zone which is set by CF to 1h. For that reason I have to do such changes in the middle of the night and even then I'm nervous doing that for important records.
You're right. I was wondering how this company is able to collect such data from real users. It's registered in an apartment in Cracow, Poland, a 1-person company till last year (here is their company info in polish: http://www.krs-online.com.pl/perfops-sp-z-o-o-krs-10397838.h... , still an apartment address) . Now looking at the Network tab in Developer Tools I can see they are running the tests from JS on their site using visitors of cdnperf.com. I can see requests for URLs like /500b-bench.jpg?t=1552402827417 to various domains.
CDNPerf and DNSPerf services most likely do testing from some pool of VPSes or hosting services, although I can't see such details in their description. If you are interested how your service performs when serving to other servers, you can use it. If you are interested how it performs when serving to real people (home, mobile, offices), I wouldn't trust it.
Your comparison is missing Akamai which is probably the most feature-rich and largest CDN.