HN user

ultrahax

55 karma
Posts0
Comments36
View on HN
No posts found.

I keep a Starlink subscription as my WAN2, after someone started a fire in a nearby fiber cabinet and managed to knock out Frontier and Spectrum at once. It’s worth it to me as a bullet-proof WAN2, especially working from home.

Related, to this day my Frontier connection black holes all my traffic a couple of hops inside their network if I do sustained line-rate UDP (WireGuard, for example) for more then three or four minutes. After some LinkedIn sleuthing I got in touch with their local NOC director who tells me it’s some piece of gear that hangs so bad he can’t remote into it and it stays down until the watchdog fires. Been that way for two years now.

As a games coder I was glad when the xbox 360 / ps3 era came to an end; getting big endian clients talking to little endian servers was an endless source of bugs.

Tangentially, I've faced some interesting challenges getting a multi-gigabit Wireguard VPN operating through my 2Gb Frontier connection.

My UDM Pro seems to top out around ~800mbit per UDP stream - pegged at 100% CPU on a single core. Likely it can't keep up with the interrupt rate, given it's ksoftirqd pegging it. Replaced UDM Pro with a pfsense machine.

Then I started getting 100% packet loss on the edge of Frontier's network after a couple of minutes of sustained UDP near-line-rate throughput. In the end, after trying and failing to explain this to Frontier's tech support, I reached out to their engineering management on LinkedIn, and got put in touch with the local NOC director. Turns out to be some intermediate hop is rebooting after a few mins, and they're "in contact with the manufacturer". Haven't heard back in a few months.

tldr as >1Gb connections become more ubiquitous, other bottlenecks will become apparent!

Citibank did this to me. Venmo'd my architect for some work he was going on my house, kablammo, account closed, no notice. Was just lucky it was an account set up specifically for work on that house and not my that's-where-my-paycheck-goes account.

I have to be careful what I say here, given I work for ATVI, I'm former Demonware myself.

I do feel safe in saying though that the login queues are _definitely_ not hype.

They're designed to constrain a quite-complex distributed system to a login rate that has been load-tested thoroughly, e.g. they know it'll work at that rate.

They handle online services for pretty much any Activision-published game, with carve-outs for Blizzard and King - they still do most of their own backend stuff themselves, with some collaborations.

I’m not sure your information wrt Call Of Duty is correct.

To my knowledge, the client timestamps their inputs and sends them to the server; the server will then rewind the state of the world to the time of the input before applying it. RTT isn’t an input. Each snapshot from the server includes the server world timestamp of that snapshot; the client will gently lerp its clock to match this per frame.

Source - I’m a COD engine developer the last ~15 years or so.

When I worked at IBM circa ~2006 you'd get a written warning called a clean desk violation if you left your workstation unlocked.

I wrote a little daemon that'd l2ping my Nokia brick phone; if it didn't get a response for 30 seconds it'd invoke xscreensaver. Saved me a lot of paperwork.

I currently work at a Call of Duty studio. My favorite hacks ( not super high tech, but the ones that had the most impact for the least code, and the ones I feel I can talk about.. ):

* Put together a little box that polls varied knobs on a USB midi device to mangle the traffic going across its two interfaces. Allows for real time latency / jitter / packet loss testing https://twitter.com/ultrahax/status/1200902654882242562

* Studio LAN game browser didn't work across subnet boundaries ( studio is a couple of class B's ). Wrote a little daemon that'd take game discovery packets from one subnet, mangle the src addr, and send it on its merry way. Everyone can see everyone's games, happy producers.

Been at current company for 15 years, from software engineer through to principal engineer. I don't see myself leaving. The work is for the most part interesting, the money is pretty decent, and the people I work with are broadly aligned on the same goals and culture.

End of page allocation has been a real life-saver. I have it rigged up to the core allocators of the game engine I work with, so you can switch it with a cmdline switch; if you suspect an overwrite, you switch on the EOP allocator ( using more memory temporarily ), and bang, you get a segfault where you went off the end / used-after-free.

Most engines I've seen will delta all the entity states against the client's last acknowledged state. Costs some memory and computation on both sides to keep the deltas valid, but keeps the state update to under an MTU, generally.

I work on something that requires a reasonably cooperative NAT and unmolested real-time UDP traffic. I've seen varied failure-modes from corporate firewalls over the years - from simple NAT table overflow causing rapid source port switching, to the firewall appliance downloading an update and deciding UDP packets of a certain size ( and ONLY of a certain size.. ) were bittorrent and hence were to be blackholed. That was an interesting one to track down. I've also seen it block diagnostic GETs to varied bits of cloud infra, due to someone at some point in the distant past hosting porn on that particular IP. Not to mention just good old strict NATs..