Simple Precision Time Protocol at Meta

https://engineering.fb.com/2024/02/07/production-engineering/simple-precision-time-protocol-sptp-meta/

by atg_abhishek • 2 years ago

217 85 2 years ago

engineering.fb.com

Facebook continues to follow the Yahoo and AOL trajectory of exceptional and generous engineering contributions amidst an increasingly disliked suite of commercial offerings.

Reminds me of a project idea where you list out all the big companies that have GitHub projects like Comcast, Walmart, Verizon, Target and even https://github.com/mcdcorp

lukevp• 2 years ago

What did yahoo and AOL contribute?

ralph84• 2 years ago

Hadoop, AOLserver, early support for Mozilla

therein• 2 years ago

Apache Traffic Server, AOLserver (no idea why I used this a long time ago)

speedgoose• 2 years ago

Yahoo had YUI which fell out of fashion.

https://en.m.wikipedia.org/wiki/YUI_Library

fragmede• 2 years ago

yahoo pipes

mtmail• 2 years ago

Yahoo released their geographic data catalogue under open license and it still lives on as https://whosonfirst.org/

Afaik https://en.wikipedia.org/wiki/Apache_ZooKeeper started at Yahoo

https://vespa.ai/ was Yahoo's search engine for news and other content product, now spinned off (https://techcrunch.com/2023/10/04/yahoo-spins-out-vespa-its-...)

ldjkfkdsjnv• 2 years ago

Actually think that behind the scenes facebook is rapidly innovating on ai and integrating it into its ad targeting. Mets is on fire

KevinAgain• 2 years ago

Sounds squarely within the "increasingly disliked suite of commercial offerings"

te_chris2• 2 years ago

Meanwhile high earning millennials quit instagram because the product mostly sucks. Hell of a cathedral they’re building though.

daavidhauser• 2 years ago

zer00eyz• 2 years ago

> innovating on ai and integrating it into its ad targeting.

Summer child.

No one who has add space to sell wants an efficient, accurate market. Those inefficiencies create competition, create volume create profit for those with ad space to sell.

The ad market had more effective targeting and better ROI 10 years ago than it does today.

Go and try to run a CPA campaign at any cost... you can't. It's all display add's CPM garbage, carpet bomb for pennies.

NBJack• 2 years ago

Can say with certainty that it's on fire alright, in the sense that the ad data is burning away.

The rapid innovation is a survival tactic. They know the ads boat is sinking, and unlike other tech companies, they don't have much diversity in their revenue. Hence the Metaverse, AI, etc. which although neat, are not exactly making the same level of money for the organization (at least not yet). In Q4 2023, ads had a revenue of ~$38B, while the Quest revenue was a loss of $4B. AI hasn't been directly monetized directly, so it's harder to say how that's doing.

Given how critical good data is to a model, I'm not optimistic this will work for them.

queuebert• 2 years ago

Meanwhile, still no cure for cancer.

sirpunch5• 2 years ago

This HN bandwagon again. Meta has had an astounding yoy growth. Did you even see their Q4 report.

test6554• 2 years ago

If they are relying on headset sales… competition is about to heat up. Ive been waiting for anyone but meta to come out with a decent, affordable kit and I think were almost there.

sofixa• 2 years ago

YoY growth and quarterly earnings report don't give you a full picture of how much a company is liked (which is important for b2c companies like Facebook) and how well it is actually performing for the medium/long term.

GE and Boeing also had amazing YoY growth and great quarterly reports, until the underlying dumpster fire they were nurturing for years exploded in their faces and now they don't have growth nor profits anymore.

kristopolous4• 2 years ago

And AOL's stock had a 500% return in the 2010s... Yahoo had 300%.

Are you saying people like Facebook's new features like the suggested posts and what they've done to Instagram? Was their VR product line secretly a success?

These things eventually catch up.

There's also a number of fairly bad bets they've made, such as the $1 bln kustomer acquisition they spun out at a 75% loss after a year https://techcrunch.com/2023/05/16/kustomer-meta-spin-out/

They have enough capital and trust to stumble around for a while but they sure look like they're yahooing

ksec2• 2 years ago

This got me thinking if anyone interested in starting a HN Hedge Fund. HNHF? XD

ahoka• 2 years ago

By dumping over-engineered crap nobody actually needs while becoming the Nestle of software companies?

ysofunny8• 2 years ago

all because they aren't in the data-center cloud provider business

yedava5• 2 years ago

Facebook is also one of pioneers in making mass surveillance the norm. The promise of technology has been used to prey on society.

paxys2• 2 years ago

Funny how people "hate" them yet they have 4 billion active users and generate $135 billion in annual revenue. A better explanation is that outside of the HN bubble Meta's product offerings are insanely popular.

yedava5• 2 years ago

At one time cigarettes were also as popular as Meta's products.

sirpunch5• 2 years ago

I find it fascinating how much Meta's business and HN sentiment towards anything Meta are mutually exclusive.

ksec2• 2 years ago

HN Hated Facebook since its inception. There used to be people on HN defending Facebook, as a stock, or as an ad platform. Post 2015 on HN all ads are evil. Barely anyone defending them or Online ads anymore.

mlichvar• 2 years ago

To me that looks like they are reinventing NTP, but not addressing all the issues of PTP.

A big problem with the PTP unicast mode is an almost infinite traffic amplification (useful for DDoS attacks). The server is basically a programmable packet generator. Never expose unicast PTP to internet. In SPTP that seems to be no longer the case (the server is stateless), but there is still the follow up message causing a 2:1 amplification. I think something like the NTP interleaved mode would be better.

It seems they didn't replace the PTP offset calculation assuming a constant delay (broadcast model). That doesn't work well when the distribution of the delay is not symmetric, e.g. errors in hardware timestamping on the NIC are sensitive to network load. They would need to measure the actual error of the clock to see that (the graphs in the article seem to show only the offset measured by SPTP itself, a common issue when improvements in time synchronization are demonstrated).

I think a better solution taking advantage of existing PTP support in hardware is to encapsulate NTP messages in PTP packets. NICs and switches/routers see PTP packets, so they provide highly accurate timestamps and corrections, but the measurements and their processing can be full-featured NTP, keeping all its advantages like resiliency and security. There is an IETF draft specifying that:

https://datatracker.ietf.org/doc/draft-ietf-ntp-over-ptp/

An experimental support for NTP-over-PTP is included in the latest chrony release. In my tests with switches that work as one-step transparent clocks the accuracy is same as with PTP (linuxptp).

bux93• 2 years ago

I listened to a fascinating podcast from Jane Street on their solution. Pasted the relevant part

https://signalsandthreads.com/clock-synchronization/ --quote "So, we’re trying to build a proof of concept. At the end of the day, we sort of figured, “All right. We have these GPS appliances.” We talked about hardware timestamping before on the GPS appliances, and how they can’t hardware timestamp the NTP packets, so that’s problematic. We thought, “How can we move time from the GPS appliances off into the rest of the network?” And so we decided that we could use PTP to move time from the GPS appliances to a set of Linux machines, and then on those Linux machines we could leverage things, like hardware timestamping, and the NTP interleaved mode to move the time from those machines onto machines further downstream.

The NTP interleaved mode, just to give a short overview of what that means… when you send a packet if you get it hardware timestamps on transmission the way you use that hardware timestamp is you get it kind of looped back to you as an application. So I transmit a packet, I get that hardware timestamp after the packet’s already gone out the door. That’s not super useful from an NTP point of view, because really you wanted the other side to receive that hardware timestamp, and so the interleaved mode is sort of a special way in which you can run NTP and that when you transmit your next NTP packet you send that hardware timestamp that you got for the previous transmission, and then the other side, each side can use those. I don’t want to get into too much of the details of how that works, but it allows you to get more accuracy and to leverage those hardware timestamps on transmission." -- end quote

mgaunard• 2 years ago

In the ultra-low-latency space people just use PPS, either one pulse per second or 10Mhz.

WhiteRabbit and the like allow you to transmit such a signal with great accuracy.

There are timestamping layer 1 switches and NICs that support taking it as input.

bArray• 2 years ago

A big problem with the PTP unicast mode is an almost infinite traffic amplification (useful for DDoS attacks). The server is basically a programmable packet generator. Never expose unicast PTP to internet. In SPTP that seems to be no longer the case (the server is stateless), but there is still the follow up message causing a 2:1 amplification. I think something like the NTP interleaved mode would be better.

Facebook has little concern for traffic amplification that doesn't affect them. I can't find a source article for it now, but there was a time when you could take down a website hosting an image by simply posting <URL>/?<RANDOM>. I believe Facebook's (many) cache servers would individually make requests to the server until they inevitably saturated the image host's connection. I remember people complaining and it falling on deaf ears.

ysofunny8• 2 years ago

but this is not about facebook, this discussion is about the protocol

given how this industry protocols work, is likely that other big corporations that run data centers are also part of the real protocol discussion, some of those will be corncerned about traffic amplification

nvarsj• 2 years ago

SPTP does look a lot like NTP over PTP. I’m guessing they deployed this last year when nothing like it existed - the ietf draft (dated Jan 24, 2024) came much later after. They might even be involved with it. Anyways, it’s nice to see progress towards a simpler protocol that retains the precision of PTP.

bagels• 2 years ago

It's older than that, but not sure how much older.

westurner• 2 years ago

How does SPTP compare to CERN's WhiteRabbit, which is built on PTP? https://en.wikipedia.org/wiki/White_Rabbit_Project

FWIW, from "50 years later, is two-phase locking the best we can do?" https://news.ycombinator.com/item?id=37712506 :

TIL there's a regular heartbeat in the quantum foam; there's a regular monotonic heartbeat in the quantum Rydberg wave packet interference; and that should be useful for distributed applications with and without vector clocks and an initial time synchronization service

jonathanyc2• 2 years ago

Does anyone know the differences between Meta's application of Precision Time Protocol and Google TrueTime? I was hoping to find some discussion in the article but found none.

The 2022 article on the Precision Time Protocol says (https://engineering.fb.com/2022/11/21/production-engineering...):

- Adding precise and reliable timestamps on a back end and replicas allows us to simply wait until the replica catches up with the read timestamp...

- As you may see, the API doesn’t return the current time (aka time.Now()). Instead, it returns a window of time which contains the actual time with a very high degree of probability...

Which sounds similar to TrueTime (https://static.googleusercontent.com/media/research.google.c...):

- A read-only transaction executes in two phases: assign a timestamp sread [8], and then execute the transaction’s reads as snapshot reads at sread. The snapshot reads can execute at any replicas that are sufficiently up-to-date...

- TT.now() returns TTinterval: [earliest, latest]

I tried Googling "Precision Time Protocol TrueTime" but the only reference I could find is a HN comment by someone else from 2022 :) https://news.ycombinator.com/item?id=33696752

j16sdiz• 2 years ago

TrueTime is designed for monotonicity. It give bounds for concurrency control.

PTP do wallclock time.

touisteur• 2 years ago

Just one slight trapdoor I triggered in my travails. I don't know whether PTP does PLL slaving but while NTP is also for wall time, it has an effect on clock_gettime(CLOCK_MONOTONIC) so if you need a local clock without external (network, I mean) influence, CLOCK_MONOTONIC_RAW is there for you. And luckily, on recent kernel it has been vDSO'd.

mjb• 2 years ago

Doing consistent reads this way is a fairly old technique, at least back to the 80s (I can dig up some papers if folks are interested).

Spanner, rather famously, uses this range approach, but a good number of other systems are based on similar approaches. The important thing for reads is getting an upper error bound from the clock, having storage that can perform reads at that time (eg using MVCC), and having a way for storage to know when it's seen all writes before a timestamp.

This can be implemented using PTP, and that's the approach we use at AWS inside some database services.

jonathanyc2• 2 years ago

Thank you! I’d be very interested in any papers you have the time to dig up. I feel like even one paper would give me a good pointer into the literature from which to start exploring.

aseipp• 2 years ago

PTP is part of the backbone of something like TrueTime. Meta uses their PTP infrastructure for a lot of the same basic fundamentals, including consistent read replicas, like Spanner does. PTP is a protocol for synchronizing the wall clock time of a set of computers under very tight error bounds, so that all of the servers have a very consistent and tight "view" of what time it is.

Now, independently of replica strategies, it's important to understand TrueTime is an API, to be clear, as you noted. It lets you represent some continuous interval of time based on the system clock error. You can then use this API to do things like ask "Did timestamp A occur before B?" And you can get an API equivalent to TrueTime on your own random Linux machine, using the Clock-Bound tools from AWS, combined with the chrony NTP daemon: https://github.com/aws/clock-bound

The API and all that is pretty basic, actually. Rather, the secret behind TrueTime and the like is just a huge amount of reliability engineering to ensure that the upper bound target (7ms IIRC from the Spanner paper) is actually maintained reliably and accurately, at global scale. That reliability means engineers can build on it with specific guarantees. You can slap chrony, ClockBound-D, and a PTP card into your rack and program away. But it's a matter of engineering guarantees more than like, theoretical computer science. Theoretically speaking, TrueTime can only help you definitively establish that some event A has actually "happened before" B in a distributed system. That's extremely powerful but it needs muscle backing it up to be true and useful in practice.

AWS has publicly advertised that EC2 has access to their 'AWS Time Sync' service, which is a globally consistent clock synchronization service designed to provide the backbone needed for services like TrueTime, and is freely available. Assuming you are willing to trust the EC2 network and AWS engineers, you can slap chrony and ClockBound-D on your AWS instances and get a TrueTime-like API with very tight global error tolerance, which would allow you to do consistent read replicas like Spanner, among other tricks

Alifatisk• 2 years ago

I see this as nothing more than custom built NTP that suits them. For the general public, I think NTP is still most suitable.

martinky24• 2 years ago

Which... is fine! They're allowed to open source solutions for problems that are unique to their massively scaled architecture! Others can learn from it -- even if others don't have the same problem.

It is worth keeping in mind that very, very, very few people/companies are required to solve the same problems a company like Meta is.

natch• 2 years ago

“grand cancellation acknowledgements”

I love it. These sound grand.

Simple Precision Time Protocol at Meta

Related Stories