sonthonax

The thing that people forget about databases is that they are just really complicated ways of writing to a file.

The crux of all these “you only need sqlite” posts isn’t that you need SQLite, it’s that your architecture only ever needed flat appendable log file to begin with.

Once you realise that SQLite is just an aggregated view over this log it calls into question if you needed a third party query engine at all, suddenly the whole NoSQL vs SQL debate becomes a meaningless implementation detail.

Server based databases aren’t special beyond the fact they’re just distributed synchronisation primitives over files on a file system. But the utility of this is quite limited in practice, writing to a table will always require some sort of synchronisation, using SQLite vs Postgres is just a case of where you put the lock - in your own code or in a 3rd party server.

They’re extremely temperamental vehicles because they’re not fit for purpose.

The original Range Rover is an upmarket agricultural vehicle, and used the same platform from 1970 until the 2000s. When you actually use them as intended, for off road farm work you realise they’re better off road than they are on road.

You just set the alembic version yourself in the alembic.version table. It’s not rocket science.

To the contrary, a simple expression language is one of those things that can easily be done in that size.

Yeah, parent comment is deeply unserious.

Imagine you're writing trading software, you have an algo go haywire and it machine guns the whole order book, and then you refuse to put a "max order size" outside of the algo to stop it from happening again because you can't figure out why it happened in the first place.

Try telling a regulator or your boss that was your reasoning.

Just not true. Lots of oil companies have a vested interest in renewables now. A lot of the infrastructure for offshore oil is being redeployed for offshore wind.

600 people on a median 200k salary in New York, when including taxes and obligatory benefits is about $150 million dollars a year just in payroll. This is an enormous spend. Unlike a game studio or a software company like Adobe, these people are all operational cost centres, they're not the bread and butter of the firm.

In a true software firm, more capital invested in headcount loosely means more features which is what drives sales. Here you can attribute a per employee profitability. When you hire more employees at a tech firm, the expectation is that you become more profitable.

At a non-tech company like a newspaper, technologists are a cost centre, you invest in technology in order to be more efficient, and hire as few people as possible.

Of course some technologists at the NYT will be profit centres. like those who work on advertising operations and games. But the NYT at its core is a news firm, it makes money on its influence and prestige which is driven by its journalism. Writers and content creators are the profit centres, everything else is a cost of doing business.

I know they have a larger range of services, but are they 20x the breadth of services? Think of what other companies do with 600 tech people and compare what the profitability per employee should be at an tech organisation of such a scale.

They have at least 600 employees, get some perspective on just how much that is. It's least $150 million dollars a year in technology payroll alone. They have more technology workers than some major newspapers have in staff alone.

Why do you think that people on 200k should have a union? How does it benefit anyone besides people looking for sinecure jobs?

Unions were needed to combat the powers of corporations where you are in some ways trapped by the relationship; like miners in appalachia who had to fight cartels of mining companies who suppressed their wages; or teachers who have one state employer.

If you're a competent tech worker, you're at the other end of the spectrum. So much so you're generally advised to switch jobs every few years in order to maximise your salary. There's nothing binding you to a particular employer, particularly in New York!

To be honest, if you're a good technologist, you should be looking at making 400k a year in at a trading firm. If you're working for the NYT you're doing it because you're weirdly attracted to the prestige of working there, or you're looking for a cushy job.

If the job is no longer as cushy as you want, have the dignity to leave and find other employment rather than trying to hold your employer hostage.

If it's low for the market why choose to work at the NYT?

Being a tech worker isn't like being a steelworker where you're kinda screwed if the plant shuts down. You're fully capable of moving job if you're dissatisfied.

Why they have 600 people is a question for management, no?

It's a question for everyone at the organisation. If you're joining a firm with a massively bloated headcount you need to think if you're going to be there in a year's time when someone in management finally gathers the political capital to slim your department down to sane levels.

It's certainly not a position to start bargaining even more concessions from the hand that's feeding you.

The New York Times is a glorified blogging platform. Not to long ago it was a Wordpress site.

I'm fully aware of how jaring it is for the median HN reader to hear this, but maintenance of a news website isn't the kind of skilled labour that commands a 250k a year paycheck anymore.

Linux outages and upgrade fuckups happen all the time. The difference is that Linux isn't on a unified upgrade cycle so issues come as a trickle rather than a dulge, so the problems are usually localised to a corporation and don't make the news.

Also Linux fanboys will usually blame the system admin for not configuring things properly if things break: "it's not the operating system, it's <something stolen from OpenBSD>".

End of the day Linux is only popular because of the inertia UNIX had on mini-computers/servers. For standard end users GNU Linux is lightyears behind Windows and macOS in terms of usability and stability.

Surely you've worked with other traders who get confused by import statements. Whole point of this file is to provide a common interface for common math operations that might be exposed by either plain python or pandas/numpy functions.

You're obviously not the target market if you're able to review good and bad code.

Wait until you learn that they believe they need Kafka. Their engineers are probably bitter they work at a media company and not a FAANG.

https://www.confluent.io/en-gb/blog/publishing-apache-kafka-...

The Monolog is our new source of truth for published content. Every system that creates content, when it’s ready to be published, will write it to the Monolog, where it is appended to the end.

The Monolog contains every asset published since 1851. They are totally ordered according to publication time. This means that a consumer can pick the point in time when it wants to start consuming. Consumers that need all of the content can start at the beginning of time (i.e., in 1851), other consumers may want only future updates, or at some time in-between.

As an example, we have a service that provides lists of content — all assets published by specific authors, everything that should go on the science section, etc. This service starts consuming the Monolog at the beginning of time, and builds up its internal representation of these lists, ready to serve on request. We have another service that just provides a list of the latest published assets. This service does not need its own permanent store: instead it just goes a few hours back in time on the log when it starts up, and begins consuming there, while maintaining a list in memory.

Absolutely insane. The only reason this works is that the NYT publishes less than 300 articles per day so you can get away with doing un-indexed full table scans of your entire database. But the engineers can put "I created a log based time-series architecture on their resumes".

Thanks for the thorough response.

But firstly:

If your firm has had KDB for ages there's a good chance it's big enough to be signed up to one of the research groups who maintain a set of test-suites they will run over a vendor's latest hardware offering, letting them claim the crown for the fastest Greeks or something. If your firm is a member you may be able to access the test-suites and look at how the data in the options tests is being written and read, and there are quite a few, I think.

Unfortunately my firm isn't that big ~150 in total and maybe about ~40 developers, if which there are 2 full time KDB devs who's job is mostly maintaining the ingestion and writing some quite basic functions like `as_of`. There's only two people who work on our options desk as developers, so there's a lack of resourcing for KDB. When I have these issues with KDB around performance, it's quite hard to get support within my firm from the two KDB devs (one of which is very junior).

I've never worked on options data and so can't opine on the problems it presents

The thing about options data is that it's generally lower frequency but a lot more complex. If spot data is 1 dimensional, and futures data is 2 dimensional, options are 3 dimensional. You also have a lot more parameterizations which leads me to the second point :)

you may not want to load data out of KDB a day at a time. Try to do the processing in KDB

Just to give you a very specific example of the processing I need to do. I have a data structure in KDB like this (sort of typescript notation):

     row = mapping<datetime, { a: number, b: number, mu: number: sigma: number, rho: number}>

    f = log_moneyness - m;
    total_var = a + b * (rho * f + (f * f + sigma * sigma).sqrt())
    vol = total_var / time

Now I have a base in which I can start generating data like the delta, but this also requires a lot of math.

I was pulling this stuff out of KDB because I already had my code in rust that does all of this.

You said you're scared to do queries that return a lot of data, and that it often freezes. Are you sure the problem is at the KDB end?

Yeah, I'm pretty sure in my case. We have some functions designed for getting data written by the KDB guys. Even for functions that return 30 something rows, like an as_of query it takes ~10s.

What did you do for your backtesting testing?

Loading your rust code into your existing KDB data lake and periodically updating it will be a significantly smaller lift than rewriting your data lake.

This is probably going to be what I do until KDB creaks over.

You sound very set on some sort of rewrite

I vacillate between the two things. I'm personally used to data engineering with parquet and spark, which are widely used outside of finance, and don't have expensive vendor lock in.

And then I realise that I'd have to own this stuff, and my job isn't a data engineer, and I'm a quant dev.

So I’ve actually contributed to this project.

My concern with adding custom libraries into KDB, while better than writing duplicative Q code is the maintenance nightmare of keeping them up to date in KDB.

It’s still an investment but I need to be aware of of the risks and downsides.

If it’s partitioned this should be even faster.

What does "move the code to the data" mean in practice?

This grasps at why I’m finding KDB so hard to use. I’ve written a pricing and risk library in Rust. Historical data really needs to be taken processed in Rust rather than KDB.

Unless there's some significant new data that might change that decision, it is what it is.

This is what I’m grasping at.

Are the challenges of writing a KDB system for back testing derivatives data which needs to work in tandem with a rust pricing library substantially different from a KDB engineered for back testing spot data.

One has complex and specific math and one doesn’t.

Is that a pun?

I care about my job, enjoy my work, and I take pride in being able to deliver things, but I don't know how to deliver value for my traders here.

I'd really appreciate some perspective from seasoned data-engineers who might have seen this KDB as a data-lake pattern before, and what they did about it. Not just technically, but how they managed the organisational change of KDB for the KDB quants.

I also just don't really know where else to ask. There's not really an online KDB community, and you have a lot of KDB devs who are really good at KDB but know barely anything else which makes me skeptical of their advice.

Service workers, which require a completely different mode of thinking from the rest of front-end development and have more in common with web server programming than DOM manipulation.

This is spoken like a true front end developer who's never used a thread before.

Severe developmental disorders or severe cognitive impairments where significant support is required, including but not exclusive to: physical disability, intellectual disability, autistic spectrum disorders, brain injury.

I think that's a deliberate misinterpretation. It obviously means 'low functioning' autism rather than someone who's awkward in adulthood but otherwise leads a normal life.

I can see you're already being downvoted because Hacker News has a really juvenile attitude to DRM.

No one has the right to YouTube (or free media for that matter).

I've noticed that M series Macs have extremely fast disk drives that the OS uses as swap quite efficiently. I've frequently used all my RAM on my Mac and barely noticed any slowdown when it starts swapping.

Isn't PyO3 just a set of macros and helpers around FFI and ctypes?

They tend to use the tools provided by Government Digital Services.

Why would anyone trade an EMP for all their cities?

ICMBs were a theoretical threat. Cold War doctrine had a more realistic (and less apocalyptic) WWIII that would be fought by tactical nukes and tanks through Europe.