evanweaver

Games are a medium for artistic expression but saying that 3D hardware was designed to improve art production, or that NVIDIA was first in market, is incorrect. The hardware was designed to improve the consumption experience of something that is a mix of programming, game mechanics (which are both math and psychology), and potentially various art forms including visual, music, and narrative. It all needs to add up to fun or it won’t find much of an audience.

Gamers aren’t primarily spending time or money for the art and neither was NVIDIA. I will grant that the hardware improvements did make the visual aspects more lifelike and detailed and that allowed for increased artistic range, but production costs generally increased accordingly.

The idea didn't spring fully formed from SGI. It was a natural extension of 2D graphics accelerators which were initially used for engineering (high value, small market) and later for business applications generally and games (lower value, large markets). 3D acceleration took the exact same path, but the utility for gaming was much higher than the general business utility.

Of course graphics hardware was also used for more creative purposes including desktop publishing, special effects for TV, and digital art, so you will find some people in those communities vaguely wishing for something better, but artistic creation, even for commercial purpose, was never the market driver of 3D acceleration. Games were. The hardware was designed for gamers first, game programmers second, game artists a distant third, and for nobody else.

The closest thing to an "art computer" around that time was the Amiga which targeted the design/audio/video production markets.

Yeah, this history is just wrong. What really happened is as so:

Early 90s: SGI invented OpenGL to make realtime 3D graphics practical, initially for CAD/CAM and other scientific/engineering pursuits, and started shipping expensive workstations with 3d accelerated graphics. Some game artists used these workstations to prerender 3d graphics for game consoles. Note that 2D CAD/CAM accelerators had already been in market for nearly a decade, as had game consoles with varying degrees of 2D acceleration.

Mid-90s: Arcades and consoles starting using SGI chips and/or chip designs to render 3d games in real time. 3DFx, founded by ex-SGI engineers, created the Voodoo accelerator to bring the technology down market to the PC for PC games, which was a rapidly growing market.

Late 90s: NVIDIA entered the already existing and growing market for OpenGL accelerators for 3D PC gaming. This was a fast-follow technical play. They competed with 3DFx on performance and won after 3DFx fell behind and made serious strategy mistakes.

Later 90s: NVIDIA created the “GPU” branding to draw attention to their addition of hardware texture and lighting support, which 3DFX didn’t have. Really this was more of an incremental improvement in gaming capability.

Early 00s: NVIDIA nearly lost their lead to ATI with the switch to the shader model and DirectX 9, and had to redesign their architecture. ATI is now part of AMD and continues to compete with NVIDIA.

Mid 00s: NVIDIA releases CUDA, which adapts shaders to general purpose computation, completing the circle in a sense and making NVIDIA GPUs more useful for scientific work like the original SGI workstations. This later enabled the crypto boom and now generative AI.

Of course, along the way, OpenGL and GPUs have been used a lot for art, including art in games, but at no point did anybody say "hey, a lot of artists are trying to make 3D art, we should make graphics hardware for artists". Graphics hardware was made to render games faster with higher fidelity.

I know people for whom the traditional way of building a web app is completely foreign. I am curious how you would describe the concept and tools to someone who has never encountered them before outside an SPA architecture.

Curious what makes a cache better to you.

I mean, yeah, this is why people stopped using this pattern. But these problems are getting solved, especially in Fauna:

1. Schemaless/document/schema-on-need databases like Fauna don't mandate the application breakage on every change that SQL does

2. It's hard to reason about if its not transparent, but it can be transparent now, see below

3. Fauna is a temporal database, which acts like version control on your stored procedures, so you can easily check and revert any change

4. Fauna is serverless and horizontally scalable without consistency/latency impact

5. This was definitely a problem when you were occupying precious CPU cores on a vertically scaled RDBMS with business logic, but compute in Fauna or in serverless lambdas scales horizontally indefinitely

Stored procedures and the integration database have come back for our users in a big way. It would be great to hear examples of how others are applying this pattern with other databases and APIs.

"There are three kinds of lies: lies, damned lies, and benchmarks." - Mark Twain

What other methods does the community use for measuring distributed latency?

This is possible in Fauna. All documents are actually collections of document versions within the configurable retention period. If you ensure that every writer decorates the document with the facets you want to search by (ip address, etc.) then you can construct indexes on those facets and query them temporally. They will return event records that show when the document entered the index (when that ip updated it) and left the index (when a different ip updated it).

Map the index additions and their timestamps onto the documents themselves and you can retrieve the entire state of each record that the ip wrote at the time that it wrote it. If you want to know specifically what that ip changed, then diff it with the previous record, for example, to filter down to updates that only changed the email address.

Can you give me an example of the kind of query you have in mind?

Fauna is temporal, so yes. Normal forms not required.

As far as I can tell, GemStone/S doesn't offer any server-side partitioning, clustering, or replication. GemFire was developed to scale the GemStone/S patterns horizontally.

The GemStone transaction docs describe a scheme that would work properly on a single machine, but don't discuss anything about distributed coordination across servers or failure modes. The installation instructions don't discuss setting up a cluster. The marketing docs discuss using thousands of VMs (clients) and scaling the dataset to "hundreds of gigabytes" based on disk storage instead of memory which is not what I would expect from a distributed system. Various benchmarks and user comments refer to using a single server for GemStone.

I will update the post to clarify that we are discussing distributed document databases only. It's easy to do anything you want on a single machine.

I thought GemFire was directly derived from GemStone, via numerous acquisitions. If GemStone has a different transaction model I don't know it.

The point is, no distributed database with a naive two-phase lock is truly transactional.

GemStone/GemFire use a transactional protocol akin to Tuxedo. Open a bunch of locks, write a bunch of updates, release the locks. As per the docs (https://gemfire82.docs.pivotal.io/docs-gemfire/latest/develo...) this does not offer isolation or even atomicity, so it doesn't give you the C in CAP at all.

These are exactly the kind of "transactions" you get when you try to implement everything at the application level rather than the database level. Couchbase transactions (in the article) are the same. And it's not that different from Vitess cross-shard transactions either, which are not isolated (https://vitess.io/docs/reference/features/two-phase-commit/). Tandem SQL used the same scheme as well I believe.

Prior to Spanner, there were no production databases that offered ACID transactions across distributed, disjoint shards.

These are fair critiques, and we expect document databases to evolve towards schema support in the the future. GraphQL is part of this trend.

If only there was some database that let you store flexibly structured documents but keep the data normalized. Perhaps you could even construct views and indexes to accelerate different access patterns.

We haven't seen much difference in practice between frequent aborts and frequent timeouts. Both are better than deadlocks.

I meant transactions issued from a server ("cloud") client to the database, as opposed to a mobile client.

I guess most ORMs are like that. Is the object shared by reference across the entire runtime or do you end up with divergent objects?

I regularly worked from bed when I had an actual office to go to...variety is good.

Firebase was originally designed more as a realtime communication mechanism than an operational database. The idea was that clients would subscribe to different nodes in a data hierarchy to receive realtime notifications from other clients that were publishing to those nodes. Depending on what was in the client view, sometimes you wanted to subscribe to a leaf, sometimes to a subtree, sometimes to everything.

As these things tend to go, when there is a place to store arbitrary data, all kinds of things get shoved into it, so the mixed model in Firestore is a compromise between the original tree-of-nodes data model and a more conventional document data model.

My assumption is the Firestore-to-Spanner mapping creates subcollections as shared tables with foreign keys to the parent documents, but I don't actually know. However, that would match the mandatory 1-to-many-to-1-to-many data layout, and makes more sense than shoving all the dependent data into the document itself or creating multiple millions of SQL tables for millions of documents.

I’m around to answer questions and discuss.

Anybody who has used Couchbase transactions, or sharded Mongo transactions, and can corroborate our analysis?

Did you delete all the extra events that have been created already?

Oh, I thought you were referring to the multi-key consistent transactions, not single key strong consistency mode. I think you are correct.

An AZ is not a datacenter.

You can indeed perform writes in other regions; this is the entire point of Calvin, Spanner, and other modern distributed transaction algorithms: maintaining consistency and maximizing availability in the face of partitions. Your perspective is about a decade out of date.

I think we figured out at least one issue here.

Fauna is a temporal database, not a time series database. The code in the test that updates the score on every post after every read is creating new historical events every time it does that. These have to be read and skipped over during the read query which will continually increase latency proportional to the number of updates that have occurred.

By default, Fauna retains this data and makes it queryable (with transactional guarantees) for 30 days, unlike DynamoDB or Redis. Reducing the retention period would help a bit, but event garbage collection is not immediate so there will still be differences for heavily churned documents. Normally, having a few updates to a document or an index has no noticeable impact but in this case it appears to be swamping the other factors in the latency profile.

It is possible to manually remove the previous events in the update query; doing that should reduce the latency. Nevertheless, Fauna is not a time series database so this is a bit of an anti-pattern.

No it doesn't. The write is not durable in other datacenters when the client acknowledgement is returned. It is still possible to lose data.

Your own docs say that by default “writes are not guaranteed to be durable even if the client receives a success response”.

Please read the blog post: https://fauna.com/blog/comparing-fauna-and-dynamodb-pricing-...

It has to to acquire and release locks, so yes.