alexdean

Very cool. Snowplow CEO here - if you need any help on integrating with Snowplow just ask in our forums: https://discourse.snowplowanalytics.com/

Snowplow CEO here. We haven't used Jitsu before but are very familiar with Segment. It looks like Jitsu sits in the Segment product family, along with Rudderstack: basically a Customer Data Platform bundle of simple JSON event tracking, Fivetran-style transactional/SaaS data ingest, and then relaying of data out to various SaaS endpoints plus cloud DWs.

Snowplow started at the same time as Segment (2012) but has evolved along a separate tech tree. Micro-service architecture, cloud native, using Kinesis or Cloud Pub-Sub as the data transit, enrichment framework plus a Confluent-style schema registry supporting very rich and versioned JSON Schema-based event payloads. We are built by and for data platform teams; our open-source behavioral data engine doesn't have a UI (our commercial Behavioral Data Platform does). Hosted trial here https://try.snowplowanalytics.com/

Definitely room for both product families in the market! I'm sure Jitsu will do great.

Back in the early years of Snowplow, we adopted a similar approach for our support of 3rd party webhooks[1] as an event stream source supported natively by Snowplow (https://github.com/snowplow/snowplow). (The important background here is that all events and entities flowing through Snowplow are described using versioned JSON Schema).

We built a kind of Maven Central for 3rd party webhooks' schemas[2], and got to work adding schemas for various popular webhooks, e.g. Sendgrid and Pingdom. But, we never met a single SaaS vendor who was interested in 'adopting' the JSON Schemas for their webhook, let alone publishing versioned JSON Schemas for their whole API. This meant that at Snowplow we have stayed on the hook for keeping these vendors' webhook schema definitions up to date in our registry.

It's a real tragedy of the commons - oceans of developer time wasted on tedious low-leverage work (updating API client code) so that each SaaS vendor can 'move fast and break things'. The ultimate irony is that if these vendors were to adopt, publish and respect versioned schema definitions internally, using something like OpenAPI, they would see huge productivity gains (think enhanced CI/CD testing, auto-gen of client code etc).

1. https://docs.snowplowanalytics.com/docs/getting-started-on-s... 2. https://github.com/snowplow/iglu-central/tree/master/schemas

I just wanted to jump in and say the same thing. I had the hardback of the Cuckoo's Egg as a teenager in the 90s. Huge inspiration to me and I have worked in and around tech ever since. Thank you.

He may think he doesn't have an 'HR department', but whoever holds the pen on his holiday policy, equipment policy, training & development policy, maternity & family friendly rights policy, expenses policy - well, there's his HR department.

Hey dylz - not yet. Historically Snowplow has been engineered in a way that is idiomatic to each host cloud - in other words, it runs using the AWS / GCP services that a data platform team would use if they were building a behavioral data pipeline from scratch on that cloud.

This said, we are steadily working to refactor all our components to be more generic, and already we run a lot of the Snowplow components on k8s for our customers.

Snowplow has had a strong following around HNers ever since we launched back in 2012. The truth is though that Snowplow open-source is still a complex project to stand up, and a lot of technology teams still aren't aware of the power of having all their own behavioral data (events from webapps, mobile etc) in their own data warehouse or lake, and available in real-time streams (Kinesis and GC Pub/Sub).

With this background, I am delighted to be able to Show HN our new Try Snowplow experience. Under lockdown last year, the team decided to go back to basics and build a new version of Snowplow called Try Snowplow; that version is now in GA.

Try Snowplow is a small version of the Snowplow technology that can be setup quickly and easily - normally under 15 mins. Like all versions of Snowplow, it runs in your own cloud environment.

The landing page is here: https://snowplowanalytics.com/get-started-try-snowplow/

And if you want to learn more about Try Snowplow before giving it a spin, there's a video here: https://www.youtube.com/watch?v=Aw5hdIjwVhY&ab_channel=Snowp...

If you are still puzzled why you want your own behavioral data in your own warehouse, check out our library of use cases here: https://snowplowanalytics.com/use-cases/

Any questions just shout! Happy to answer anything Snowplow-related, and thanks again for Hacker News' support of Snowplow over the years.

Agree, building this in 2021 is not a good use of data engineering time.

As well as the SaaS packages like Amplitude and Mixpanel, you also have great open-source tools and platforms for mobile and product analytics like PostHog (https://posthog.com/), Countly (https://count.ly/) and Snowplow (https://snowplowanalytics.com/).

Disclosure: Snowplow co-founder.

I chortled when I saw the health warning against railway-oriented programming. Reading that post deeply influenced (and continues to influence) the architecture of Snowplow (https://github.com/snowplow/snowplow) and I titled a chapter of Event Streams in Action 'Railway-Oriented Processing' (https://www.manning.com/books/event-streams-in-action). Thanks Scott.

It's not that simple, for example Waymo has external investors alongside Alphabet, https://blog.waymo.com/2020/03/waymo-raises-first-external-i...

Yes, Snowplow uses JSON Schema extensively for event definition and validation (https://github.com/snowplow/snowplow/).

Do any of these companies offer webhooks, or are they all oriented at human users reading emails?

Great question - we invented a system for this at Snowplow, called SchemaVer:

http://snowplowanalytics.com/blog/2014/05/13/introducing-sch...

SemVer doesn't work for data - for one thing, there is no concept of a "bug" in your data (so patches are meaningless).

We have hundreds of companies actively using SchemaVer via the Snowplow (https://github.com/snowplow/snowplow/) and Iglu (https://github.com/snowplow/iglu/) projects.

I'm not sure that's true of all the client libraries - I'm pretty sure that the Java library is used heavily internally within Amazon to leverage AWS services. Just look at the release history: https://github.com/aws/aws-sdk-java/releases

But you're right, maintaining even a blessed third-party library like boto is a massive undertaking: 437 contributors, 431 open issues, 192 open PRs. Competing against CloudFormation using anything other than the Java client library is crazy IMO.

This is a cool idea for a project! The closest equivalent I'm aware of is Qubit OpenTag (https://github.com/QubitProducts/OpenTag).

I'm interested in adding SnowPlow support to this (https://github.com/snowplow/snowplow) - our tracking API is very similar to Google Analytics's.

We've just gone through the quite involved exercise of mapping SnowPlow to Google Tag Manager (http://snowplowanalytics.com/blog/2012/11/16/integrating-sno...) so I was a bit surprised in the code to see this mapping for GA events:

    track : function (event, properties) {
                window._gaq.push(['_trackEvent', 'All', event]);
            }

I think you might be making the assumption that events consist of an event name plus arbitrary JSON envelope. This is a very MixPanelish view of the world - it doesn't really translate to Google Analytics, Piwik, SnowPlow, Omniture...