ameyamk

Most writing today focuses on how individuals use AI tools. I wrote this post to share an early attempt at a playbook for building truly AI-native engineering teams

Larridin | AI Native Software Engineer (All Levels) — Bay Area, In-Office

Larridin is the measurement layer for AI in the enterprise. We’re building AI-native systems that correlate AI behavior to business results, expose absorption bottlenecks, and enable enterprises to invest with conviction. Backed my top tier investors such as a16z, Google and Bloomberg and with early traction!

We want high-agency software builders (3–10 yrs) who think AI-first, move fast, and want ownership of large parts of product and business. This is an in-office role in the Bay Area for engineers who want to own outcomes, not just tickets.

DM' me on Linkedin to apply: https://www.linkedin.com/in/ameyakanitkar/ This is also primarily backend focussed role.

Looks good. Curious, how is auth handled? Lot of docs have permissions etc. Can you clarify how this is handled in both indexing side and searching side of things?

Larridin | Backend/ Fullstack Engineering Roles + AI/ML Roles | Bay Area (in person preferred)

Join our team and help redefine the future of workplace productivity with AI. We are 16z backed company, early stage!

https://www.larridin.com/careers

Larridin | Backend/ Fullstack Engineering Roles + AI/ML Roles | Bay Area (in person preferred)

Join our team and help redefine the future of workplace productivity with AI. We are 16z backed company, early stage!

https://www.larridin.com/careers

This was a brilliant strategy. It allowed Google time to build Chrome at the same time slowing down : (1) explorer (2) Got decent data on CAC for browsers and what channels work and what don't.

Its interesting to see new era of browser wars playing out with Chrome firmly in the lead, but Microsoft attacking with Edge where ChatGPT on bing only available on edge browser and no where else. Would be interesting to see how Google would respond.

Great work! Is there a place I can checkout for good quality non spammy "business simulation" games?

Good move! Would help consolidate Hadoop ecosystem. MapR would find it hard to compete now.

Both these companies are being challenged with a new crop of Databricks and the confluents of the world.

Although implementation may be ugly and/ or shocking - the concept in itself has interesting roots.

Although social media in itself is vital - we are seeing more usage of it correlates with negative behavior pattern (as is excessive tv watching).

Should govt intervene?

They do intervene for substance addiction...

You can use this for any execution. eg. here is a job type to trigger shell command such as ' echo "hello" ' http://azkaban.github.io/azkaban/docs/latest/#command-type

Note execution environment for such jobs is Azkaban executor server itself, so you have to take care of resource management (eg. one job taking all RAM on the machine will affect other jobs running on the same machine)

At LinkedIn we heavily use Azkaban for this. (Open source: https://azkaban.github.io/) Azkaban API can be used to launch offline computation jobs as necessary - Azkaban ensures monitoring, SLA alerting, failed restarts and other dependency management etc.

$4 billion food delivery business! Thats pretty neat. Uber is showing signs that they can succeed in general logistic space also besides just taxi business.

Why airbnb needs so much cash? They are not exactly subsidizing either side of the market?

This is pure monopoly play - I hope Lyft continues to compete and do not merge ... I hate to be in world where no taxi service exists and I am at the mercy of random surge pricing...

Pager duty is one of the must have 3rd party apps in pretty much every user facing system I can think of!

The the UX side it can improve - but it's rock solid in what it does! Well done!

I have been looking for this for a long time!

+1 Twitter should build this!

I wonder if there is investment vehicle I can short to make some profit if these loans do actually go bad?

Question remains - if this is winner takes all market. If it is - only one company will survive. As a consumer most important thing for me is to get cab as quickly and cheaply as possible maintaining the basic standards of cleanliness, and politeness from service providers. If quality of service is more or less the same - then I'd argue that the platform generating more demand and supply in the market will prevail and take on outsize market share (80%+). if this is true - then only one will prevail.

why are so many rich people moving to Australia?

This looks great. Some open questions -

1. How does it scale horizontally? if I am consuming from say topic with 100 partitions - do I have to deploy this on library on multiple nodes? or expectation is we need to embed this inside spark streaming/ storm/ samza/ any other real time processing framework?

2. Is RocksDB instance local to your consumer/ thread/ node? or can state be stored across nodes? What happens when joins are needed across the partitions?

I have been doing real time processing for a long time now - and this does capture most of the typical things you end up doing with kafka with spark/ storm etc- so this does look a significant step forward.

I am just trying to understand where it exactly fits.

$2 billion is not staggering amount. Given it has over 300 Million active users - thats just $6~$7 for user acquisition.

If you want to build twitter today - its definitely going to cost more. It has built a brand - and hundreds of millions of loyal fans.

Question is - is it worth the valuation thats put on the company. Given now they are cash flow positive - that they have lost $2bn in the past is immaterial.

Why does YC needs to grow into 2000 companies/ year program? I feel they should tackle it like a University. Stanford does not try to grow 10x from 20k students to 200k students.

Part of the appeal is exclusivity (besides of course its hard to scale).

I'd rather see YC at 200-300 companies a year - and ensuring these companies continue to grow/ succeed after the batch is over rather than churning out 1000 more.

It would be nice to list tear downs in the order in which they were reviewed - so I can read new ones quickly when I visit. Looks like current list is alphabetically sorted.

How are the requirements for salary are taken care of? H1b employee must be paid market rate salary (not in stocks but actual cash)

Its actually very simple with some clear winners emerging now:

Log / Stream Processing - Kafka \n Scalable Storage - HDFS \n Data Processing - Spark, Map reduce (in that order) \n Historical Analytics - Hive/ Spark SQL \n Real Time Processing - Spark Streaming, Storm \n NoSQL - Cassandra/ HBase \n NoSQL (In memory) - Redis \n Search - Elastic Search \n

Some more honorable mentions: kibana on elastic search - for analytics visualization \n druid - for analytics \n

Above are the basics - if you add them you will have 90% of the standard stack for big data.

I just wonder instead of going back and forth and create more controversy - why don't they just show which tests are working right now - and which don't. Demonstrating with data is the best defense they can put forth.

its ok - if only small portion of tests are valid at this point but if they show progress with data - this validates they are on the right track.

I wonder if companies like door dash/ instacart are operating at negative margins?

Uber tinkering with price too much has not gone down well with uber drivers in bay area either.

Cool. So in a way its Immutable read only HBase (with guaranteed data locality no memstore overhead and compactions overhead). Cool. Nice solution. I wonder this can be patched back to HBase - as a "read only mode" ?

Hi Varun,

Can you also elaborate, how you read HFiles and serve it out from Terrapin servers? Are you using similar functionality as HBase? (With block cache like design if yes how do you keep both in sync).

Your blog is missing this interesting detail.

Bulk Uploads into KV stores are slow - so Terrapin allows KV access over immutable HDFS files.

Very typical use case for recommendation systems etc. We face similar problems with latencies on HBase (At Groupon).

So this solution seems interesting. Would be good to have comparison of other solutions Pinterest tried before building this. eg. loading data into Cassandra instead of HBase etc.

In nutshell - very specific use case - but the one which comes across very often