al_james

This is very interesting! I agree about the operational complexity of many systems, cough Kubernetes.

For most systems, state storage is the toughest problem, have you considered adding some form of storage layer over the top or would you recommend another solution that allows all the workloads to share state?

Cant even login via the AWS access portal.

Goodfit.io | https://www.goodfit.io/ | Senior Software Engineer (Typescript) | REMOTE UK - Occasional onsite (<once a month) | Full-Time

We work with some of the hottest and highest grown B2B companies in the market, to help them understand their target market and reach potential clients at the right time with the right message. Our platform takes data from hundreds of sources across the internet, and combines it into a real-time and constantly updated view of each opportunity our clients monitor. We process a lot of data to do this, and utilise best-in-class data processing technologies to do this at scale, robustly and efficiently.

What we are after: - Experience in resilient data driven applications in a Serverless environment and data engineering using modern tools - Knowledge of Typescript, Node JS, the AWS ecosystem and SQL (Postgres / Redshift) - Ideally experience in DBT + SQL data pipelines - Comfortable in a small remove first team, looking for ownership and wants to work with an exec team with a track record of growing successful startups. Stack: Typescript, Serverless, React, AWS, Redshift, Postgres, SQL, DBT

Read more: https://www.goodfit.io/jobs/senior-software-developer Ping us to apply: hello <at> goodfit.io

Caim (www.caim.org) | Python Engineer, Front End Engineer, and Engineering Lead | 100% Remote | Volunteer/Unpaid | Part-time

We’re a 501c3 charity whose mission is to help the US become a no-kill nation. Our platform empowers animal rescues to connect with each other, fosters and other rescue advocates in order to move animals from overcrowded, high kill areas to places they're more likely to be adopted. Our organization launched in 2022 and we’re looking for engineers to join the team so we can continue building out our web app. Caim is 100% volunteer run at this time.

We are looking for people who can help out 5+ hours per week.

Python Engineer – must have knowledge of python and any popular web framework (preference for Django) – https://www.caim.org/volunteer/full-stack-engineer

Engineering Lead - must have knowledge of python and any popular web framework (preferably Django) + experience helping more junior engineers get started – https://www.caim.org/volunteer/engineering-lead

Front End Engineer - must have experience with raw CSS, Bootstrap, jQuery, Django and CSS/DOM - https://www.caim.org/volunteer/front-end-engineer

Please apply to volunteer through the links above.

Caim (www.caim.org) | Python Engineers and Engineering Lead | 100% Remote | Volunteer/Unpaid | Flexible Part-time

We’re a 501c3 charity whose mission is to help the US become a no-kill nation. Our platform empowers animal rescues to connect with each other, fosters and other rescue advocates in order to move animals from overcrowded, high kill areas to places they're more likely to be adopted. Our organization launched in 2022 and we’re looking for engineers to join the team so we can continue building out our web app. Caim is 100% volunteer run at this time.

We are looking for volunteers who can help out with anything from a few hours a month upwards.

Python Engineers – must have knowledge of python and any popular web framework (preference for Django) – https://www.caim.org/volunteer/full-stack-engineer

Engineering Lead - must have knowledge of python and any popular web framework (preferably Django) + experience helping more junior engineers get started – https://www.caim.org/volunteer/engineering-lead

Contact: tabitha@caim.org and mention this post on HN.

Goodfit.io | https://www.goodfit.io/ | Senior Software Engineer (Typescript) | REMOTE (UK + EU CET) | Full-Time

We work with some of the hottest and highest grown B2B companies in the market, to help them understand their target market and reach potential clients at the right time with the right message. Our platform takes data from hundreds of sources across the internet, and combines it into a real-time and constantly updated view of each opportunity our clients monitor. We process a lot of data to do this, and utilise best-in-class data processing technologies to do this at scale, robustly and efficiently.

What we are after: - Experience in resilient data driven applications in a Serverless environment and data engineering using modern tools - Knowledge of Typescript, Node JS, the AWS ecosystem and SQL (Postgres / Redshift) - Ideally experience in DBT + SQL data pipelines - Comfortable in a small remove first team, looking for ownership and wants to work with an exec team with a track record of growing successful startups. Stack: Typescript, Serverless, React, AWS, Redshift, Postgres, SQL, DBT

Read more: https://www.goodfit.io/jobs/senior-software-developer Ping me to apply: al <at> goodfit.io

A friend set up https://textareaplease.com/ which I keep coming back to. Useful to have ability to eval JavaScript on the text too.

The mixed column and row store HTAP approach is very similar to how single store works (https://www.singlestore.com/) which offers a mysql compatible interface with a few small differences.

Blueprint.store | London, UK | Senior Node.js Full-stack developer | Remote UK (UK only) | Full time Blueprint is a venture-backed technology start-up backed by some of the best investors in the world - enabling brands and creators to sell inside SMS and WhatsApp. Blueprint is now launched to a global waitlist having been in development for 12 months. We’re looking for an ambitious Senior Software Engineer to join our tight knit development team as we move from launching with our first clients to scaling up our business and platform. Our UI is written in React and the backend built on Serverless using AWS Lambda, Node.js, DynamoDb and MySQL (Aurora). Our roadmap consists of challenging projects, including Natural Language Processing (NLP), payment processing and delivering an exceptional user experience for our clients. You will work directly with our CTO and the company founders. Our engineering team is currently 5 full-stack developers, and we are looking for full-stack developers who can contribute in all areas of the platform. Salary depending on experience: £60k-£90k

https://jobs.blueprint.store/senior-software-engineer-nodejs... . Apply via form or email me at al {at} blueprint.store.

Sorry, at this time we are not working with new recruiters and are unable to offer Visa sponsorship.

Blueprint.store | London, UK | Senior Node.js Full-stack developer | Remote UK (UK only) | Full time Blueprint is a venture-backed technology start-up backed by some of the best investors in the world - enabling brands and creators to sell inside SMS and WhatsApp. Blueprint is now launching to a global waitlist having been in development for 12 months. We’re looking for an ambitious Senior Software Engineer to join our tight knit development team as we move from launching with our first clients to scaling up our business and platform. Our UI is written in React and the backend built on Serverless using AWS Lambda, Node.js, DynamoDb and MySQL (Aurora). Our roadmap consists of challenging projects, including Natural Language Processing (NLP), payment processing and delivering an exceptional user experience for our clients. You will work directly with our CTO and the company founders. Our engineering team is currently 5 full-stack developers, and we are looking for full-stack developers who can contribute in all areas of the platform.

Salary depending on experience: £60k-£90k

https://jobs.blueprint.store/senior-software-engineer-nodejs... . Apply via form or email me at al {at} blueprint.store.

Sorry, at this time we are not working with recruiters and are unable to offer Visa sponsorship.

Yes confirmed its because netlify's origin servers use Google Load Balancers. According to this blog they claim "We can easily move the entire brains of our service between Google, Amazon, and Rackspace in around 10 minutes with no service interruptions.". Lets see if thats true. https://www.netlify.com/blog/2018/05/14/how-netlify-migrated...

Netlify is also failing for us, and reporting bad TLS certs. Not sure if they use GCP https://www.netlifystatus.com/

Looks powerful. You mention the CRDT is a WAL, won't it grow indefinitely?

Blueprint.store | London, UK | Senior Node.js Full-stack developer | Remote UK (UK only) | Full time Blueprint is a venture-backed technology start-up backed by some of the best investors in the world - enabling brands and creators to sell inside SMS and WhatsApp. Blueprint is now launching to a global waitlist having been in development for 12 months. We’re looking for an ambitious Senior Software Engineer to join our tight knit development team as we move from launching with our first clients to scaling up our business and platform. Our UI is written in React and the backend built on Serverless using AWS Lambda, Node.js, DynamoDb and MySQL (Aurora). Our roadmap consists of challenging projects, including Natural Language Processing (NLP), payment processing and delivering an exceptional user experience for our clients.

You will work directly with our CTO and the company founders. Our engineering team is currently 5 full-stack developers, and we are looking for full-stack developers who can contribute in all areas of the platform.

Salary depending on experience: £60k-£90k

https://apply.workable.com/blueprint-store/j/EE934DE909/ . Apply via form or email me at al {at} blueprint.store.

Sorry, at this time we are not working with recruiters and are unable to offer Visa sponsorship.

Blueprint.store | London, UK | Senior Node.js Full-stack developer | Office or Remote (UK only) | Full time Blueprint is a venture-backed technology start-up backed by some of the best investors in the world - enabling brands and creators to sell inside SMS and WhatsApp. Blueprint is now launching to a global waitlist having been in development for 12 months. We’re looking for an ambitious Senior Software Engineer to join our tight knit development team as we move from launching with our first clients to scaling up our business and platform.

Our UI is written in React and the backend built on Serverless using AWS Lambda, Node.js, DynamoDb and MySQL (Aurora). Our roadmap consists of challenging projects, including Natural Language Processing (NLP), payment processing and delivering an exceptional user experience for our clients.

You will work directly with our CTO and the company founders. Our engineering team is currently 5 full-stack developers, and we are looking for full-stack developers who can contribute in all areas of the platform.

Salary depending on experience: £60k-£90k

https://apply.workable.com/blueprint-store/j/EE934DE909/ . Apply via form or email me at al {at} blueprint.store.

Sorry, at this time we are not working with recruiters and are unable to offer Visa sponsorship.

Blueprint.store | London, UK | Senior Node.js Full-stack developer | Onsite or Partly-Remote (UK only) | Full time Blueprint is a venture-backed technology start-up backed by some of the best investors in the world - enabling brands and creators to sell inside SMS and WhatsApp. Blueprint is now launching to a global waitlist having been in development for 12 months.

We’re looking for an ambitious Senior Software Engineer to join our tight knit development team as we move from launching with our first clients to scaling up our business and platform.

Our UI is written in React and the backend built on Serverless using AWS Lambda, Node.js, DynamoDb and MySQL (Aurora). Our roadmap consists of challenging projects, including Natural Language Processing (NLP), payment processing and delivering an exceptional user experience for our clients.

You will work directly with our CTO and the company founders. Our engineering team is currently 5 full-stack developers, and we are looking for full-stack developers who can contribute in all areas of the platform.

Salary depending on experience: £60k-£90k

https://apply.workable.com/blueprint-store/j/EE934DE909/ . Apply via form or email me at al {at} blueprint.store.

Sorry, at this time we are not working with recruiters and are unable to offer Visa sponsorship.

Blueprint.store | London, UK | Senior Node.js Full-stack developer | Onsite or Remote (UK only) | Full time

Blueprint is a venture-backed technology start-up backed by some of the best investors in the world - enabling brands and creators to sell inside SMS and WhatsApp. Blueprint is now launching to a global waitlist having been in development for 12 months.

We’re looking for an ambitious Senior Software Engineer to join our tight knit development team as we move from launching with our first clients to scaling up our business and platform.

Our UI is written in React and the backend built on Serverless using AWS Lambda, Node.js, DynamoDb and MySQL (Aurora). Our roadmap consists of challenging projects, including Natural Language Processing (NLP), payment processing and delivering an exceptional user experience for our clients.

You will work directly with our CTO and the company founders. Our engineering team is currently 5 full-stack developers, and we are looking for full-stack developers who can contribute in all areas of the platform.

Salary depending on experience: £60k-£90k

https://apply.workable.com/blueprint-store/j/EE934DE909/ . Apply via form or email me at al {at} blueprint.store.

Sorry, at this time we are not working with recruiters and are unable to offer Visa sponsorship.

A lot of companies in London are waiting for the 100 cases in London mark to trigger the WFH fallback. This will be breached tomorrow so expect London tech companies to be totally remote by end of week.

Ometria.com | London UK AND Southampton UK | FULL-TIME ONSITE | Several roles: Backend Python, Frontend Javascript, DevOps

Ometria’s mission is to help retailers create marketing experiences their customers will love. We understand the challenges that retailers face, and we offer them a very innovative solution that provides insights on their customers, and tools to reach them more effectively across numerous channels.

Backed by top VC funds and successful entrepreneurs, and working alongside over a hundred of the fastest growing retailers, we are now looking for a more developers to join our small but growing engineering team.

We are hiring for:

- Senior Backend Python developers

- Senior Frontend Javascript developers ( React)

- DevOps - AWS - configured with Terraform, using Docker & Kubernetes

https://www.ometria.com/careers/ (Not all jobs are on that page yet, feel free to contact me personally at “al <at> ometria.com”)

So the big question is... Can we point this at aws and get Google to manage our aws kubernetes Installs...

Given the massive shambles eks turned out to be, that would be great.

Ometria.com | London UK | FULL-TIME ONSITE | Several roles: Backend Python, Frontend Javascript, Data science, QA, VP engineering

Ometria's mission is to help retailers communicate with their customers in a more personalised way.

We have a modern microservices based system and we use modern tools like docker and Kubernetes on AWS. We process a lot of data and store it in Postgres and Redshift. Increasingly we are using serverless technologies like AWS Lambda. We also are experimenting with Go for some services, so if thats an interest thats a plus.

Backed by top VC funds and successful entrepreneurs, and working alongside over a hundred of the fastest growing retailers, we are now looking for a more developers to join our small but growing engineering team.

We are hiring for:

- Backend python developers

- Frontend javascript developers (Ampersand JS, but considering moving to React)

- Data Scientist / ML engineer (Python stack)

- VP engineering

- QA engineer

https://www.ometria.com/careers/ (Not all jobs are on that page yet, feel free to contact me personally at "al <at> ometria.com")

Thats very very interesting. Many thanks for sharing.

So its still a serialize/deserialize cycle, but the access libs built on top of the persistent memory look interesting.

What I can't work out from both the article and the comments here: from an application point of view, do I use this like I use memory, or do I use it like I use a disk?

No matter how fast a disk is, using it means either some expensive serialization/deserialization step (and also the associated memory access to create the 'working' object that my logic actually works on) or writing my algorithms to forego in memory objects (and the associated features offered by my programming language, e.g. classes / objects or whatever) and working from the raw byte values.

What I really want, and would be a game changer as to how we use things, would be that my programming languages heap can be made persistent (or at least a part of it). In this case instead of:

  var mything = new Thing();
  load_thing_from_disk(mything);

  persistent var mything = new Thing();

However I cant help thinking that some way to harness persistent fast memory without needed some complex disk->logic mapping would be a game changer.

Edited: spelling and wording

Yup. We run Postgres on i3 instances on their native SSDs and its way faster for OLAP use cases. Check out avien.io for a hosted RDS style solution that does this.

Ometria.com | London UK | FULL-TIME ONSITE | Several roles: Backend Python, Frontend Javascript, Data science, QA, Machine learning

Ometria's mission is to help retailers create marketing experiences their customers will love. We understand the challenges that retailers face, and we offer them a very innovative solution that provides insights on their customers, and tools to reach them more effectively across numerous channels.

Backed by top VC funds and successful entrepreneurs, and working alongside over a hundred of the fastest growing retailers, we are now looking for a more developers to join our small but growing engineering team.

We are hiring for:

- Backend python developers

- Frontend javascript developers (Ampersand JS, but considering moving to React)

- Data Scientist (Python stack)

- Machine Learning engineers

- Engineering manager

- VP engineering

- QA engineer

https://www.ometria.com/careers/ (Not all jobs are on that page yet, feel free to contact me personally at "al <at> ometria.com")

Yeah, thats a good point. Redshift does not have the same level of 'probabilistic counting' functions, that can be used from rollups. Redshift does have HLL (SELECT APPROXIMATE COUNT(*)) however that can only be applied when scanning the full data, I am not sure its possible to store a HLL object in a rollup and later aggregate them.

Thanks Ozgun. Thats a great video in 3.

A great article, and I am a big fan of algolia, Citus and Redshift. However this article ends up making an odd apples to oranges comparison.

They state that "However, achieving sub-second aggregation performances on very large datasets is prohibitively expensive with RedShift", this suggests that they want to do sub-second aggregations across raw event data. However, later in the article, the solution they build is to use rollup tables for sub-second responses.

You can also do rollup tables in Redshift, and I can assure you (if you enable the fast query acceleration option) you can get sub-second queries from the rolled up lower-cardinality tables. If you want even better response times, you can store the rollups in plain old Postgres and use something like dblink or postgres_fdw to perform the periodic aggregations on Redshift and insert into the local rollup tables (see [1]). In this model the solution ends up being very similar to their solution with Citus.... and I would predict that this is cheaper than Citus Cloud as Redshift really is a great price point for a hosted system.

So the question of performing sub-second aggregations across the raw data remains unanswered... however that really is the ideal end game as you can then offer way more flexibility in terms of filtering than any rollup based solution.

Right now, research suggests Clickhouse, Redshift or BigQuery are probably the fastest solutions for that. Not sure about Druid, I dont know it. GPU databasees appear to the be the future of this. I would be interested to see benchmarks of Citus under this use case. I should imagine that Citus is also way better if you have something like a mixed OLAP and OLTP workload (e.g. you need the analytics and the row data to match exactly at all times).

Aside: It would be great to see Citus benchmarked against the 1.1 billion taxi rides benchmark by Mark Litwintschik. Any chance of that?

[1] https://aws.amazon.com/blogs/big-data/join-amazon-redshift-a... [2] http://tech.marksblogg.com/benchmarks.html

Ometria.com | London UK | FULL-TIME ONSITE | Several roles: Backend Python, Frontend Javascript, QA, Machine learning

Ometria's mission is to help retailers create marketing experiences their customers will love. We understand the challenges that retailers face, and we offer them a very innovative solution that provides insights on their customers, and tools to reach them more effectively across numerous channels.

Backed by top VC funds and successful entrepreneurs, and working alongside over a hundred of the fastest growing retailers, we are now looking for a more developers to join our small but growing engineering team.

We are hiring for:

- Backend python developers

- Frontend javascript developers (Ampersand JS, but considering moving to React)

- Machine Learning engineers

- Engineering manager

- VP engineering

- QA engineer

https://www.ometria.com/careers/ (Not all jobs are on that page yet, feel free to contact me personally at "al <at> ometria.com")

We use pg_repack and hit that exact problem (left over triggers etc). In the end we resorted to creating and dropping the pg_repack extension after each run.

(edit: Missed an "and")