HN user
DevX in Rust for backend development is quite good nowadays. Things are WAY easier than it was a couple of years ago.
I migrated a couple of projects from Java, TS and Go to Rust and honesly, I couldn't be happier.
(But in case he’s listening, please add Rusts discriminated enums, it really hurts to move from a language with them to a language without them.)
I'm not a programming language expert, but my understanding is that discrimitated enums is jut a way to support sum types. Dart now has the sealed class modifier [1], that fills this role.
Same here. But have you tried duckdb? You can do sql in the pandas dfs and it is fast af.
Thanks for sharing this.
Did you notice any improvements with your condition (type 1 diabetes) once you started to do intermittent fasting?
This looks really nice. I was reading about this and realized it had similar ideas to this [1] Phd thesis from Jon Gjengset. I checked his twitter [2] and it was based on his work indeed.
Great that someone is productionalizing this!
Btw, is Jon involved in ReadySet?
[1] Partial State in Dataflow-Based Materialized Views - https://github.com/mit-pdos/noria
Depending on the cost per database, a service like this could allow one to build SAAS apps with Single Tenant model [1] in a really simple and elegant way.
I always liked this model since it gives the best isolation and not having to code with multi-tenant in mind really simplifies development and security. But maintaing an infra for this is hard and costly.
[1] https://docs.microsoft.com/en-us/azure/architecture/isv/appl...
Dan Luu impact at Wave.com is noticable already. Since he joined in March, Wave is pushing one great blog post per week (in contrast to 1 in 2021, 0 in 2020 and 3 in 2019).
I tried creating this culture of blogging but failed, would love to hear strategies that worked to achieve this.
wow.... this is amazing!
I dindn't get this:
`OrioleDB implements default 64-bit transaction identifiers, thus eliminating the well-known and painful wraparound problem`
I thought that PG's 32-bit transaction identifier and wraparound issue was due to the current MVCC implementation that keeps old versions of rows in page tables, so there is the need to vacuum to clean up the dead rows. So my assumption was that using another approach for MVCC, like the UNDO log, this problem wouldn't exist (since page tables would only have the latest version).
What am I missing here?
This is really nice! Congrats!
I once started building as a side project something similar but focused on querying cloud resources (like S3 buckets, ec2s, etc... discovering the biggest file from a bucket was trivial with this). I abandoned the project but someone else built a startup on the same concept - even the name was the same: cloudquery.
I built it using the multicorn [1] postgres extension and it is deligthful of how easy it to get something simple running.
That is pretty landing page!
Just read that the meaning of oku [1] is 1) private, intimate, and deep; 2) exalted and sacred; and, 3) profound and recondite
anyway, your brazilian users will find this funny since oku has the same sound of "o cu" that literally means "the butt hole".
[1] https://en.wikipedia.org/wiki/Oku_(theory)#:~:text=3%20Bound....
They mention this in the article. But to sum up, each connection in PG is handled by its own OS process. Postgres behind the scenes is composed by multiple single-threaded applications.
This comes with the advantes for Pg developers (and us!) that they don't need to deal with tons of data races issues, but the trade off is that memory wise, a process takes way more memory than a thread.
Comparing Fauna with DynamoDB and Redis gives the wrong impression. Trade-offs are totally different.
However, the product in question (Upstash), looks very interesting: serverless storage with a pay as you go model with a maximum price cap is an excelent idea.
I would love to have more options in this space, especially in GCP to use together with Cloud Run.
Dremio is a really nice product built using arrow.
Apache Drill is an interesting project, from all the MPP engines that appeared a few years ago, it was the most similar one to BigQuery (the first public version) and the most flexible.
However, the competion was fierce and each Big Data vendor (MapR, Cloudera and HortonWorks) was pushing its own solution: Drill, Impala and Hive on Tez. Competion is always a good thing, but it fragmented the user base too much so no clear winner emerged.
At the same time, Spark SQL got sufficiently better to replace these tools in most use cases and Presto (from Facebook) got the traction and the user base that none of these projects had by being vendor agnostic (and its adoption by AWS in Athena and EMR also helped boost its popularity).
I don't believe grafana will help you here.
If you have mongodb enterprise you can use Mongo bi connector and connect it with any bi tool. I recommend metabase since it is open source and very good for slice and dice and reports.
The mongodb bi connector is a postgres with foreign data wrapper to mongodb. So even if you don't have a mongodb subscription, you may be able to pull this off using the OS version https://github.com/EnterpriseDB/mongo_fdw
In Germany, Takeaway [1] bought most of the competion and the small players quit (like Deliveroo) since they couldnt compete... well looks like this is actually working:
https://corporate.takeaway.com/investors/results-and-reports
Prices were raised and I doubt restaurants and delivery guys are making more money in this scenario than they used to.
This is an SDK not a full client for Kinesis.
This is the only official implementation from AWS: https://github.com/awslabs/amazon-kinesis-client
The client handles horizontal scalling, checkpoiting, shards split and shards merges. Using just the SDK, you have to build this yourself (unless you are using Kinesis for use cases that dont need it to be done correctly).
This is the doc for developing consumers using the SDK https://docs.aws.amazon.com/streams/latest/dev/developing-co...
And in the second paragraph of this documentation: "These examples discuss the Kinesis Data Streams API and use the AWS SDK for Java to get data from a stream. However, for most use cases, you should prefer using the Kinesis Client Library (KCL) . For more information, see Developing KCL 1.x Consumers."
Because the official client from AWS is written in Java.
It is possible to write clients in any language, however, it is not that simple especially when you need to handle logic of scalling out or in your kinesis stream (that will split or join shards) and when you have multiple consumers in the same consumer group (you will need a distributed locking mechanism and a logic to steal locks if one consumer dies).
So it is not trivial.
I believe you will enjoy this:
https://medium.com/revolut/recording-more-events-but-where-w...
Not exactly the same architecture you are proposing, and quite complex tbh, but it is working for them.
I don't know about aurora serverless.
But aws offers a proxy exactly for this purpose.
Aurora is indeed amazing. However, my experience with it regarding costs is that main cost you have running it is not compute, but IO ($0.2 per million requets [1]).
This cost is kind of hidden since to estimate this in the early stages of a project is an art. In one project on my team the IO cost is about 8x more than cost of instances. But imo it is still worth and I never actually calculated how much we would pay if we were running on RDS + provisioned IOPS.
Not a blog or a Twitter, but Jon Gjengset channel on YouTube has awesome live (or records of) coding videos on Rust. The topics are not for beginners, but even thought I'm starting on Rust as well, I've found his videos an excellent resource.
Can you please elaborate a little bit more of how you replaced it by postgres? Because it is strange that a single box of postgres in way less powerful instance type would perform the same as your cassandra cluster. This kind of seems that the first solution was way over engineered or was built for different requirements.
Brazil's new administration already approved more than 160 pesticides in 6 months. So they are catching up quickly.
https://g1.globo.com/economia/agronegocios/noticia/2019/05/2...
OT: I visited a Chinese company last year in Beijing and what impressed me most was the richness of open source solutions there. To mention a few tools they were using that I haven't heard before: jstorm[1] (storm inspired stream processor) , dubbo [2] (grpc like for Java), codis [3] (redis clustering/sharding solution), canal [4] (mysql to Kafka forwarder).
Most of the code base of this company was java, go and PHP. And the comments were on Chinese and they were doing a task force to translate to English.
[1] https://github.com/alibaba/jstorm [2] https://dubbo.incubator.apache.org/en-us/ [3] https://github.com/CodisLabs/codis [4] https://github.com/alibaba/canal
Same here. And Salvatore's blog post about hll is pretty nice:
I think it is only available through Safari
yes it is, as long as you can represent it in SQL.
for instance, for a query like:
select day_of_year, year, count(*) from orders where year >= 2016 group by day_of_year, year
you can create a line visualization that will display the different years as different lines in about 2 or 3 clicks.
AFAIK, there is no book for Metabase. But the documentation and the project is really straightforward.
I think it depends of the organization. Most companies still use something similar to a DW/DL that speaks SQL, so Metabase/Redash still cover this.
Tableau is awesome, but I believe that most of its features are not used soo often. A simpler tool can provide almost the same or more value than Tableau in some cases. Well, a tool that you don't need to pay expensive licenses for each user makes it more democratic, everyone on the organization can visualize and create new analysis.
What do you mean by reports?
If by reports you mean a daily email sent with a table and some graphs attached them both suck. The only tool that I know that excels on this is Reporting Services from Microsoft.
But they are really OK for dashboards and visualizations. Metabase has a pulse feature where you can schedule to send a visualization to email or slack, but not something very fancy.