hendiatris

I run

sudo shutdown +15 (or other amount of minutes)

when I need a compute instance and don’t want to forget to turn it off. It’s a simple trick that will save you in some cases.

And Brasil, Portugal and other portuguese-speaking places

They probably threw a fat CIDR block in their IP blacklist to fight off a spam campaign, and your IP was caught in the dragnet. This is how the big companies do it. They’ll evaluate for risk of false positives and as long as that stays below a threshold, they proceed.

In the lower level arrow/parquet libraries you can control the row groups, and even the data pages (although it’s a lot more work). I have used this heavily with the arrow-rs crate to drastically improve (like 10x) how quickly data could be queried from files. Some row groups will have just a few rows, others will have thousands, but being able to bypass searching in many row groups makes the skew irrelevant.

Just beware that one issue you can have is the limit of row groups per file (2^15).

You may be able to get close with sufficiently small row groups, but you will have to do some tests. You can do this in a few hours of work, by taking some sensor data, sorting it by the identifier and then writing it to parquet with one row group per sensor. You can do this with the ParquetWriter class in PyArrow, or something else that allows you fine grained control of how the file is written. I just checked and saw that you can have around 7 million row groups per file, so you should be fine.

Then spin up duckdb and do some performance tests. I’m not sure this will work, there is some overheard with reading parquet, which is why it is discouraged to have small files and row groups.

I will write something up when the dust settles, I’m still testing things out. It’s a project where the data is fairly standardized but there is about a petabyte to deal with, so I think it makes sense to make investments in efficiency at the lower level rather than through tons of resources at it. That has meant a custom parser for the input data written in Rust, lots of analysis of the statistics of the data, etc. It has been a different approach to data engineering and one that I hope we see more of.

Regarding reading materials, I found this DuckDB post to be especially helpful in realizing how parquet could be better leveraged for efficiency: https://duckdb.org/2024/03/26/42-parquet-a-zip-bomb-for-the-...

This is a huge challenge with Iceberg. I have found that there is substantial bang for your buck in tuning how parquet files are written, particularly in terms of row group size and column-level bloom filters. In addition to that, I make heavy use of the encoding options (dictionary/RLE) while denormalizing data into as few files as possible. This has allowed me to rely on DuckDB for querying terabytes of data at low cost and acceptable performance.

What we are lacking now is tooling that gives you insight into how you should configure Iceberg. Does something like this exist? I have been looking for something that would show me the query plan that is developed from Iceberg metadata, but didn’t find anything. It would go a long way to showing where the bottleneck is for queries.

Look at the website for that polling company. It is bizarre. None of the people on the people page have the company on their LinkedIn pages. Seems to be astroturf.

Edit: look at the photos of the people… AI generated perhaps?

Sqitch is an incredibly under appreciated tool. It doesn’t have a business pushing it like flyway and liquibase, so it isn’t as widely known, but I vastly prefer it to comparable migration tools.

How quickly does the append-only chain grow? What are the storage needs for it?

There are some inaccuracies in this article. For example, the villa on Capri was built by Tiberius, not Augustus. https://en.wikipedia.org/wiki/Villa_Jovis

If you’re going to work with weather data use a columnar database, like BigQuery. If you set things up right your performance will generally be a few seconds for aggregation queries. I setup a data platform like this at my previous company and we were able to vastly outperform our competitors and at a much lower cost.

The great thing about this data is it is generally append only, unless errors are found in earlier data sets. But it’s something that usually only happens once a year if at all.

Prepare to pay around $20k per (https://www.pamono.es/mod-sillon-alta-de-oscar-niemeyer-para...)

Here’s video: https://youtu.be/fAADWfJO2qM

Done! I hope they can act fast.

I read this and think “ah, well there are many more non-rich people so the density requirements make this sort of thing complicated” and then I look out the window over the European city I live in where it has been achieved. Then I think about the town an hour away with a vibrant community and pleasant human-scale environment. What makes it all possible? Good transit. Clean buses. The US simply doesn’t have that, and probably not even the political will to make it happen. What a shame.

The thing that matters is oxygen exposure more than anything. You can’t (trivially) reverse the effects of oxygen. This is why Coravin exists: replace the air (with oxygen) with an inert gas. You can do the same for cheap by putting the wine in a container (preferably gas) that reduces its exposure to oxygen, ideally entirely. There are companies selling other tools that do this, like an inflatable device that goes into a bottle.

Unfortunately it’s not a relational database, so no constraints or any of the other features.

Please provide more detail. I find the USPS to be one of the best things about this country. Every morning I get an email with photos of the mail I will receive, and a listing of the packages that are coming my way. I can’t remember the last time they’ve ever lost anything of mine. I’m sure not everyone’s experience is like mine - I’m curious how they can be so bad that they create a sense of loathing.

Zalto wine glasses. $60 per, and well worth it if you enjoy wine.

A honyaki chef’s knife and a vinyl cutting board to go with it.

Which one? IMS?

Global Parametrics | Engineers, Weather Scientist | Full-Time | Onsite/Remote London, UK, Washington, DC, Lexington, KY

We work to transfer the risk of the inevitable – natural disasters – everywhere on earth. We work with climate model datasets and are building a system to structure and operate contracts to transfer the risks of natural disasters, primarily in non-developed countries. We're backed by the British and German governments and we have a fantastic team of PhDs, engineers, scientists and just generally nice people.

Email us a resume and a little bit about you at careers@globalparametrics.com

Jobs here: http://www.globalparametrics.com/jobs/

Global Parametrics | Software Engineers | Full-time | Remote

We are developing risk transfer products, particularly for low and middle income countries, based on parametric models. Our work deals with running parametric models, loading them into our data system via DAGs, developing micro services to calculate hazard and index values, visualize risk and much, much more. We’re looking for people familiar with GCMs, insurance/risk, developing micro services, creating infrastructure to handle large amounts of data, and an interest in climate change and resiliency efforts. If this interests you, email me at globalparametrics.com via tjohnson

Global Parametrics | Spatial Database Architect | Lexington, KY or Savannah, GA / Remote | Full-time | globalparametrics.com I work for this company and we need someone who is motivated by working on difficult scientific and computing infrastructure problems. To work for Global Parametrics is a chance to work on helping low and middle income countries become more resilient to natural disasters. Many companies talk about how they are improving people's lives – here's a chance to actually do it.

Here is the job post:

We are seeking a database architect for maintaining a large and growing repository of geophysical scientific data (50TB). The bulk of the data consists of

* Global Climate Data * Numerical weather forecasting models (GCMs) * Seismic and Earthquake events * Hurricane storm tracks

This data must be updated daily and organized for consumption by scientists through consistent data access layers and APIs. Experience with large scientific datasets is preferred, and suggest that you highlight this experience compared to large transactional databases.

Specific Responsibilities:

* Responsibility for creating an optimized database architecture for efficient delivery * Maintaining and updating the database with new data sets * Refine and automate regular data harvesting processes * Refine DB queries and indexes to speed performance * Expertise in SQL and NoSQL environments

Environment:

The candidate will work largely independently, but also collaboratively with other code developers in an agile development framework. Results are prioritized over process. This senior position will report directly the to the chief technical officer.

Global Parametrics (GP) offers innovative resilience solutions in emerging economies impacted by natural disasters. GP is a for-profit social venture, with government backing from the UK and Germany.

Email careers@globalparametrics.com with your CV, a brief explanation of who you are, and a summary of your relevant technical experience to apply.

Global Parametrics | Spatial Database Architect | Lexington, KY or Savannah, GA / Remote | Full-time | globalparametrics.com

I work for this company and we need someone who is motivated by working on difficult scientific and computing infrastructure problems. To work for Global Parametrics is a chance to work on helping low and middle income countries become more resilient to natural disasters. Many companies talk about how they are improving people's lives – here's a chance to actually do it.

Here is the job post:

We are seeking a database architect for maintaining a large and growing repository of geophysical scientific data (50TB). The bulk of the data consists of

* Global Climate Data

* Numerical weather forecasting models (GCMs)

* Seismic and Earthquake events

* Hurricane storm tracks

This data must be updated daily and organized for consumption by scientists through consistent data access layers and APIs. Experience with large scientific datasets is preferred, and suggest that you highlight this experience compared to large transactional databases.

Specific Responsibilities:

* Responsibility for creating an optimized database architecture for efficient delivery

* Maintaining and updating the database with new data sets

* Refine and automate regular data harvesting processes

* Refine DB queries and indexes to speed performance

* Expertise in SQL and NoSQL environments

Environment:

The candidate will work largely independently, but also collaboratively with other code developers in an agile development framework. Results are prioritized over process. This senior position will report directly the to the chief technical officer.

Global Parametrics (GP) offers innovative resilience solutions in emerging economies impacted by natural disasters. GP is a for-profit social venture, with government backing from the UK and Germany.

Email careers@globalparametrics.com with your CV, a brief explanation of who you are, and a summary of your relevant technical experience to apply.

In light of this piece I'm curious about how countries with large sober populations (majority Muslim countries?) compare to others while accounting for other factors

A R factor is a sequence type much like a character atomic vector except that the values of the factor are constrained to a set of string values, called “levels”. For example, if you have a table of measurements of some widgets and each row corresponds to a single measurement of a single widget, you could have a factor-typed column called measurement.type containing the values “length”, “width”, “height”, “weight”, and “hue”, with the corresponding numeric measurements stored in a “value” column.

This is a very bad example of what factors are for in R, because it makes it seem like factors are for defining variables or keys in key value pairs. You can use them for that, but it isn't the intended use. A better example would be:

suppose you were comparing the amount of sugar in fruits based on several growing locations, and you had three columns:

| Fruit | Location | Density (g/L) |

Fruit would be a factor variable (let's say it takes the possibilities of apple, banana, orange), and location could be too, if it were a discrete set of possibilities (as opposed to lat/lon coords)

This author seems to forget that R was built for working with data in an analytical setting, unlike all of the languages he's comparing it to. It has creeped into other areas, but that seems to be because in the hands of a skilled user it is far easier to implement a data analysis solution. I'm sure someone will come in and say how much better pandas is, but on the small datasets, I'll stick with R, especially with how brittle and buggy matplotlib is.

This isn’t a big deal. You should probably be more concerned about the pesticides. These machines do dealcoholization, remove volatile acidity (vinegar), alter acidity (done with ion exchange, as opposed to adding tartaric acid directly to the wine). Yeah, there are probably some negative side effects as a result of these techniques, but I'd bet big that they are vast improvements over the alternatives and that the negative effects are dwarfed by those of pesticides