HN user

jimsparkman

90 karma
Posts3
Comments34
View on HN

Since duckdb is OLAP, it would directly compare with other columnar database technologies like Redshift, Presto/Athena, etc.

Most of these systems strongly encourage or outright enforce JSONL, so that’s the defacto standard, and most tooling or pipelines are going to generate that nowadays.

You can obviously still have a row of arrays, and different systems have slightly different approaches on how to deal with those. In Spark, this is referred to as “exploding”, in Presto you would cross join to unnest an array, in Redshift you can glob on the super type.

I’m not sure I have a particular favorite, only that the database support such an operation since it is a common occurrence.

While certainly focused on larger vehicles, by all appearances Ford is much more exciting and innovative than its traditional domestic counterparts. The new electric F150, electric mustang, the Bronco series, etc. have all seemed to drum up quite the fanfare. I can’t say the same for GM.

Keeping in mind DE can mean different things at different companies, I spend a lot of time working on infrastructural components to just get at data reliably. Working in a product company with disparate generators of data, I’m often building out network connectivity (VPC peering, VPNs, etc.), subnets, ACLs, firewalls and load balancers across our visualization tools, managing job flows, controlling AWS costs, building read replicas for production databases, yadda yadda. There might be a ton of hoops to jump through before I can even start to process data and it’s the type of work that wouldn’t make sense to hand off to my DS counterparts.

With direct integrations to EMR, Lambda, and Athena, its a great tool for building pipelines and effectively costs nothing on its own and is completely headache free.

Admittedly, its not always apparent. I think a general rule of thumb for me is if I can effectively write the script in my mind ahead of time. Anything beyond that and I might regret choosing a shell script for the purpose.

Sometimes a shell script just proves a concept out and is then rewritten later in a production language.

Shell scripts are obviously imperfect, but I don’t necessarily agree with this absolutism view. In the right circumstances, you can quickly prototype an idea or process information in a shell script with such low friction. Truly the hacker mentality, which I don’t think a “real” language can always provide in the spur of the moment.

My company ran something similar recently. The cyber security team opted for “an important change to your available PTO.” It was from an internal address but from a made up name. The company is large enough (1,300 people) that you wouldn’t know if the name was actually an employee in HR or not. Because IT runs software that proxies and mangles all the links in an email, it’s super hard to evaluate the legitimacy of a URL anyways. Of course, most people clicked the link.

I’ve used Vimium in Chrome for years and years now, I couldn’t imagine going without it. It’s such second nature I don’t even think about it.

WITH (CTEs) make queries so much more readable and digestible. As a programmer who now does data and SQL, I latched on to these as soon as I found I could reduce repetition in a query with them.

I relocated to SLC recently after taking a new position in my company. It has been a very positive experience so far. Tech scene is bustling, a number of interesting meetups. From a geographical perspective, the mountains offer incredible opportunity for adventure. A mix of desert to high alpine.

I’ve lived numerous places domestic and abroad, and haven’t seen anything quite like the accessibility to adventure as is found in SLC. Trail/mountain running, numerous ski resorts, Park City (of Winter Olympics fame), rock climbing, mountaineering and backcountry skiing, etc. All within 20 minutes drive.

It’s surprisingly amazing how close to the city you can be and yet feel so far away. You can get ~6,000 feet above the valley within a few hours run/hike and see moose, mountain goats, porcupine, etc.

Also interesting is the number of outdoor gear companies that have offices here. Petzl, Black Diamond, Backcountry, etc.