snidane

The architecture is sound - typically called ELT these days. Dump contents of upstream straight into a database and apply stateless and deterministic operations to achieve the final result tables.

SQL server is where this breaks though. You'll get yelled by DBAs for bad db practices like storing wide text fields without casting them to varchar(32) or varchar(12), primary keys on strings or no indexes at all, and most importantly taking majority of storage on the db host for tbese raw dumps. SQL Server and any traditional database scales by adding machines, so you end up paying compute costs for your storage.

If you use a shared disk system with decoupled compute scaling from storage, then your system is the way to go. Ideally these days dump your files into a file storage like s3 and slap a table abstraction over it with some catalog and now you have 100x less storage costs and about 5-10x increased compute power with things like duckdb. Happy data engineering!

Looks like the classic mistake of every data team. Every single office person works with data in one way or another. Having a team called 'data' just opens a blanch check for anyone in the organization to dump every issue and every piece of garbage to this team as long as they can identify it as data.

That's why you build data platforms and name your team accordingly. This is much easier position to defend, where you and your team have a mandate to build tools for other to be efficient with data.

If upstream provides funky logs or jsons where you expect strings, that's for your downstream to worry about. They need the data and they need to chase down the right people in the org to resolve that. Your responsibility should be only to provide a unified access to that external data and ideally some governance around the access like logging and lineage.

Tldr; Open your 'data' mandate too wide and vague and you won't survive as a team. Build data platforms instead.

In post Czechoslovakia if you moved out of your hometown, you lost valuable networks. Healthcare and childcare were always connection based - you needed to know the local doctor, teacher, homebuilder - to get connected with specialists, skip the queues, get preferred treatment, faster processing of documents, etc. Once you part with your network, you start to build new one from scratch which takes many years. Unlike in US which is more market based - as long as you have the money you can recreate similar lifestyle elsewhere in the country.

Second, there was never really a need to rush to move to another location offering better opportunities. As consequence of the 1990s policies the local capital vanished into the hands of western entities and with it the opportunities worth moving for. The post 2000 capital which moved to the region just found spots with cheap labor to build new factories or logistic centers to keep the German powerhouse running. With an unfair advantage of cheap eastern energy, cheap eastern workers across the border and cheap euro currency as a result of sharing it with the unproductive european south.

This style is familiar to those writing dataframe logic in df libraries with sql semantics - spark, polars or duckdb relational (https://duckdb.org/docs/api/python/relational_api.html).

It definitely makes things easier to follow, but only for linear, ie. single table, transformations. The moment joins of multiple tables come into the picture things become hairy quick and then you actually start to appreciate the plain old sql which accounts for exactly this and allows you to specify column aliases in the entire cte clause. With this piping you lose scope of the table aliases and then you have to use weird hacks like mangling names of the joined in table in polars.

For single table processing the pipes are nice though. Especially eliminating the need for multiple different keywords for filter based on the order of execution (where, having, qualify (and pre-join filter which is missing)).

A missed opportunity here is the redundant [AGGREGATE sum(x) GROUP BY y]. Unless you need to specify rollups, [AGGREGATE y, sum(x)] is a sufficient syntax for group bys and duckdb folks got it right in the relational api.

What do you need non-columnar layout for? Do you expect thousands of concurrent single row writes at a time?

If you use embedded duckdb on the client, unless the person goes crazy clicking their mouse at 60 clicks/s, duckdb should handle it fine.

If you run it on the backend and expect concurrent writes, you can buffer the writes in concatenated arrow tables, one per minibatch, and merge to duckdb every say 10 seconds. You'd just need to query both the historical duckdb and realtime arrow tables separately and combine results later.

I agree that having a native support for this so called Lambda architecture would be cool to have natively in duckdb. Especially when drinking fast moving data from a firehose.

Does anybody know what is the relation between SQL as SEQUEL and then the competing language QUEL from Ingres db?

It always looked to me as if somebody back then in the database wars tried to word play on each other, one way or another.

https://en.m.wikipedia.org/wiki/QUEL_query_languages

Two people already form a 'company' and the hard part of convincing somebody else of your idea has already been accomplished. You can argue growing from 1 person to 2 is the most critical part of company growth. With 2+ cofounders this critical part is already figured out making it much less risky proposition than a single founder who hasn't convinced anybody else to join them yet.

How does one output properly csv quoted rows? It seems thr csv flag works only for parsing inputs.

There is sc-im, which is the closest you'll get to a full spreadsheet app in terminal with vi controls.

https://github.com/andmarti1424/sc-im

As someone coming from a culture with T-V distinction [1], I always wished we dropped one of the branches like English did in the past.

The informal T vs formal V causes confusion in conversation with semi-strangers. Eg. At work you never really know which way to speak to someone at a watercooler. If you choose the informal T you make them your equal. Ehich might be perceived as insulting to them, since they might want to keep a perception of superiority to you for eg. being older, more tenured, etc. Often you'd rather choose not to even engage in a conversation and just keep your thoughts to yourself. Better than ending up in a inferior position when choosing the safe V, or risking insulting someone when using the informal T form.

This felt to me like one of the reasons why English became the dominant language for business over time. Together with simplified morphology (you only need to learn the plural by adding 's' at the end, vs. 5+ other tenses of each word) it just ended up being much easier to pick up and less risky to engage in conversations and therefore higher chances of adoption by non-speakers.

[1] https://en.wikipedia.org/wiki/T%E2%80%93V_distinction

This looks great!

Please consider removing any implicit network calls like the initial "Checking GitHub for updates...". This itself will prevent people from adoption or even trying it any further. This is similar to gnu parallel's --citation, which, albeit a small thing - will scare many people off.

Consider adding pivot and unpivot operations. Mlr gets it quite right with syntax, but is unusable since it doesn't work in streaming mode and tries to load everything into memory, despite claiming otherwise.

Consider adding basic summing command. Sum is the most common data operation, which could warrant its own special optimized command, instead offloading this to external math processor like lua or python. Even better if this had a group by (-by) and window by (-over) capability. Eg. 'qsv sum col1,col2 -by col3,col4'. Brimdata's zq utility is the only one I know that does this quite right, but is quite clunky to use.

Consider adding a laminate command. Essentially adding a new column with a constant. This probably could be achieved by a join with a file with a single row, but why not make this common operation easier to use.

Consider the option to concatenate csv files with mismatched headers. cat rows or cat columns complains about the mismatch. One of the most common problems with handling csvs is schema evolution. I and many others would appreciate if we could merge similar csvs together easily.

Conversions to and from other standard formats would be appreciated (parquet, ion, fixed width lenghts, avro, etc.). Othe compression formats as well - especially zstd.

It would be nice if the tool enabled embedding outputs of external commands easily. Lua and python builtin support is nice, but probably not sufficient. i'd like to be able to run a jq command on a single column and merge it back as another for example.

Inspiration:

  - csvquote: https://news.ycombinator.com/item?id=31351393
  - teip: https://github.com/greymd/teip

Out of core computations. While your python and R script will choke after reading few hundred megs, my compiled binary cli will keep streaming through many such files with memory usage sitting somewhere near zero.

Great tool.

BUT, leaks memory like crazy.

Despite documentation stating the verbs are fully streaming.

Fully streaming verbs These don't retain any state from one record to the next. They are memory-friendly, and they don't wait for end of input to produce their output.

https://miller.readthedocs.io/en/6.7.0/streaming-and-memory/...

Another trick is to search the book title on youtube. There is a fair chance the author wants to promote the book and delivers the entire message in one hour speech in their own words instead of in hundreds of fluffed pages in the book. You can decide if you would like the buy book after watching the video.

Google made terrible mistakes with their main cash cow of search.

They removed the important feature to search only forums, ie. human generated content, and promoted SEO spam to the top instead. Public forums became undiscoverable and people moved to walled gardens of facebook and similar instead.

Then Google killed the search by trying to make it some AI answering robot. Now they ignore what you even ask it and just return to you what they think you'd want.

All that people were asking for was a better search engine and all we got was an inferior version of a chat bot.

You need to have a vision, then measure time of being blocked. If you're blocked, you're not progressing towards the vision. If you have a team of sufficiently competent people, you should expect to achieve your vision eventually.

Problem of many organizations is they are progressing without a vision and have no idea when progress gets blocked. Since nobody really measures blockage, workers typically drop the thing that got blocked and go chase metrics on something else and don't tell anyone.

Who is the rightful owner of a name or similarly, a piece of land, or an idea, patent?

The first settler? The first settler that held it for at least a year, 10 or 100? The most powerful entity claiming it?

In the modern western mind there is the notion that whoever grabs it first rightfully owns it. Which is a simple rule, but encourages squatting and holding but not using. The squatter can then hold ransom against somebody who would be the rightful owner.

At least with patents or electromagnetic spectrum there is a time of expiration. You come first and claim it for a few years after which it becomes public domain. Or we hold an auction each 5 years to maintain stable and efficient allocation of a finite and scarce resource.

With concepts like programming languages, the case with stronger base wins, like in the example of Go. People associate the Go label with Pike's Golang, not with the previous Go!.

They said the same about Galileo.

99% of the time you'll be correct labeling people as conspiracy theorists with the blanket rule - any time they are not mainstream approved. You'll end up being horribly wrong on that 1% that actually matters.

Entities without competition inflate prices until they capture all the extra new pie.

You receive increase in salary from your boss one year and next year it gets eaten away by monopolies like healthcare, education, housing and landlords.

Fortunately, getting LVT effective is just a change of a rate how much you charge for exclusive land use. The legal framework is already set up for it in many countries, so no exchtra change needed, let alone changing the 'economic fabric'. Land based taxes were used as the primary tax mechanisn in the past, going back to ancient times, for their simplicity.

The point with LVT and UBI is that you need LVT to make UBI not blow up. But once you have LVT in place you'd realise there is actually no need for UBI, because the original underlying problem - worker's disposable income being sucked away by monopolies - is suddenly gone. Under LVT the government sucks away the money from landlords, not landlords sucking it away from workers. The government can then decide to give it back to people (originally called a citizens' dividend) or reinvest into infrastructure.

Number one thing UBI does is to provide options.

No longer you'll hear employers say "if you don't like it here, go somewhere else", when the 'else' part is another abusive company like that, going homeless or committing suicide. Under UBI the option of laying flat and not dying is a valid option. As a result, either a lot of people go laying flat, or working conditions improve, because smart employers will want to attract this huge dormant working force.

The number one thing UBI doesn't handle well is rent inflation. You hand out $1000 dollars per person monthly, expect rents to go up by about $1000 monthly as landlords realise there is all this extra disposable income in peoples' hand right now.

However, this is just an exaggerated effect of monopolies sucking out all aggregate disposable income out of economy that is already happening. Monopolies by definition don't have price down pressures, so they always price expand to capture anything people might have extra. Since landlording is the biggest aggregate monopoly in the world, landlords capture any disposable workers' income. No matter if they get a raise from their boss, the landlord always takes it away.

The only solution is an antimonopoly measure that actually works. Anti-trust bureaus always end up corrupt and can't be relied on. Systematic rule based antimonopoly taxes seem to be the only current viable option. One such being the famous land value tax (LVT), which captures extra pricing on rents - enabling workers to actually keep their hardly earned income raises.

Since UBI won't work without curbing monopoly price expansion, and landlords' rents are the biggest aggregate monopoly out there, there won't be any UBI without LVT.

Exact same - can I use fzf gistory search using Ctrl+R like I can in shell?

Flatten the damn thing and process it relationally. Linear data scans and copying are so fast on modern hardware that it doesn't matter. It's counterintuitive for people to learn that flattened nested structure with massive duplication still processes faster than that deeply nested beast because you have to chase pointers all over the place. Unfortunately that's what people learn at java schools and they get stuck with that pointer chasing paradigm for the rest of their careers.

That's a hardware management question. The optimized binary used in my shell script still runs orders of magnitude faster and cheaper if you orchestrate 100 machines for it than any Hadoop, Spark, Beam, Snowflake, Redshift, Bigquery or what have you.

That's not to say I'd do everything in shell. Most stuff fits well into SQl, but when it comes to optimizing processing over TB or PB scale, you won't beat shell+massive hw orchestration.

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes.

What Python is to Java, Shell is to Python. It speeds you up several times. I started using inline 'python -c' more often than the python repl now as it stores the command in shell history and it is then one fzf search away.

While neither Shell or SQL are perfect, there have been many ideas to improve them and for sure people can't wait for something new like oil shell to get production ready, getting the shell quoting hell right, or somebody fixing up SQL, bringing old ideas from Datalog and QUEL into it, fixing the goddamn NULL joins, etc.

But honestly, nothing else even comes close to this 10x productivity increase over the next best alternative. No, Thank you, I will not rewrite my 10 lines of sh into python to explode it into 50 lines of shuffling clunky objects around. I'll instead go and reread that man page how to write an if expression in bash again.

If you need to maintain some sort of deductive correctness - ie. my inputs are correct and my code is correct therefore my outputs are also correct - you're gonna expose yourself to only a tiny amount of real world problems.

Data engineering is typically closely aligned with business and it's processes are inherently fuzzy. Things are 'correct' as long as no people/quality checks are complaining. There is no deductibe reasoning. No true axioms. No 'correctness'. You can only measure non-quality by how many complaints you have received, but not the actual quality, since it's not a closed deductive system.

Correctness is also defined by somebody downstream from you. What one team considers correct, the other complains about. You don't want to start throwing out good data for one team just because somebody else complained. But many people do. Or typically people coming from SWE into DE tend to, before they learn.

That sounds like infra ops, not data engineering. It's a common theme, though, for DEs to slip into maintaining infra and analysts taking on the role of engineering data pipelines.

Engineers demanding unit test for data is a perfect test to weed out the SWEs who are bit DEs. Ask about experience with data quality and data testing when you interview candidates and you'll distinguish the people who will solve a problem with a simple relational join in 1 hour (DEs) vs those who will try to unknowingly build a shitty implementation of a database engine to solve a problem in one month (SWEs trying to solve data problems with C++ or Java).

Technology can be both good and bad, depends on the holder.

The government clearly has a goal they want to achieve and cleaning up their data with AI seems like a solution more likely to work than throwing a small army of bureaucrats at it.

The right wingers when it comes to government control advocate for either 1. don't let government have much control (starve the beast) or 2. if govt already has control, let them at least do it efficiently without wasting taxpayers' money.

The left on the other hand want to empower governments with more and more control, but once the govt has it, they want to prevent it from doing it efficiently - by disallowing efficient use of data and technology - which catches them red handed as they pretty much just want to empower themselves through grants, inefficient paper shuffler bullshit job and other forms of transfers at the expense of others.

My Viagogo experience in 2019. After an hour waiting in line for the event I get my ticket scanned only to realize they sold it to somebody else at the same time. Another 30 mins waiting at the security trying to get somebody from their call center to pick up the phone and tell me I was out of luck that day. I only got a refund in form of viagogo credits which expired worthless as there were no events happening due to covid. Buyer beware.