pacofvf

After becoming frustrated with the difficulty of implementing reliable and transparent data quality checks, I developed a new framework called Weiser. It’s inspired by tools like Soda and Great Expectations, but built with a different philosophy: simplicity, openness, and zero lock-in.

If you’ve tried Soda, you’ve probably noticed that many of the proper checks (like change over time, anomaly detection, etc.) are hidden behind their cloud product. Great Expectations, while powerful, can feel overly complex and brittle for modern analytics workflows. I wanted something that falls between lightweight, expressive, and flexible enough to integrate into any analytics stack.

Weiser is config-based; you define checks in YAML, and it runs them as SQL against your data warehouse. There’s no SaaS platform, no telemetry, no signup: just a CLI tool and some opinionated YAML.

Some examples of built-in checks:

1. Row count drops compared to a historical window

2. Unexpected nulls or category values

3. Distribution shifts

4. Anomaly detection

5. Cardinality changes

The framework is fully open-source (MIT license), and its goal is to be both human- and machine-readable. I’ve been using LLMs to help generate and refine Weiser configs, which work surprisingly well, far better than trying to wrangle pandas or SQL directly via prompt. I already have an MCP server that works well but it's a pain in the ass to install it in Claude Desktop, I don't want you to waste time doing that. Once Anthropic fixes their dxt format, I will release a MCP tool for Claude Desktop.

Currently it only supports PostgreSQL and Cube as datasource, and for destination for the checks results it supports postgres and duckdb(S3), I will add snowflake and databricks for datasources in the next few days. It doesn’t do orchestration, you can run it via cron, Airflow, GitHub Actions, whatever you want.

If you’ve ever duct-taped together DBT tests, SQL scripts, or ad hoc dashboards to catch data quality issues, Weiser might be helpful. I would love any feedback or ideas. It’s early days, but I’m trying to keep it clean and useful for both analysts and engineers. I'm also working on a better GUI.

GitHub: https://github.com/weiser-ai/weiser Docs: https://weiser.ai/docs/tutorial/getting-started

I'm happy to answer questions or hear about what other folks are doing to address this problem.

It looks very important, popular and well established, but what is it?

It's easier to explain what Cube is if we first define what the Semantic Layer(SL) is. In a few words, the SL is the abstract representation of business objects, for example: sales, users, conversion rates, etc. Cube provides the language to define the SL, an API to access it, access control mechanisms and a caching layer. It's important to emphasize that Cube is a stand-alone SL, decoupled from any BI visualization tool. That's the "headless" part, and I would also add that is "feetless" since it supports multiple source DBs. Looker the other big name in the space has the incentive of selling you more usage of BigQuery and of locking you in with their UI, it just recently started to open up to the idea of APIs. The idea is that you have a central place where you define the SL and then you don't need to duplicate the definition on every downstream application, which may lead to errors or inconsistencies.

Is it that it can perform a single query across multiple databases?

Cube allows you to join data from multiple databases at the caching layer, that's fundamentally differently than a federated query engine. But from the downstream application perspective it has the same outcome. By being done at the caching layer it has inherent advantages and limitations vs federated queries.

I really like these series of articles by David Jayatillake that go into deeper detail:

1. https://davidsj.substack.com/p/semantic-superiority-part-1 2. https://davidsj.substack.com/p/semantic-superiority-part-2 3. https://davidsj.substack.com/p/semantic-superiority-part-3 4. https://davidsj.substack.com/p/semantic-superiority-part-4 5. https://davidsj.substack.com/p/semantic-superiority-part-5

They are using cloudformation which predates Kubernetes and it's meant to solve the same problem. CloudFormation still uses docker, originally it used AMIs. I've worked for a lot of Fortune 500 companies that don't use Kubernetes and use Docker.

Upon written notice to you, we may (or may appoint appoint a nationally recognized certified public accountant or independent auditor to) audit your use of the Services and Mapbox Software to ensure it is in compliance with these Terms. Any audit will be conducted during regular business hours, no more than once per 12-month period and upon at least 30 days’ prior written notice (except where we have reasonable belief that a violation of these Terms has occurred or is occurring), and will not unreasonably interfere with your business activities. You will provide us with reasonable access to the relevant records and facilities.

LOL no.

NiFi is meant for stream processing and Airflow for batch processing, if your NiFi triggers an Airflow DAG that means that your entire process is batch processing and you shouldn't use NiFi in the first place. If you still want to do stream processing then use Airflow sensors to "trigger" it.

For the love of god, don't use NiFi to trigger an Airflow DAG.

Where is Spain and Italy in your Options?

my point is that the benchmark is deceiving, again if you are doing data analytics or ML, then you already are using numpy/pandas/scipy, so thats not a valid argument.

"Python is slow" argument just shows complete ignorance about the subject. (and there may be good arguments for not using python)

First of all if you are doing "for i in range(N)" then you are already doing it wrong, for ML and data analytics you should be using NumPy "np.arange()", Numpy arange doesnt even run in "python" it's implemented in C. So it may even be faster than swift '..<' . Let me know when you can use swift with spark.

SinTrafico | Frontend, Full Stack, Backend, Data Engineering | Onsite Mexico City, Remote in Mexico or Latin America | http://sintrafico.com

SinTrafico is a profitable and growing Mexican Startup. We are the leaders in the mobility industry in Mexico and soon in Latin America and beyond.

We are looking for senior developers on different roles and we are open to do remote work and also willing to help to relocate to Mexico City.

Stack: AWS, Python, Vue.js/React.

You should definitely apply if you have experience dealing with Spatial and Geographic data.

Please send me your CV at paco@sintrafico.com

But that didn't happen, because Obama understood the rule of law, you can't defend Trump illegal actions by saying that if Obama had done that then the "leftists" would be cheering that.

Well that's the same with any cloud provider, one compromised account with enough access could be catastrophic. In BofA there is an AWS IAM equivalent. Also the BofA cloud is not accesible from the Internet.

I had experience in Quartz (BofA cloud), and deploying is 100 times easier than AWS, everything is automated. Imagine building a cloud service where you trust all your clients, and all must share the same information if they have the correct auth, what I trying to say is that their use-case actually made it simpler, that's where the savings in software had came from.

How long does it takes to get a O-1 Visa?

I would add Flask too, I guess everyone on that realm of python learned the lesson from werkzeug, which after all these years some functions/classes still doesn't have docstrings and variable names are awful.

But tourists drive prices up for markets that aren't regulated like hotels or restaurants, Healthcare is regulated and we will probably see some kind of legislation towards this issue with the new Mexican government, it probably will make it even cheaper for you guys to come here but you will have to enroll on our socialized healthcare.

We have socialized health care, so you may think that you spend some dollars on the Mexican economy but, the American Caravan distorts our markets and drive prices up.

They should be barred from running again for office if a shutdown lasts more than 3 days.

2007 Dollars != 2018 Dollars.

I've read both, Haaretz's it's more in depth, It give us an insight of what happened at Olmert's cabinet and his conversations with Nethanyu.

What do you use for hosting the vector tiles?

Funny, in Mexico we use a lot "Ahorita" too, which it's just the diminutive for "Ahora" ("Right Now" in English). The word is very ambiguous here, it can mean "Right now" or "In a few minutes/hours/days" or if used sarcastically "Never".

Edit: ohh my gosh I just remembered another case where it's used for past events:

- Perdí mis llaves (I lost my keys)

- ¿Cuándo? (When?)

- Ahorita (Just a few seconds/minutes ago)

They should use Cinnamon, or just absorb Mint, that's the most popular distro right now anyway, and it is ubuntu-based.

my workflow with complex queries despite the fact the outcome is always the same: Write a SQL query to see if it works as supposed, use a query builder in another language, analyze the code and the query because it's not as fast as I would liked to, realize the entire code in another language would be faster if it were a stored procedure in SQL, write stored procedure, repeat.

Well besides the 1/100000 of a day second the most interesting about the Decimal Time (DT) was the calendar system it used, Unix Time works like that, but it's in 1/86400 seconds instead of 1/100000 seconds like the DT or the Julian Day(JD) time system, and the Unix time is in seconds instead of days.

For example to convert from DT to JD you would only need to substract the Epoch in JDs from when the DT was adopted (something like JD-2339410, eg. Current_JD - (JD_epoch - 1792_in_JD) ), to convert from Unix time to JD and you would had to do the same but for Jan 01 1970 and then multiply by 86400 ((JD − 2440587.5) × 86400). Everything would be easier if we used 1/100000 of a day seconds.

Uptime 36 days 112GB written, I listen music like 6h to 8h per weekday.

Yes even the Peso has already recovered from its after hours low, that means that the market already had priced in the "Trump risk", it just erased any gains propelled by the Clinton optimism of the past few days, right now is trading better that its all time low from a couple of weeks ago.

When did HN become /r/worldnews ?

Dirt poor countries with a quarter the resources are beating them badly, Korea, Malaysia, Chile.

where the Mexican side provides cheap day labor and a "safer" platform for criminal activity due to the financial disparity.

Mexico now effectively a narco state ruled by the cartels

The solution we've pursued thus far does indeed make Mexico richer, but it also makes parts of the US poorer.

Well, as we can see from southern Tucson, that city has become progressively worse as more non citizens have taken it over.

I think we can agree that the guy at 2016 IAC ruined BM this year.

In my country, Car makers own dealerships, well actually only a small number of them, most Dealerships are franchises, so we have a mixed model that works pretty great for all involved. New makers that arrive to my country expand with their own dealerships, and after they gain traction they start to sell franchises, when well stablished car makers want to expand to new markets they usually do it with their own dealerships too.