caerbannogwhite

In reply to HackerThemAll's https://news.ycombinator.com/item?id=48659482 comment (which I think is a bit pretentious and unfair, and HN does not show my direct reply to the comment):

I think that's a red herring: the query is sandboxed DuckDB-WASM SQL (never executed as script), it's never injected into the DOM as HTML, and the page enforces a strict CSP that blocks inline script regardless. NoScript probably flags it because it's SQL-shaped text in a cross-site query string, and it matches its injection heuristic.

Well done for laying that down in that way! I'm just wondering how much it would be on my side to support that, since I can see that you support cases like "count(*)" and "group by" inside visualize. I can see you have a full bison grammar, I only have a custom parser at the moment, but at least your implementation is in C. But I'm happy to follow thomasp85.

Thank you for clarifying that! I don't think they got that and they also trusted NoScript a bit too blindly

Thank you, that's quite a lot of great quality feedback!

Agreed, it's not production-grade yet, and robust standard errors are the priority. HC and cluster-robust SEs are the biggest credibility gap, which is why I just added an (Eigen-based) linalg kernel for the next release: it's the groundwork so the regression fitter can ship HC0–HC3 and clustered SEs (and bootstrapped, via the existing bootstrap aggregate). Margins and GLMs (IRLS on the same kernel, so not really a backend rewrite) are the next layers.

About the scope, agreed, and already narrowing. I'll credit posit ggsql as the real grammar-of-graphics tool; mine stays a minimal built-in. The plan is to go deep on the statistical core and lean on the DuckDB ecosystem for the rest.

For the market fit I'd frame it a bit differently. Not trying to pry anyone off Stata/R for rigorous work, I know that those aren't going anywhere soon. The niche is stats where the data already lives: people in SQL/DuckDB who currently round-trip to R/Python for a regression. R can run in the browser now (webR), but not in the same engine as the data. The stats-duck runs inside duckdb-wasm, so there's no separate runtime and no marshalling. And DuckDB is already far faster than R for the ETL/wrangling around the stats.

The plan is to figure out from real usage. For now, I'll focus on the core.

Appreciate it, good call! I'll rename the ggsql bits and the docs to describe it as ggsql-inspired. I'll also point to ggsql for the real thing. Thanks for being kind about it!

Yes, that's exactly its main purpose! I initially started because I needed a dataset browser. I work with clinical trials, so we usually get raw data files in all possible formats, from CSV to EXCEL and, of course, SAS formats. But since I was already using DuckDB, I thought about extending it a bit further, so you can quickly get a glance at the data.

Thank you! I'll add note about it

First of all, nice to meet you! The honest answer is timing: the first VISUALIZE commit on my side was april 25th; ggsql-duckdb's first commit was April 23rd. So I genuinely didn't know it existed!

About the name: yours is the official Posit one, and you were there first, so I'll rename my branding; there should be one ggsql, and it's yours. Mine only exposes VISUALIZE as the keyword anyway.

The actual name of the extension is the-stats-duck, which runs inside duckdb-wasm (it powers an in-browser data tool) and emits a Vega-Lite spec inline for the host to render. Your implementation (which I think is Rust based and an in-process HTTP server that opens a browser), is a native pattern, but correct me if I'm wrong! mine is deliberately thin and wasm-safe, not a whole engine.

About the parity, you're right, and I'm not chasing it; for real grammar-of-graphics, ggsql should be the tool! but, if that's ok with you, I'd love to keep the syntax aligned!

That's exactly the inspiration! I made a post here on HN about that a few weeks ago: https://news.ycombinator.com/item?id=48108815

My plan is to release a blog post about all VISUALIZE current features next week, explicitly mentioning Posit's alpha GGPLOT.

/edit: clarifications

(author here) I originally started this project to solve one of my main daily issues: having to quickly check data sparse in a bunch of datasets in different file formats (.xpt, .sas7bdat, .csv, .tsv, Excel, etc) and several folders.

I wanted it to - be performant: some files can have 50k rows or more, so an HTML table could not be the solution - be easy to configure: web-app, no installation or configuration, so my colleagues could use it too - (later) have a way to quickly explore data via queries: what's better than DuckDB for this?

Then, a few weeks ago, I read Thomas Lin Pedersen / Posit's alpha release here on HN (https://news.ycombinator.com/item?id=47833558), and I thought it could be nice to have charts in Bedevere. So I asked Claude to add the GGSQL grammar to my DuckDB extension (the-stats-duck - separate repo: https://github.com/caerbannogwhite/the-stats-duck).

Here are some other things you can try:

VISUALIZE bill_length_mm AS x , bill_depth_mm AS y , species AS color FROM penguins_clean DRAW point SCALE x DOMAIN 30 60 SCALE y ZERO false SCALE color TO viridis ;

VISUALIZE bill_length_mm AS x , bill_depth_mm AS y , species AS color FROM penguins_clean DRAW point FACET BY species COLS SCALE x ZERO false SCALE y ZERO false ;

Feedback is very welcome!

Currently working on Bedevere Wise (https://bedeverewise.app), a browser-based SQL data viewer for the file formats still widely used by biostat / clinical-data people: SAS (sas7bdat, xpt), SPSS (sav), Stata (dta), plus Parquet, Excel, and CSV. Everything runs locally: DuckDB-WASM (the SQL engine), the file parser, statistical and plot library (a DuckDB extension that I built).

I wanted a "drop file → SELECT * FROM it" and run a few other explorative queries on a dataset (provided in one of the formats mentioned above). Sometimes, it might even be a whole nested subtree with dozens of files that includes all or most of the formats I mentioned above (trust me, I've seen it many times). I also wanted something easy to use for my colleagues: no installers, no configurations, no faff. And, most importantly, files never leave the device (which matters for clinical data).

The plotting half is GGSQL. I read Thomas Lin Pedersen / Posit's alpha release a couple of weeks ago (https://news.ycombinator.com/item?id=47833558) and that's when I realised I could add the "Grammar of Graphics inside SQL" into Bedevere. So pasting

VISUALIZE bill_depth_mm AS x , bill_length_mm AS y , species AS color FROM penguins_clean DRAW point ;

in the editor pops a chart without anything leaving the device.

This is the demo for the impatient (I admit I am usually one): https://bedeverewise.app/demo (The query is ready to be run in the editor).

Feedback is very welcome.