julien040

I haven't seen it mentioned yet, but it reminds me of PQL (not PRQL): https://pql.dev

It's inspired by Kusto and available as an open-source CLI. I've made it compatible with SQLite in one of my tools, and it's refreshing to use.

An example:

  StormEvents
  | where State startswith "W"
  | summarize Count=count() by State

I'm trying to push the boundaries of SQL by building Anyquery[1]. It's a SQL query engine that allows you to run queries on anything (GitHub, Todoist, Parquet, Google Sheets, logs, emails, etc.)

It's mental gymnastics to transform different data sources (e.g. a spreadsheet) into a SQL database with write support, but I do enjoy the journey and learn a lot from it.

[1] https://github.com/julien040/anyquery

Hey HN, I’m Julien, a CS student, and I built anyquery, a SQL query engine for pretty much anything.

Anyquery can run SQL queries on local/remote files (CSV, JSON, Parquet, HTML, etc.) and SaaS (GitHub, Notion, TodoIst, Shopify, etc.). Anyquery can also transform a Google Sheets or an Airtable base into a SQL database with INSERT/UPDATE/DELETE support. Additionally, it can act as a MySQL server to leverage its ecosystem (BI tools, ORMs, etc.). Finally, you can run PRQL and PQL (KQL inspired language) queries with it.

Under the hood, it uses Go and SQLite's virtual tables. Therefore, you can use the SQLite ecosystem (e.g. sqlite-vec, datasette, etc.)

I would love to hear your feedback about the project.

Website: [https://anyquery.dev/](https://anyquery.dev/docs/#installati...

Source code: https://github.com/julien040/anyquery

(I know it’s a repost. I thought that posting the previous SHOW HN during the weekend was badly timed. If you find it inappropriate, please don’t hesitate to flag the post. https://news.ycombinator.com/item?id=41203559)

Hey Breck, I'm not sure you understood properly what Anyquery is. It's a CLI to run SQL queries on "anything" (APIs mostly and files). Therefore, it's hard to provide a version without downloading anything.

But I guess it's because the landing page isn't clear enough. Thanks for the feedback, I'll try to fix that.

Thanks! For querying CSV, it's quite similar to DuckDB (just a little bit slower). It's almost the same syntax (read_csv table, same parameter name, etc.)

I think anyquery strength rather comes from querying things other than files

France did something like that last month. To support the "Centre national de la musique", a new 1.2% tax was added on digital music streaming services. But rather than absorbing the cost, Spotify just raised its subscription cost. In the end, the government just taxes its citizens more instead of getting a bigger share of revenue from these companies.

https://newsroom.spotify.com/2024-03-07/spotify-to-adjust-it...

Hey, sorry. I didn't see the answer.

You have a valid point. Using a versioning field would be nicer for mods. Thank you!

I was searching for solutions to create a public collaborative tool, similar to Wikipedia, when I came across this database. I need version control and easy rollback for the moderation team. Dolt seems to be a great fit for this purpose.

  Location: Europe
  Remote: Office / Hybrid / Remote
  Willing to relocate: I'm willing to relocate in Europe or North America
  Technologies: I'm fluent with TypeScript, React, Python, SQL and Golang
  Résumé/CV: https://cdn.julienc.me/resume.pdf
  Email: contact@julienc.me

- Gut: an alternative CLI for Git (470 stars on GitHub and made it to the HN front page)

- HN Recommend: a recommendation engine for Hacker News (also made it to the front page of HN too)

Long time ago, I saw this project: https://github.com/everypolitician/everypolitician-data

I haven't used it, so I don't know about the quality of the data

It's fixed now. Thank you for reporting it.

This is awesome! I have always wanted to learn about D3.js, but I've never really started. I hope these well-made examples will help. Thank you for creating this.

It's pure speculation, but articles embeddings are computed using 512 tokens, which is roughly equivalent to 400 words. I think that using only one word does not allow the model to fully understand the context.

It's very interesting. I may try it in the future.

Oops, on the API side, there is a check to ensure the text is long enough (5 characters), but I forgot to add this check client-side. Thank you for pointing out the issue.

Try this https://hn-recommend.julienc.me/?q=Golang if you want stories related to Go.

Edit: add link

Thank you for reading it.

The hit rate is low. I've only tried to get embeddings for stories with a score greater than 100. SQL Query "SELECT count(*) FROM story WHERE score > 100;" gives me 155,228 stories and the corpus size is 108,477 stories.

108,477/ 155,228 = 0,6988236658

The main problems were 404 links and posts that weren't articles (such as tweets).

Hi, are you talking about a problem like this one? https://cln.sh/MFG3DPZn+

Yes it's an issue. Sadly, I can't fix it. I'm using the closed source "text-embedding-ada-002" model from OpenAI.

As I can see, the longer the input, the more accurate the results. Perhaps you can try something longer, like "What is a design system for UI?"

Here is the v1 of the shortcut: https://www.icloud.com/shortcuts/a7e9d236b35342c5aed1d022801...

For now, it only pushes the shared URL to the recommendation engine. If I have more time, I'll try to find a way to extract the URL from the HN page.

Are iOS share widgets using Apple Shortcuts? I wish to learn more about this technology, so it would be a pleasure to try building it.

Thank you for suggesting this.

The API is already made and can be found at https://github.com/julien040/hn-recommendation-api. I don't think it would be too difficult to build a Chrome extension that fetches it.

I thought about an algorithm with weight adjustable by the user. Now, the API returns a field with the distance between the post and the query (the square of the Euclidean distance). It's used by the interface to rank results by relevance.

Perhaps I can compute a score for each story, where each field has a weight and rank the results using this score. For example, the score could be 0.2 x score + 0.1 x comments + 1/distance - timestamp/ 10^9. The stories with the highest rank would be shown first, and the weight (0.2, 0.1, 10^9) could be adjusted by the user, as some might prefer recency while others prefer popularity.

The website features only stories with a score greater than 100 but I don't think that is the problem.

Unlike HN Algolia, it doesn't match words; it uses embeddings so stories are matched by their similar meaning rather than similar words. To find it, you might try to be more specific, such as "Paul Graham Y Combinator <facts of the article>". I'm sorry HN Recommend doesn't match your use case

I didn't try Doc2Vec. I wanted a hosted solution because I wouldn't have been able to compute all this locally (more than 100,000 posts).

If you tried it, did you have great results with? I may use it in future projects.

I'm a bit divided on this feature. On one hand, I would like to have this feature; it would be awesome to see the recommendation of people from different jobs. On the other hand, I'm a bit concerned about privacy. The system must ensure that each group is big enough to avoid the leak of someone's recommendations. I don't want anyone to know exactly what I'm liking and what I'm watching.

If I recall correctly, myCANAL (the French Netflix) used to have a similar feature. You could access the recommendations of personalities of the channel, but it was curated manually.

You can now sort by recency. I hope this helps.

I just released a new update, thanks to everyone's feedback. Now, you can sort results by relevancy, age, or score using the select.

The data is a few weeks old. Do you know when the URL was published?

By sorting by recency, I was worried I would get less revelant results. Perhaps I should add a thresold to not have too old posts

At first, the website concept focused on getting posts similar to a URL. Querying with text didn't yield relevant results.

Your solution appears better suited for this use case. Thank you.