danthelion

At Estuary, we’re creating a real-time data streaming platform that doesn’t rely on Kafka and uses JSON as the primary data format stored in an object storage. Many people are interested in how we achieve millisecond-level latency in our data streams, so we will be publishing a series of articles on this topic!

Imagine a system monitoring payment transactions. Each transaction stream (e.g., purchase events) could be joined with customer account data (e.g., past purchasing patterns or blacklist flags). Streaming joins enable flagging potentially fraudulent transactions leveraging live context.

Materialize is powered by timely dataflow and differential dataflow, Estuary Flow uses a streaming mapreduce architecture: https://estuary.dev/why-mapreduce-is-making-a-comeback/

Gazette is at the core of Estuary Flow (https://estuary.dev), a real-time data platform. Unlike Kafka, Gazette’s architecture is simpler to reason about and operate. It plays well with k8s and is backed by S3 (or any object storage).

With the 2-pass strategy, we can write arbitrary row group sizes while using a fixed amount of memory, with probably 100-200 MiB of overhead for the parquet file processing, depending on how large the metadata is for the scratch file. without the 2 pass strategy, the amount of memory is proportional to the size of the row group.

I wrote git-genie to automate commit message writing with GPT & pre-commit hooks, works surprisingly well (most of the time) - https://github.com/danthelion/git-genie

Git-genie can now be installed as a pre-commit hook which enables commit message generation automatically based on the staged changes using GPT.

Yes, that was precisely the motivation! (+ some interest in learning how to publish a Chrome extension).

Only Chrome, might check out how to port it when I have some time.

You can now, added the URL as a dimension!

The dashboard was made using [rill](https://github.com/rilldata/rill), which uses Sveltekit

Thanks for flagging this! These were missing from the initial dataset I used as a starting point, but I can easily extend it myself.

I have missed the data file entirely and just extracted all the entities from the rendered website, then enriched all the repositories with some extra info through the API (still ingesting some data for a few repos, hitting the API limit frequently).

The goal is to have a deeper level view of public government orgs, including a more granular view of contributors and how they interact with code.

Great idea about wikidata! That's another data rabbit-hole to climb down into.

I can see how the "Country" name was confusing; thank you. Renamed it to "Government" for a bit more clarity.

That would be great, scraping that site is the entrypoint of my data pipeline!

Yeah, I definitely agree, more sophisticated metrics are in the pipeline. Wanted to get an MVP out as quickly as possible.

This GitHub page was the starting point: https://government.github.com/community

I created a dashboard to monitor government activity on GitHub

Yes, and Data Engineering is just Software Engineering "in disguise"

This way my approach when I created a similar tool, called git-genie[0]. It’s more of an educational tool first, explains the generared git command in detail.

[0] https://github.com/danthelion/git-genie

git-genie does this with the —explain flag! https://github.com/danthelion/git-genie

Also GPT-based

What if... GPT _is_ the CRUD? https://github.com/danthelion/gpt-crud

Location: Budapest / Hungary

Remote: Yes

Willing to relocate: Yes

Technologies: Python (pandas, scikit, airflow, luigi, etc.), Docker, Kubernetes, GCP, CI/CD, bash, ETL

Résumé/CV: https://drive.google.com/file/d/1-63YRu9xkMKkLxnN4CwI0_IEI0c...

Email: danivgy at gmail dot com

Linkedin: https://www.linkedin.com/in/danthelion

GitHub: https://github.com/danthelion

Hey, I'm Daniel. I'm a Software Engineer from Hungary with around 6 years of experience in a wide range of technologies. I love solving problems which require me to study previously unknown topics. I love working with data (mainly the ETL and infrastructure) part and web backend services.

I used to play around with similar projects for macos, wrote an app which displays current song info in the menubar (https://github.com/danthelion/pyet) and a python wrapper for the applescript “api” https://github.com/danthelion/wasp

Applescript is actually a lot of fun if you decide to look into it

Missed one! Spotify just released their own dataviz library yesterday https://labs.spotify.com/2018/11/15/introducing-chartify-eas...

I would recommend looking into Python-based workflow managers: Luigi[0] then Airflow[1], to get a hang of scheduling, DAGs, etc. Both are fairly simple to get started with (especially for a seasoned Python developer) and are used in production environments as well.

[0] https://github.com/spotify/luigi

[1] https://github.com/apache/incubator-airflow

Hey, I just created a similar application using the Cloud Text-To-Speech API from Google and the textract (https://textract.readthedocs.io) library to extract text from lots of kinds of documents:

https://github.com/danthelion/doc2audiobook

It runs inside a Docker container so fairly easy to try it out.

Around 1 month total time since the idea, 2-3 days net dev time.

http://reddittopbooks.com

I wanted to learn Django and wanted to start with something fairly easy. I don't really have any idea how and where to promote it, didn't get much attention in my `Show HN` thread and most subreddits remove it because of the affiliate links to amazon.

Other than that it was a fun experience and at least I can say that Django is a great framework to work with!

Yes, I even mention HackerNews Books in the `About` section as my main inspiration for this site! I wanted to try out Django and this seemed like a fun little project for starting.