cgopalan

Thanks! Its true, many things are frustrating, like you say - bulk downloading. Yes DuckDB is amazing, and I use it in more than one of my projects. It has a lot of use cases for the data lake.

I agree, it would be more beneficial to query on patterns, but the main use case I saw for this app was non-tech folks wanting to see data in s3. I assume they wouldn't do fancy stuff like querying multiple files. But maybe I should add that functionality. Thanks for the suggestion!

A self-hosted, better browser interface to preview s3 files. This overcomes the limitation of the s3 console not allowing querying or viewing files beyond a certain size. It also gives you information about the column types. It could be a good use case to present this to non-IT teams in orgs who just want a preview of the data in the files or want to do some simple explorations. Frequently, they are denied access because of the possibility of doing something with confidential data via the s3 console.

https://github.com/cgopalan/s3fileviewer

For occasions like birthdays or Christmas where people want to give you gifts, I have always wanted to ask them to make donations to charities of my choosing instead. So I built an app to enable this: http://donateyourgift.com/ It is very simple but I didn't find anything quite singly-focused like it, so I built it just to scratch my own itch.

I was born and brought up in India, moved to the US at 27, but only now in my 50s chanced upon the amazing music that is qawwali. Till a couple of years ago I had no idea that such a treasure trove existed as part of Hindustani classical music (I was only familiar with the Indian side of Hindustani classical and the qawwalis in Indian films). I binge out on Coke Studio Pakistan songs and videos and find them mesmerizing. Fareed Ayaz and Mohammad Abu are treasures. Thank you for this, I’ll be sure to listen!

For occasions like birthdays or Christmas where people want to give you gifts, I have always wanted to ask them to make donations to charities of my choosing instead. So I built an app to enable this: https://apps.apple.com/ca/app/donate-your-gift/id6760786102 It is very simple but I didn't find anything quite singly-focused like it, so I built it just to scratch my own itch.

This looks amazing purely from an engineer's perspective that wants to level up on something they have worked on for a while with enough understanding to build working systems with it. Just confirming, this is more useful for developers that have worked with Erlang/Beam correct? Not so useful for a Beam newcomer?

Good for you then. I myself have not come across the (loud/louder) TDD exponents advocating for using TDD in system/integration testing, they mostly focus on unit tests. If you can point to some examples, it would be a learning experience for me. If not that's fine too, I am glad that there are voices out there like yours.

My experience is that TDD just ensures that the code is unit-testable. This can lead to more complex code when it need not be complex. I definitely write tests, but my default approach is to make the test simulate how a user would use the functionality. So mostly a higher level test like a system test. And you can do this only when you have a bigger picture of the program that you write (which need not be driven by unit tests, just need to elucidate what the program needs to do and break it into steps). I don't rule out unit tests, but my approach is to start with tests that resemble system tests, and if i specifically need a unit test for a hairy algorithm in a function, use a unit test to help.

Also the higher level you test at, the less probable that you have to change the tests when you change a piece of functionality.

I have pondered this question before and I have seen people recommending "Philosophy of software design" by John Ousterhout, but my qualms with Clean Code is not that it needs a substitute, its just that its a fairly simple set of concepts about which Bob makes a big deal. I did read some of his books, but I realized its only about 10% of what makes a competent software engineer. My suggestions to people starting out or even seasoned programmers are that get an idea of what he advocates (TDD, SOLID and all that) but then design of programs is just a small part.(And I also can debate the usefulness of both TDD and SOLID. Personal opinion coming: they are great for small or greenfield projects but almost always don't hold up in the real world).

Learn about other kinds of (much more effective) testing like System/Integration testing, Property-based testing. Spend a lot of time learning about databases and SQL. Maybe get into somewhat esoteric topics like constraint solvers and logic programming. You may not use these but it helps to know there's a wide world out there, and they do bend your brain enough to enable you to think differently.

Time is limited. It does matter what we spend it on.

You mean homogenous instead of heterogenous, right?

Thank you!

I use it on my 2015 Macbook pro. Its amazing how quickly you can get set up, kudos to the authors. Its a dog in terms of response time for questions, but that's expected with my machine configuration.

Also they have Python (and less relevant to me) Javascript libraries. So I assume you dont have to go through LangChain anymore.

Thanks for the talk link, it completely kept me engrossed and was a pleasure to watch! Addendum question: what is the state of database drivers for Elixir? Does it have mature libraries for Postgres for eg? I imagine if it's targeting web development, it will need those.

You should look at the command `python -m venv`. Its built in, and is a breeze to create virtual environments. I guess it does not provide shortcuts for activation etc, but I am ok with that.

I thought one of the advantages of pyenv was that you switched to also creating virtual envs using pyenv and managed virtual environments with it too. Maybe not.

I used pyenv in my previous company. At first it was fine, but after a while with multiple versions of Python installed, things stopped working. The right virtual env was not activated, etc. (It could have also been because our software updates like OS and security upgrades were pushed by desktop support). I removed all of it, and just resorted to installing multiple versions of Python the old way (downloading from python.org) and then use `python<version> -m venv /my/virtual/env` to manage my virtual environments. Things are more stable, and I don't feel like its magic. I am not going back to pyenv.

Amidst all these great discussions, I would like to point out that this article really helped me get my head around a B tree and why its a great optimization on top of the Binary Search Tree. Thanks to the author!

On his website: https://mitchellh.com/zig

That's great to know. Thanks!

Sorry, thought it was a fairly familiar acronym. IC = individual contributor.

Mitchell is the only person I can think of who went through the cycle of tinkerer/IC -> founder -> CXO and then back to IC in his own company. Also, his writings on Zig has been tremendously helpful for someone like me who is curious about it. Huge respect from a fellow IC!

Does a possibility of a Python API figure anywhere in your roadmap? Or am I missing the point here? Seems like it would be a smart choice to have one on the statically-typed side, and one on dynamically-typed side?

I am trying to see if I can work on more interesting problems. Julia Evans has this great quote: "I have one main opinion about programming, which is that deeply understanding the underlying systems you use (the browser, the kernel, the operating system, the network layers, your database, HTTP, whatever you’re running on top of) is essential if you want to do technically innovative work and be able to solve hard problems."

My using snowflake on the job made me more curious about modern analytical databases, so much that I want to get more into the internals. I am on a journey that's making me read more about database internals, take free courses online, and this in turn is leading me to some auxiliary things I need to know for knowing about databases, like cpu architecture (cache handling, branching etc) and further down the rabbit hole. To some extent, it's helping me overcome the feeling of inadequacy due to not having a computer science education even though I have been in the software field for more than a couple of decades. Not that it has harmed me much - I have always felt that you could learn things on the job in software development. But it just feels somewhat right to go back to the basics.

You say you are using dbt, so doesnt "dbt test" provide you with the functionality to test? I assume by testing sql pipelines, you want to test if the data written to intermediate stores conform to what you expect. You should be able to do that with dbt test. If you are using an analytical database like Snowflake you could direct the results of the dbt run and dbt test to a test database and do your testing there.

Learn TLA+. I have been doing the usual things for software verification like reviewing PRs, writing tests, but these are all after implementation. As a senior engineer doing more design work, I am wondering if I can do some sort of verification at the design stage. Not sure how much I can apply TLA+ to regular enterprise software development, but I want to explore.

Hired a tutor from Mexico for about $20 an hour, and do one lesson every week. I have been learning it for a year and happy with the progress. Btw I dont expect to be completely fluent for a few years, but the journey is fun. Plus we have developed some great cameraderie, and I look forward to the class every week.

And then of course:

- Take it outside

- Expect the unexpected

:)

Thanks for this suggestion, I have seen this before and its not really my area of interest. I do want to read Petzold's "Code" though. I hear a second edition is out!

Just wanted to point out that "content with programming ability" to me is not the same as "I am done learning". I just feel that the competency I have in turning a problem into code is good enough for the time being, and it would serve me better to increase my skills in other areas for now. Software engineering after all, is much more than programming.

This looks very interesting. Thanks!