akbo

You can stream AVRO records. AVRO is a well defined, binary, space efficient serialization format that can be used to serialize individual records (for streaming) or a whole set of records (for file storage). That should cover all of the authors requirements, right?

More info: https://www.confluent.io/blog/avro-kafka-data/

I have a different perspective. In a drug trial you have to prove that a new drug is safe before handing it out to a bigger number of people. The default case (if you don't test the drug) is that nobody gets it.

In this case, it is different. People will put pressure on the government to allow concerts again rather sooner than later and the default case is probably that there will be concerts again before the Corona situation is over. If you don't do the test, you have to make the decision whether / when to allow concerts again blindly.

So basically you have the situation where people will take the new drug anyway and you have the choice whether you want to test its safeness before they do.

If you do the test and it turns out to be unsafe, at least you have good arguments when you need to make the decision.

Have you tried remote development with VSCode? IT works really well! https://code.visualstudio.com/docs/remote/ssh

I love it! What I always missed in, e.g., matplotlib, is the interactivity. I need to be able to zoom in and out without changing code. Hover information is another great feature coming to mind.

I'll just leave this here: https://en.wikipedia.org/wiki/Ninety-ninety_rule

This is the corresponding wiki, which is also apparently open source and can be self-hosted and lives in the same GitHub org as the editor: https://github.com/outline/outline

I see where you're coming from.

My use case is this: as a data scientist, I start new code bases all the time. Each project, simple experiment, data analysis, etc. needs its own cleanly separated dependency environment so I don't end up in dependency hell (I have 12 conda environments on my machine right now). Conda allows me to handle these environments with ease (one tool and a handful of commands -> as detailed in the article). With conda, I also have my data science Python cleanly separated from my system Python.

Of course there are other tools that can handle this use case. But pip alone won't do the trick. I don't like to have three separate tools for this (pip + venv + pyenv).

When I put something into production, I naturally want to keep using my conda environment.yml and have the same environment in dev and prod instead of switching to pip + requirements.txt, which might introduce inconsistencies.

Sounds like you just need to train your colleagues to be a bit more disciplined with their package managing. It's not that hard to be clean about dependencies with conda. Maybe my take on it can inspire you here: https://haveagreatdata.com/posts/data-science-python-depende...

Re the extra "virtual environment", you can just use conda's base environment in prod. Here's how to do that: https://haveagreatdata.com/posts/step-by-step-docker-image-f...

Dependency management can be pretty overwhelming for a lot of people entering Python. This is especially true in the data science realm, where many don't have a SWE background. Even after you have selected a tool, it can be easy to use it in a poor way. I have recently written a short article on how I use conda in a disciplined way to manage dependencies safely: https://haveagreatdata.com/posts/data-science-python-depende...

Also, I'm not sure why you called CSV unsafe.

One example of it being unsafe that happened to me: I got a CSV file written by a program with a broken implementation of a CSV writer that didn't quote string fields when there was a newline in them (in my case only the first half of a newline: carriage return). Then I read the file with a broken implementation of a CSV reader that assumed that the carriage return meant a new record and filled both parts of the broken line with N/As instead of throwing an error. This way the data in the sink didn't match the data in the source. This is the loss of data integrity, which I would call unsafe. It doesn't happen if you have a file format that serializes your data safely.

Due to the format being underspecified, many people roll their own unsafe CSV writer or CSV reader, thus every CSV file (where you don't completely control the source) is potentially broken.

Edit: Browsing your Github account I found that you implemented a CSV parser in Rust. I didn't know that when I wrote the above comment, so I was definitely not trying to imply that your particular CSV parser is unsafe.

These tools probably do a good job at processing CSV/TSV/DSV (haven't tried them). However, I would love if we could just stop using delimiter-separated value files alltogether.

Why? Because the file format is an underspecified, unsafe mess. When you get this kind of a file you have to manually specify its schema when reading it because the file doesn't contain it. Also, due to its underspecification, there are many unsafe implementations that produce broken files that cannot be read without manual fixing in a text editor. Let's just start using safe, well-specified file formats like AVRO, Parquet or ORC.

As a data scientist, I have had lots of issues because the data I got for a project was a CSV/TSV/DSV file. I recently spat out a rant on this topic, so if you want more details, check out https://haveagooddata.net/posts/why-you-dont-want-to-use-csv...

If you are interested in grading Sudokus by the difficulty of the human solving techniques required, you might want to check out the following two books:

- "Sudoku Programming with C" [https://www.apress.com/de/book/9781484209967] where a Sudoku solver and grader is implemented.

- "A to Z of Sudoku" [https://www.wiley.com/en-us/A+to+Z+of+Sudoku-p-9781847040008] where many human solving techniques are described and rated by their difficulty.