nooorofe

it was fixed with workaround, in newer versions, they copy statically compiled nodejs to the server and it works.

Airflow is far from perfect, but I don't understand your concerns. I work in a big and messy company and even messier department. We have jobs running in Databricks, Snowflake, sometimes we read data from API end points, or even files uploaded to SharePoint (my group is not building DW). Airflow lets me organize it in a single workflow. At least I know that every failed job is reported by email and I don't need to search multiple systems - all starts from Airflow.

Why should biz logic that just needs to run Spark and interact with S3 now need to run a web server?

Webserver is mostly UI. Scheduler service triggers the jobs.

We have groups which run everything as Bash Operator, no dependency issues that way.

You maybe have a very specific use case in mind, the main points of using Airflow for me

* Single orchestration center: manual job control (stop, pause, rerun), backfill; automated scheduler/retry; built-in notification

* Framework built around "reporting period" - it enforces correct abstraction, if a data batch is broken, I can rerun it and rerun all dependent downstream. How do you fix data in event driven workflow?

* managing dependencies

In most cases all Airflow does is running your job with passing it "date" parameter. You can test your code without Airflow - just pass it a date and run from command line.

if you want to convert some scheduled pipeline to some event-driven architecture

Airflow has sensors and triggers. https://airflow.apache.org/docs/apache-airflow/stable/author...

But in the core it is built around data pipeline concept, event driven pipeline will much more fragile. Airflow intentionally doesn't manage business logic, it works with "tasks".

There is nice summary on the topic: https://aws.amazon.com/blogs/big-data/choosing-an-open-table... ("Optimizing read performance"). Those technologies primary "Data Management at Scale" but they also extend capabilities provided by raw storage formats such as parquet. So they may help you, but the question if you are really need it. I haven't worked with BigQuery, it may include [similar features](https://cloud.google.com/bigquery/docs/search-index).

You need to define what "latency" means in your case and what is "quite high levels". We are talking about analytical data storage, it is designed for efficient batch processing. To find a single record is not a primary goal of the architecture - you will need some kind of caching/indexing for fast search. Sometimes adding "limit 1" for your single record search may solve the problem.

Be sure you are using efficent data storage format as parquet, check size of the files to be sure you don't have the ["small file problem"](https://www.royalcyber.com/blog/data-services/managing-small...), then check if you are using relevant BigQuery features. And before and after those checks run "explain" on your query, if you don't use partition keys or indexed columns your search results won't be instant in any big data system.

    from zoneinfo import ZoneInfo
    import pendulum
    import datetime

    def pendulum_to_datetime(pendulum_dt: pendulum.DateTime) -> datetime.datetime:
        return datetime.datetime.fromtimestamp(pendulum_dt.timestamp(), 
    ZoneInfo(pendulum_dt.timezone_name))
    
    # test 
    
    df = pd.DataFrame([[1, 2], [1, 2]], columns=['a', 'b'])
    df["time_column"] = pendulum_to_datetime(pendulum.now())
    
    print(df)

       a  b                      time_column
    0  1  2 2023-11-19 18:14:16.027777-05:00
    1  1  2 2023-11-19 18:14:16.027777-05:00


    >> df.dtypes

    a                                         int64
    b                                         int64
    time_column    datetime64[ns, America/New_York]
    dtype: object

pendulum is probably your best bet

It is best until you work with Pandas DataFrame (https://stackoverflow.com/questions/47849342/making-pandas-w...)

I doubt it was .exe, probably .msi (installer) or zip file.

no sure it is relevant, probably all heavy lifting done in C++ and Rust https://searchfox.org/mozilla-central/search?q=render&path=&...

https://github.com/evolus/pencil

I think the article and most of the commentators are missing a real problem. The real problem is that there is an underlying complexity, which has no simple solution - collaboration of teams/groups of people with different expertise and even different interests is a universally unsolved problem. When a company passes the "startup" stage and gets real customers, it isn't a startup anymore. It has more tasks to handle and needs more people to work, people with different expertise. There are sales, operations, developers (with multiple expertise), lawyers, coordinators (of all kinds). Innovation is not a self-evident target, at some point startup needs to cash innovation. There are frameworks which attempted to address that complexity, for example scrum or more flexible superset "agile manifesto" or even waterfall. They all may work or fail, but there is no guarantee, no recipe.

I am using Java occasionally, there isn't any special learning curve if you know basic syntax. Same as in Python you don't start with Jdango and virtual environments, you shouldn't start using Java with Spring Boot/Spring.

Google library you need "java http client"

https://openjdk.org/groups/net/httpclient/intro.html - it doesn't look complicated.

First example is using "reactive-streams", you don't have to use that syntax if you aren't familiar (same as in Python there are new features, which beginners may learn later), other examples strait forward.

If library you need doesn't exist in standard library, download jar, add it your `classpath` and use.

Skip any tutorials which are using Spring (unless you want to learn Spring). If you want to write more advanced Java code, learn about [Maven](https://maven.apache.org/)

Same thing about JavaEE/JavaSE, if you google about it, you will find that you may use any, there are different libraries included in installation, it doesn't mean you can't add them later.

Term "toxic" is toxic by itself.

Periods when engineer don't understand the problem should be spent on analysis of the problem domain. "Now I am working on defining the problem domain" - is an activity to work "I don't understand the problem" task. During that period probably zero code will be written.

That author only encounters problems that have been fully solved before

He doesn't, otherwise there would be no talk about "plan B" and risks. When you actively write a project code, you should know that solution is possible. Having plan doesn't mean "problems have been fully solved before". You may have POC which doesn't end in resolution, but it should be clear what is POC for and a failure is possible outcome.

Talend Studio?

https://www.talend.com/products/talend-open-studio/

Apache Airflow is tool for *analytical data* pipelines. If you want to create a daily/monthly data feed from your operational DB, move to it into Data warehouse, run jobs to process/aggregate that data and create daily/weekly/monthly reports, Airflow may help with all those.

very basic use cases:

* run job for given reporting_period * backfill data for reporting_period between N and M * lookup failed jobs

Nothing in Databricks supports that.

he has "a wonderful secretary who looks at the incoming mail"

People like [Sam Walton](https://en.wikipedia.org/wiki/Sam_Walton) born in poor farmer family or [Jeff Bezos](https://en.wikipedia.org/wiki/Jeff_Bezos) son of 17 year old high school student mother and 19 years old father would have very low chances to succeed in not egalitarian societies. More talented people get opportunity to use their abilities, more Walmart's will be created.

do you have doubts those people are skilled? https://en.wikipedia.org/wiki/Brad_Fitzpatrick https://news.ycombinator.com/item?id=21727925

there is a strict mode https://sqlite.org/src/wiki?name=StrictMode

No Java

I am not sure what "essential tool" means if pandoc included in it. For me Intellij, Eclipse, Dbeaver are more essential (+ few other Specific tools based on Eclipse).

C# - I am not sure, MS Office?

Python Design Patterns

My brain puts it next to: globe of China, Jewish Olympic games (real thing).

I have mixed feeling about Nim. I've tried to learn it few years ago. It was "sold" like a Python replacement, it looks like a Python a bit, but when you write code it feels nothing like python at all.

Few things made me confused a bit

* With objects, when I need to use ref and when not.

    nim> type
    ....   Node = ref object
    ....      le, ri: Node
    ....      data: int
    ....
    ....
    nim> var n = Node(data: 9)
    nim> n.data
    9 == type int

    nim> type
    ....   NodeO = object
    ....     lw, ri: ref Node
    ....     data: int
    ....
    ....
    nim> var no = NodeO(data: 9)
    nim> no.data
    9 == type int
    nim>

That time development/coding wasn't my full time job, Python was the "to go" option of automating some tasks, sometimes Java. In both cases there was no magic in the code - `module.something` easy to track to the origin.

Nim's orientation to standard library is nice. I've started checking Nim when I need a faster alternative for parsing big json and csv (compressed and flat) files. I both cases to was easy to archive, easier than Java (with Java I need to match version and bring dependencies) and the speed was fantastic.

Probably the main reason I haven't continued with Nim, was the company's antivirus at some point started to blocks nimble.exe execution on Windows. Another reason is switching to work into data engineering aria and there is no real need for fast compiled program. IDE support wasn't great too, VScode suggested a lot of irrelevant things, but I think it is kind of broken for any language now (out of box).

Yes, that part is different, reusability of the code is not expected to be the same as normal software code. Not different that the code starts from requirements (which are different) and it should be correct, which basically is the "following normal development practices".

I don't like the code. My start point would be that

    import pandas as pd
    import numpy as np

    if __name__ == "__main__":
        n_samples = 10000
        samples_np = pd.DataFrame(np.random.randint(1, 7, n_samples), columns=["face_value"])
        print(samples_np.face_value.mean())

    if __name__ == "__main__":
        n_samples = 10000
        sample_pd = pd.DataFrame({'face_value': [1, 2, 3, 4, 5, 6]})
        print(sample_pd.sample(
            n=n_samples, 
            replace=True, 
            random_state=np.random.bit_generator.randbits(20)).face_value.mean())

Type annotation, I like the idea, but in the end code looks like Java, but doesn't performs like Java. It is very hard to make it right in Python, also some of them wrong.

( @dataclass(frozen=True): - don't need ":" Gaussian.sample - missing return )

when return added it doesn't return `-> Sequence[float]:`

    Gaussian().sample(90).dtype
    >>> dtype('float64')
 

I don't believe "scientific code" is fundamentally different from any other code, I would go with following normal development practices

1) review design ("don't reinvent wheel")

2) add tests

3) make code review

4) version control

etc.

For a bigger code base projects fixed typing contributes to speed of development. Add to it dependency management you will have clear win-win for something like Java over Python.

Talend is sold sort of well

(ex. https://help.talend.com/r/mjoDghHoMPI0yuyZ83a13Q/YYVSsDiyJ3v...)

What about running those apps on Android and iPhones?

Same here, I can use nim.exe on desktop, but nimble.exe is killed SentinelOne. It works on Linux machines. Before they were deleting everything named bash.exe (not anymore).

There is another side of it. Changing position for better working environment is unpredictable. With rare exceptions the job description and what was discussed during job interview is very different from the actual job I do after accepting an offer.

It is only a plan only (partially implemented). I am separating code to clean and ad-hoc. Clean code is "supported" - maintained (jobs monitored/failures handled/bug fixed) by more professional developers, if somebody what to have a custom job, they more or less on their own. When I am asked to fix problem in such "custom" job, first thing I do is refactoring code to follow standards (configuration, hardcoded paths and values, logging, alert notification to predefined list of people related to project, handling recovery, etc.), than it becomes a part of main pool - "maintained code".