HN user
http://twitter.com/p_e
An often overlooked framework used by NASA among others is Kedro https://github.com/kedro-org/kedro. Kedro is probably the simplest set of abstractions for building pipelines but it doesn't attempt to kill Airflow. It even has an Airflow plugin that allows it to be used as a DSL for building Airflow pipelines or plug into whichever production orchestration system is needed.
Exactly - this release train metaphor that Spotify uses explains this visually:
https://medium.com/pm101/spotify-squad-framework-part-i-8f74...
One of the reasons smart contracts work so well is that they retain some of the properties of libraries yet are monetizable.
If you are feeling overwhelmed with yet another machine learning pipeline automation framework, you should check out Kedro (https://github.com/quantumblacklabs/kedro).
Kedro has the simplest, leanest, functional-programming inspired pipeline definition and also spits out AirFlow and other formats readily + comes in with an integrated visualisation framework which is stunning & effective.
Anyone else came here expecting a Haskell post?
Ditto.
You should try http://roamresearch.com/ and follow #RoamCult on Twitter - they are way ahead on note taking.
The author is forgetting that Google is a search monopoly.
AI was (and continues to be) the biggest threat to their success in search so they invested heavily in that (it was called ML back then).
Clearly they are still doing well even with subpar cloud software, so this has proven out.
This seems to lend more credit to ReasonML. Compile times are faster and it doesn't bring the associated complexity of object types while still retaining the benefits of static typing.
Paywalled.
Tell them you're looking to join a company precisely to get mentorship, see how a successful company operates, learn to work in larger product lines, etc. That way you're pre-emptively going around the Founder damage issue.
One way to get around this is to use Kedro https://github.com/quantumblacklabs/kedro, which is the most minimal possible pipeline interface, yet allows you to export to other pipeline formats and/or build your own exporters.
Airflow is an incredibly powerful framework to use in production, but a little unweildy for anything else.
You can use something like Kedro (https://github.com/quantumblacklabs/kedro) to get started building pipelines with pure Python functions. Kedro has its own pipeline visualiser and also has an Airflow plugin that can automatically help you generate airflow pipelines from Kedro pipelines.
Use Roam Research.
I had a similar problem until I knew I had to work on a start-up and pick a market to go help. It took me ~2 years to choose that market but I haven't looked back (crypto developer tools).
Look back, not forward.
Take all the things you have spent the past 5-10 years being interested about. For me, it was coding, programming languages, developer tools, design & minimalism, products, B2B, mathematics, economics, investing, productivity. Crypto dev tools is the intersection of those - a new field allowing me to keep doing the things I'm already provably interested in rather than making bets about the future.
Side note: the best way to build Airflow pipelines is through Kedro https://github.com/quantumblacklabs/kedro-airflow.
Haskell is not boring.
Kedro puts emphasis on seamless transition to prod without jeopardizing work in experimentation stage:
- pipeline syntax is absolutely minimal (even supporting lambdas for simple transitions), inspired by the Clojure library core.graph https://github.com/plumatic/plumbing
- sequential and parallel runners are built-in (don't have to rely on Airflow)
- io provides wrappers for existing familiar data sources, but directly borrows arguments from Pandas, Spark APIs so no new API to learn
- flexibility in the sense you could rip out anything, for example, the whole Data Catalog replacing with another mechanism for data access like Haxl
- there's a project template which serves as a framework with built-in conventions from 50+ analytics engagements
5-6, then iterate.
I'm interested in trying Rascal, but it's meta-meta: https://www.rascal-mpl.org/.
MUJI Paraglider Cloth Foldable Rucksack (packs in my suitcase for travel).
A world class engineer
- does their part to grow the company by 5% every week
- improves the system that builds the product by 2-3% every week
This means different things at different companies.
I used to work as a management consultant in McKinsey and did several pricing engagements.
The Firm in question is almost exclusively focused on pricing projects and "price before product" framing is probably helpful in convincing potential clients. Take with a grain of salt.
I'm going to take a very different stance here from others.
First, you should ask yourself - what kind of engineer do you want to be. Why are you building?
Then, read about the people who became the best and their early days. What did they do when they were in your position? Did they study a set of topics or did they pursue their passions? Taking and highly-weighing advice from people not at the top of the field could be very detrimental to your development as an engineer.
From reading books like: Coders at work, Masters of Doom, Making of Karateka, you will realize that the best engineers were driven by their ideas and projects very early on. They ended up learning to make things happen. But don't take my word for it, find these people, study their stories and pick the path you would enjoy the most.
This approach will not just make you a strong career engineer, but could turn you into one of the greats - the people that get to push the field forward.
Can you talk about how your product is different from Databricks MLFlow?
The test is whether the core YC aspects can scale?
* How can we motivate people to build on a weekly basis?
* Can we be a credible source of advice so people can save time by not second-guessing it?
* Can we identify which problems need resolving and intervene at depth?
* Can we connect the community through this shared experience?
Lots of assumptions to test.
The first post is on "why I studied Confucian philosophy in my undergrad".
They nailed content.
I agree that most apps might not be saving lives. I chose YPlan because the Founders used a rigorous strategy of looking for problems rather than looking for solutions.
This is beyond the post, but I agree that value should land where it can deliver the biggest impact. However, making money writing apps for 20-somethings isn't necessarily a bad thing, what matters is where that money goes afterwards. More on that in http://80000hours.com/.
This is false. I studied Maths at Cambridge and learned some courses completely on my own.
Ah, yes, iGoogle and myYahoo are cases of home page dashboards. Nobody would really use those for financial analysis and the widgets are generally skeumorphic and distinct looking. I would say it's not a strong enough execution of the idea.
I agree it would not necessarily be a killer product for Yahoo. It would, however, allow Yahoo to streamline and combine some existing efforts.