datahead

What is the general outline of going from a model of a craft (drone, sailboat, etc) to building a sim that can do reinforcement learning over a physical object interaction with its env?

I want to start playing with models, sims and collected data for sailboat racing- I know the RL/data science stuff, and I assume a good model of your craft takes time to build, and can be improved with collected data. What are some areas to explore when chaining model -> sim -> RL for performance?

I realize this is an extremely complex topic, with several PhDs worth of potential input- if you had to explain to someone technical what it looks like and where to keep digging, what would you say?

oh, your luser account is 'letsdothisagain', great thannnnks. I'll take care of that right now.

Curious what recent work you are referencing? Lots of new so I don’t want to miss things. In my experimentation I basically wrote off LLMs from doing any math directly. That includes aggregations or precise calculations. I’ve been playing with sql generation and allowing the db to do its thing.

Hi holoway, I've been following your project for a couple of months now. How do you compare yourselves to upbound/crossplane?

IME crossplane has been a "much better terraform" and also borrows from some of tf's open source provider code. Seems to be one of the best IaC pattern for k8s centric shops.

Pros: - adoption of k8s core engine, state mgmt

- model everything as a CRD, consistent definition pattern both infra and app

- open source

- great UI when layered with argoCD

- declarative

Cons:

- steep abstraction learning curve (for me anyway)

- docs lacked key context for newbs (also getting way better, big efforts here)

edit: formatting

How did you approach testing this? Anything to watch out for when defining abstractions? I'm following Crossplane's abstraction model.

@cyberax said, "My problem with the current IAS systems is state storage. It should not be needed! Instead, the IAS tool should introspect the systems it's managing and build the necessary state on the fly.

@firesteelrain said, "you can do that through abstraction. You "include" your Terraform Azure Provider or Terraform AWS Provider. At the end of the day, your module needs to know what it’s interacting with but not the higher level of abstraction. We have done it at my work to make it cloud agnostic just in case we need to go to another CSP"

Single ops eng in a 3 person startup here. Ops eng is only one of my hats right now :) I found crossplane to be a solid tool for managing cloud inf. My assertion is that "the only multi-cloud is k8s" and crossplane's solution is "everything is a CRD". They have an extensive abstraction hierarchy over the base providers (GCP, TF, Azure, AWS, etc) so it's feasible to do what firesteelrain did. My client requirements span from- you must deploy into our tenant (could be any provider) to host this for us.

I can setup my particular pile of yaml and say - "deploy a k8s cluster, loadbalancers, ingress, deployments, service accounts (both provider and k8s), managed certs, backend configs, workload identity mgmt, IAP" in one shot. I use kustomize to stitch any new, isolated environment together. So far, it's been a help to have a single API style (k8s, yaml) to interact with and declaratively define everything. ArgoCD manages my deployments and provides great visibility to active yaml state and event logs.

I have not fully tested this across providers yet, but that's what crossplane promises with composite resource definitions, claims and compositions. I'm curious if any other crossplane users have feedback on what to expect when I go to abstract the next cloud provider.

cyberax's note on state management is what led me away from TF. You still have to manage state somewhere, and crossplane's idea was- k8s is already really good at knowing what exists and what should exist. Let k8s do it. I thought that was clever enough to go with it and I haven't been dissapointed so far.

The model extends the k8s ecosystem, and allows you to keep going even into things like db schema mgmt. Check out Atlas k8s operator for schema migrations- testing that next...

I also like that I can start very simple, everything about my app defined in one repo- then as systems scale I can easily pull out things like "networking" or "data pipeline" and have them operating in their own deployment repo. Everything has a common pattern for IAC. Witchcraft.

Back in my early days of data eng for ML pipelines, I stumbled onto Drake- and it opened my eyes to managing pipelines as DAGs. This pattern is supremely effective, and I try to teach anyone who might benefit.

https://www.factual.com/blog/introducing-drake-a-kind-of-mak...

Don't overlook the comment about form. Form is everything, weight is secondary. Get a professional trainer to help you setup your form once you move to barbell work and then you can go from there on your own.

r/homegym will have some good recs for gear to get started. Plenty of bodyweight training you can do too- great for turning on your main muscle groups (youtube).

A barbell and some plates w/ rack will cover the main lifts. Squats, deadlifts, presses (chest and shoulder) are the first ones to build. I recommend Starting Strength for an on-ramp to get going. Adjustable dumbbells are great too- plenty of options out there. Some of this equipment is expensive, look for used and consider it an investment in health.

Happy Lifting!

I did as well, and was rooting for him.

One of my favorite lessons "learned the hard way" by someone else. You fundamentally cannot run a business on poor data practices. It will sink you before you realize what happened.

https://www.canadianbusiness.com/the-last-days-of-target-can...

I'm curious if our interactions could be biased. For example, we're clients of RH (RHEL & Openshift) so I'm guessing we interact with more presentable humans. Everyone I've met has been genuinely wonderful and supportive across eng, sales, product, TAM, etc. One of the few vendors I enjoy working with.

I used inclusive in this context to mean that a junior engineer is expected to speak up and offer input in an executive meeting if they have something to offer. Cutting through hierarchy, "good ideas come from everywhere". This could be idealistic but it helped me in eliciting input.

I don't know what it's like to be on the inside or close periphery to daily core operations at RH. I agree and understand completely what you are saying. "Free speech" has (recently?) been a guise to act terribly toward each other in a number of settings. Trolls gonna troll. It's important to maintain decorum in the workplace and a culture that defies that can turn toxic quickly.

I like Whitehurst and have had the chance to meet him a few times. If you want to understand his ethos check out "The Open Organization". It details his transition from Delta to Red Hat and the transformation of his leadership style being around a lot of smart engineers and team players ready to constructively challenge each other. Good read for anyone interested in building inclusive company culture in tech.

Honestly, I thought IBM would be feel less rudderless with him at the helm. I wanted IBM to become more 'red' and not RH to become more 'big blue'- but it appears this isn't the case after all... seems like a short tenure.

Anyone know more about the new leadership?

I support getting the sampling rate and quality up. To what level? Idk.

Have you ever been in a room with a DJ playing low quality music on a decent sound system? It's awful, grating. Now at least your everyday aspiring DJ has access to higher bitrate tunes and doesn't need to blast low quality junk out of the PA system.

Overnight on the USS Yorktown (CV-10) was awesome, one of the highlights of all the scouting trips we took. It felt like we had full run of the ship, and my buddies and I explored quite a bit. My first impressions of WWII history and Battle of Midway where USS Yorktown (CV-5) sank were from the tours and videos shown that day.

First, if you haven't worked in a large "non-tech" enterprise, you might be surprised to find out it's just 100 startups with a common name and financing. The bigger the company, the less technical cohesion across teams. It's a blessing and a curse. I'm technically in a "business unit" as opposed to IT, so we're more focused on tangible KPIs. When you turn around innovative results in the biz domain consistently, no one really questions your tooling.

We started a new space Data Eng/ Data Sci and then it became critical. We've been able to maintain products by building long term platforms/systems that meet current and (anticipated) future needs in clojure. Initially, product and platform/system was all the same team. We've grown a little and now individuals tend to focus more on one or the other, but we're still <15 eng and shipping daily.

We maintain everything we've built to date, and we planned for that going in. Clojure's other superpower is that it is stable AF over long periods of time, handles refactors and testing well.

As for signoff in adopting clojure... our VP is old hat in clojure/lisp. We got lucky.

Great article. Even in a large enterprise, where the prevailing organizational ideology centers around building the largest headcount you can for political sway...

I've kept my team small on purpose, and we were able to be very effective by adopting clojure. We have sway because we deliver- and that's a different type of leverage than headcount. We don't get thrown every hot potato, instead we're consistently aligned with the critical portfolios.

A small dedicated team with tools such as clojure will outdeliver, and outcompete a larger team who cannot remain agile and require significant overhead to manage.

I'm basically rehashing PG's 'Beating the Averages', but it's been my experience as a dev mgr. http://www.paulgraham.com/avg.html

I found this [1] breakdown helpful to understand the legal position of this hypothetical.

unlimited [...] discussion, planning and coordination

turns into evidence once "intent" and an "overt act" thresholds are crossed, for all involved.

[1] https://criminal.findlaw.com/criminal-charges/conspiracy.htm...

Interesting presentation that explores your question. +1 intersection from this commenter.

Tim Ewald - Clojure: Programming with Hand Tools https://www.youtube.com/watch?v=ShEez0JkOFw

Hacking Healthcare (HC specific) and The DevOps Handbook (general) are the books I buy for everyone joining the team.

I recommend Eric Topol's books as well, including The Creative Destruction of Medicine and Deep Medicine. Lots of food for thought.

Hi Duff! Apologies for the mixup, should have seen that in your handle. No, I'm not part of CHL/TXR.

While not open source by any stretch, I see the personal EHR space being ushered along by companies following Apple's lead. Aside from complex patients, I think the generally healthy/mild-chronic person is uninterested in owning/managing their health data unfortunately. Apple is contributing useful tools to understand fundamental determinants of health including cardiovascular, sleep, fitness at massive scale. It just happens to come with your watch and phone, and their vision for health is starting to come into focus. This puts the patient in the position of generating the primary data (sensors, etc.) and sharing it with their care team on their terms (more or less). As telehealth becomes more prominent, I suspect the patient will be required to engage with their data more often as it will be the means of conveying a shared understanding vs. observations recorded in the clinical setting and stashed away in centralized EHRs.

Furthermore, if labs and other diagnostics are available directly to consumers it puts the individual in a position of ownership. I think the default position is whoever generates the data owns it, and determines how easy/hard it is to share with others. If the individual is empowered to generate information about themselves- this will start to swing toward "patient owned." I too look forward to more of this, but it will have to come with more direct to consumer and digital offerings.

One of the coolest examples I've seen of individuals taking ownership in open source med tech is openaps.org . I'm not one, but T1D's are some of the most resourceful and resilient folks around. Good on them for building a community to solve real problems together. Shout out to the #wearenotwaiting crew.

Hi Duff! Happy to see you on HN. EDIT: not Fred.

I work for a large hosp. operations company and serve as the Dir. Engineering for our clinical operations group. Hacking Healthcare is required reading for new members of my team. It serves as an excellent introduction (with a healthy amount of critique) to the dynamics in the hc technology ecosystem. Thank you for providing this perspective on the industry and its challenges with tech.

We've been successful developing using open source technology internally. In fact, I take a fairly hard stance on disallowing proprietary healthcare specific "solutions" from working their way into our stack (aside from the EHR itself, it has staying power). We're lucky in that we are positioned as somewhat of a startup within a larger org, and are able to take that approach.

To avoid some of the issues you raise, we generally are working to reduce the surface area of the EHR to become simply the transactional backend which is then mirrored to a larger ecosystem of custom apps. This has the effect of boxing in the regulated entity. We focus on data integration (by spending $$$$ on custom HL7 interfaces, unfortunately not everyone can afford) to get outside of the walled garden. This means we can use the information/data for new and interesting purposes without worrying about the EHR vendor's roadblocks/tolls. More importantly to some people, we don't disrupt the billing cycle that originates from the EHR.

Do you notice any trends where healthcare operations/providers are starting to develop internal technology that integrates with the EHR to compliment vs. replace the core transactional system?

For anyone interested in background on Datomic, these are fun and informative resources.

The talk where I realized Rich is a data head too. Introducing his design goals and life experiences with db(s). https://www.infoq.com/presentations/The-Design-of-Datomic

"Clojure for the Brave and True" author, Dan Higginbotham, on key themes and overview. http://www.flyingmachinestudios.com/programming/datomic-for-...

Congrats to the Datomic team and Cognitect. I hope this move to cloud opens Datomic's design principals and ideas to a wider user base.

Our team signed a contract with Cognitect for Datomic late 2017.

This clause was in-place and stood out to me as well. I had a chance to ask their legal team about it. The clause is written in legal-ese, which always sounds overbearing.

I asked the question in the positive sense, "what if have some really nice metrics from our use cases, and want to talk about them at a conference?" They simply asked to be consulted and request written permission to share. The intent, like others have noted, is to request (legally: insist) that Cognitect have a chance to review and point out potential implementation issues (good or bad) prior to customers making performance statements about their product.

The clause can/does put a damper on 'notes from the field' reports, which often help when deciding on tech direction.I look for community based reports to reinforce perceptions of a tool (to a degree). Completely agree with OP, do your own performance testing.

One thing I will say is that it would be hard for someone who hasn't invested in learning the inner-workings of Datomic's decoupled architecture to pick apart storage speed vs. transactor speed. For example, storage speed (SQL, DEV, Dynamo, etc.) is not a concern of Datomic, but a key dependency to measurable perf. This may change in the AWS service announced today, and become more uniform on dynamo and S3 "storage resources". https://docs.datomic.com/cloud/whatis/architecture.html#stor...

Datomic is a unique product and there are many ways to make it sing (or blow up) depending on how you use it. We designed data models, streaming processes, and queries with Datomic in mind and have had success. Exactly how much success, I'm not at liberty to say just yet.

Solar_Fields and Bigger_Cheese, my apologies for not specifying the db. In this case I'm talking about a production level data warehouse specifically designed to perform complex queries and not a prod app db. Yes, I agree- best practice is to separate the two. Glad you found the article interesting.

This technique applies to many RDBMS, not just Oracle (as others have noted). Teradata, PostgreSQL, MS SQL all have 'analytical' functions like this. Analytical functions (over, partition by, etc) are extremely powerful and can help simplify architecture/design for the data science/analytical communities.

One of the persistent issues on my team is the reliance upon a dataframe representation w/ R or python to do this type of aggregation and windowing. Most people will eschew learning the 'advanced' SQL and instead bring data locally to do imperative style munging on it.

This creates a few issues, mainly adding complexity to the analytical stack: - Instead of querying the data and doing ETL/feature engineering in the db- you are moving data around (usually to less powerful machines, such as a laptop) for simple exploration.

- This wastes time and usually results in more dependencies (dplyr for example- no hate Hadley), sometimes even limiting you to single threaded operations. Teradata, for example, is massively parallel and will perform these operations in short order. I've seen Data Scientists wait 6hr for R to do the same thing a SQL query against a prod system returns in 3min.

- Code is not portable. A query can be executed and results retrieved through ODBC, JDBC or native connections. Without these, data engineers are often asked to install R (including libs) on some intermediate machine just to do munging/ETL/feature engineering. If SQL driven, moving from quantitative exploration to operational is quite easy (maybe just a query tune).

All that to say, I'm glad this post is highlighting some of the advanced SQL that I hope more people rely upon. All of these ideas are better articulated in MAD Skills [0]

[0] http://db.cs.berkeley.edu/papers/vldb09-madskills.pdf

Interesting that they cost about the same {"the-gun": 4750 , "noahlytics": 4800} + subscription @ $100/mo. Possibly a competitive pricing scenario for high school program adoption.

I also see a key feature being the tracking of performance over multiple sessions. Noahlytics will likely answer: who is shooting the best this week? May impact starting line-ups for coaches, etc.

Go Bruce! Figwheel is awesome.