liyanchang

See also https://99percentinvisible.org/episode/invisible-women/

"Car crash test dummies are also generally male, based on an average man... and ignores anatomical differences, plus specific individual circumstances like a person being pregnant"

There's lot of places where technology is tilted in unintentional ways

Yeah. I’d love to see the academic paper with formalizations that help me understand the true scope of this problem. Your example is a great one that prompts many questions. Is it possible to travel directly to the commit o(1) or will the code have to calculate the diff of each commit and apply them one at a time o(n) and how much definition and dependency mapping humans need to do to have it work correctly?

I’ll openly admit that we don’t have everything figured out. You’re absolutely right that currently, we constrain ourselves on what we migrate to admittedly simple migrations.

I think there’s a whole set of problems to be solved in this space and frankly, I’m as surprised as anyone that given how SQL is declarative, we use procedural code to do migrations so part of my post was hoping people would tell me what tool I should be using or how this approach fails over time. So your examples are really helpful for me as I think through if it’s possible to do automatically, workaround, or get by without.

It seems to me that we just lack the ability to express these transitions mathematically that can help us do them. And of those, there’s probably only a subset which are possible to do without taking downtime.

In particular, the class of migrations that you outlines are a combination of DDL and DML changes and also have quite a bit of code complexity to do without downtime. It’s definitely a current weakness.

I’ll openly admit that we don’t have everything ironed out. In fact my next big project is to tackle derived columns (rename is a column where the transformation is the identity function).

It requires a bit more finesse and integration into our code base as it requires multiple deploys with code that knows how to handle both columns.

That's correct. The DML changes as part of CI are somewhat new so we haven't ironed it all out yet.

Here's the scenario that I think you're laying out: 1. Commit A creates column foo 2. Commit B has DML that reference column foo 3. Commit C removes column foo

This works fine if our CI deployer does each commit individually. First roll out any schema changes, then run any DML SQL.

However, our deployer might pick up all those changes and since we roll out the schema migrations first (in this case a create + drop -> NOP) and then runs the DML (which will error), this is an issue because of the rollup.

In practice, we have yet to see this case (most of the time, the dev who write the DML is close enough to the code to know if it's going to be dropped soon and we don't drop that many columns - in part because we know that there be dragons) but truthfully, I haven't thought about it much and need to think through what the impact is beyond this example. Thanks for helping me refine my thinking and I'll have something to ponder on this weekend!

This is good to know. As someone who didn't do much with databases before, I was frankly worried given how it didn't seem like many others were taking this approach when it made so much sense (we did have the advantage of having a defined schema which I know isn't always available). Seems like I just didn't know what to search for.

Git would never have worked it required devs to write the up/down patches - why should we have to write the up/down migrations for my schema?

Excited to see more tooling around declarative schema!

I've been really happy with how my current company[0] has been doing migrations and I've seen a couple others do it but it seems like it should be more widespread.

Database Schema as Code

Instead of writing up and down migrations, you define what the end state should look like. Then the computer will figure out how to get here. This is just how the industry started managing server configurations (Puppet) and infrastructure (Terraform).

We use protocol buffers so it was pretty straight forward to have a definition of what our tables should look like. We have a script that figures out what the delta is between two states (either proto files or a db) and can calculate the schema migration SQL (e.g. CREATE TABLE, etc).

From there, we run it through a safety check. Any unsafe migration (either for data loss or performance issues e.g. DROP TABLE) requires an extra approval file.

There's no real difference between an up migration and a down migration (except that one tends to result in an unsafe migrations). It's calculable at CI time so we can give devs a chance to look at what it's going to do and approve any unsafe migrations. API compatability checks enforce that you need to deprecate before you can drop.

DML, that is data changes, are handled via standard check in a sql file and CI will run it before the code deploy and after the schema migration.

Alembic is the one other place I've seen this concept (a couple others mentioned this) so it's not new, but surprised I haven't seen it more places.

[0] Shameless plug: We're hiring if you're interested in changing how healthcare is paid for, delivered, and experienced. https://www.devoted.com/about/tech-jobs/

Devoted Health | Boston, MA | Software Engineers, SREs, Designers, Data Scientists, Head of Corp IT | Full Time | ONSITE | https://www.devoted.com/

We're building a better and more human health insurance company to improve the lives of seniors in America

We are guided by a deep belief that every senior should be treated like we would treat a member of our own family: with loving care and a profound commitment to their health and well-being through world-class technology and customer service.

Stack: Go, Python, Postgres, GraphQL, gRPC, React, TypeScript, Docker/Kubernetes, Terraform, AWS

Disclosure: I'm an engineer at USDS and these are my own opinions.

I'm personally appreciative of the work that you've done and your continued work, in spite of the many challenges and frustrations. There are many ways to serve. It's always incredibly rewarding to be working along side dedicated and talented public servants and contractors. Thank you.

Thanks. I appreciate that. I think you actually have a keen eye for the issues.

Since my time in government, I've become more sympathetic to agency employees, contractors, management - pretty much everyone. The entire environment is really challenging to do the right thing.

As an executive, the typical tools you have at your disposal, budget and HR, are circumscribed by congressional appropriations and federal hiring guidelines. Your ability to promote and demote and reorganize are limited by laws and unions and you definitely can't grant stock options. The metrics that you typically use to measure success are much fuzzier because there isn't a PNL but it's really easy for the press to find a single case where your org does a poor job.

I still maintain hope that good things can be done - I have seen it - and good management makes a huge difference.

Disclosure: I'm an engineer at USDS and these are my own opinions.

Disclosure: I'm an engineer at USDS and these are my own opinions.

We are very proud to be working along side the existing agency and contracting staff. You are right that many times, they do know how to fix it and need some help from us getting management to let them do it. Some times they are skeptical, sometimes there is friction - that's all completely understandable.

Ultimately, my time in government has opened my eyes to the talented folks inside government right now and I'm proud to be working along side them. I hope you didn't get a poor impression of us and the work that is happening.

Disclosure: I'm an engineer at USDS and these are my own opinions.

I absolutely agree that the interface that most citizens use to government can be better. President Obama agrees with you (from his sxsw remarks):

"I could change the politics of America faster than just about anything if I could just take control of all the DMVs in the country."

"If their primary interaction with government is the IRS, you just don’t have a good association with government when you’re writing that check."

USDS has been working with the IRS to help them get their Transcript service back online with better identity proofing though admittedly just a small step in a better online experience. (irs.gov/transcript)

A number of state and local governments have been making improvements as well (much harder to see if your state or your local government has not been). I'd encourage you to search them out or even start it.

Disclosure: I'm an engineer at USDS and these are my own opinions.

We hire for experience, which comes in all sorts of shapes and sizes. Based on my personal observations, the team is actually the most diverse team I've ever worked on by a number of dimensions. We've pulled folks from retirement, others who are chronologically younger but have been contributing to Python and Debian for years, those who have uprooted their family. The job is an incredible mix - it requires being able to debug a slow performing database index one moment, briefing a deputy secretary the next, all while empowering the other federal employees and contractors to do great things.

I would be remiss not to take this opportunity to link to our hiring page. https://www.usds.gov/join

I do recognize that there are lots of talented folks that aren't able to move to DC or can't make it work for a variety of reasons. That's okay. There are many other ways to contribute to our country [0] - working for state/local government, volunteering in your community, being a good parent, inventing the next breakthrough, actually using your turn signals, and many more.

[0] Yes I recognize that not everyone here is US based. Though the point actually is probably true for all countries.

Disclosure: I'm an engineer at USDS and these are my own opinions.

That's right. The US Government is probably not going to win on compensation - salary limits, no stock options, no lunch. However, while it's not a money making enterprise, it's enough to do just fine. I recognize the sacrifices that many of my colleagues have made to be here, not to mention the sacrifices that federal employees and contractors have made (some who are very good and could be making more in the private sector) and that makes this even more worthwhile.

The thing that government can offer is impact. I've always known that government has a big impact on people's lives but not sure if I could personally make an impact. That's what USDS, 18F, a number of other opportunities are offering.

This is not a job for everyone. But if you're a certain type, there's nothing quite like it.

Disclosure: I'm an engineer at USDS and these are my own opinions.

You make a great point - USDS is just one part of the solution. In fact, in almost all of the USDS projects, we work very closely with agency employees and contractors (many of which are just as talented and have chosen to serve their country). Most of the times, I spend very little time hands-on-keyboard and helping empower the existing team.

As for the longer term solution, there is a less publicized version of what folks are doing. USDS has a number of contracting officers who are helping teach others in government how to be savvy customers of technology. 18F and GSA have been doing a lot on this front as well to help bring in really good contractors and writing agile purchasing agreements. The Office of Federal CIO is rewriting and simplifying tech policy and the Office of Federal Procurement Policy has been working on the procurement side. These are the long term changes that I'm personally excited about.

Disclosure: I'm an engineer at USDS and these are my own opinions.

So in my admittedly short time in the government [0], I've witnessed how all of these problems are due to good intentions. That's what makes this all really tough because everything you think is bonkers actually has a reason.

The 1400 page travel regulations is a result of trying to prevent fraud - every single issue that comes up results in a new rule.

The fact that it takes some projects years to deploy is that we would like to plan and make sure that every resource is well-spent, that it's in a number of languages and accessible to the blind.

It makes it hard for everyone - I've met lots of smart talented civil servants and government contractors who want to do things differently but have their hands tied behind their back.

[0] 2 years feels like forever to me but flash in the pan to many of the dedicated civil servants I've met.

Hi Nick,

Agreed! There are lots of places where this tool could be improved! (Disclaimer: I work with people at 18F, Dept. Of Ed, USDS who built this tool, but I wasn't involved and don't have any privileged information and so here are some of my personal thoughts)

If I can add a bit of color:

1. This was developed open-source on GitHub. That means you're welcome to file an issue or comment at https://github.com/18F/college-choice/issues (Given the interest and number of issues filed so far, I'll probably take this opportunity to say that it might be a while before the team gets back to you and appreciate your help and patience!). This shouldn't impress you, but it is, because it's a complete paradigm shift from how government typically works.

2. Average salary is not a great metric and unfortunately, as far as I can tell, the data isn't sliced by major. You're right in your analysis of how it skews the data. I actually think loan repayment is probably the more interesting metric - have others in your income class been able to repay their debts? You're also right that this tool doesn't help people see that choosing college A over B because of a $X,000 delta in average salary isn't a good idea as it's subject to all sorts of personal decisions.

3. This is really cool not because the tool is perfect, but because the government is releasing an API to go along with this - a number of private companies and non-profits have already integrated the data into their tools. This data, which hasn't been possible to get before, can now be integrated into the tools that were helpful to you (the ones that calculate in-state tuition, and ask you for sat/act scores). The underlying API also has many more data fields than the tool exposes. If fact, you can build one yourself! (https://api.data.gov/signup and https://collegescorecard.ed.gov/data/documentation/)

I am not an accountant - I was looking most of these up so do let me know if I got something wrong and I'll edit as needed. Just trying to compile things in one place.

Gross Merchandise Value is how much money flows through your system while Revenue is how much lands in your bank account. For instance, a payments processor like Stripe might have a GMV of $100 million while their revenue would only be the 3% commission (in this case $3 million).

A contract is a legally binding and enforceable document. A letter of intent is when one party outlines what they are likely or would like to do - with some bits of it being enforceable like non-disclosure agreements. A memorandum of understanding is a letter of intent signed by all parties involved - it is still non-binding. A term sheet from a VC is like an LOI - however, it doesn't actually happen until after due diligence, negotiation, etc and only official when signed.

Burn rate is the delta in your bank account. Expenses is how much money left your bank account and revenue is how much entered. Thus burn rate is expenses - revenue and is -1 * profit.

Users are people on your site. Customers are paying users.

Signups are how many people created an account. Active users are how many people logged in over a certain period of time.

Cash flow positive means you have more in your bank account than you did before. However, a kickstarter which raised 1 million would be cash flow positive but not be profitable as it has many outstanding obligations.

Churn rate is the percentage of your users/customers who left over a certain duration. Retention is 1-churn.

Involuntary churn is when the customer leaves because they go out of business or in the case of dating apps, no longer need your services. Voluntary churn is all other churn.

Gross refers revenue - expenses of the product. Net is revenue - expense of the product - administrative costs - depreciation - payroll taxes etc.

Top line is referring to gross while bottom line refers to net. Top line growth means more revenue and bottom line growth means cost cutting.

Oh! Thanks for letting me know. Had no idea about that particular team. Will happily give you both credit next time!

Pensions have a 3 year cliff - but you're only allowed to work for 2 as a schedule A and you don't accrue years as an IC so one can conclude with very high confidence that I will not have a pension :D.

Hi Andrew. I'm really sorry that you went through that. We know and acknowledge that our hiring process isn't perfect - I can't count the number of conversations where we at USDS share stories about how we personally were in some limbo before interviews or what we had to do to get drug tested or how we were super confused.

This of course is no excuse - there is a lot of work to be done and we are working on it.

Thanks for the kind words. Yes, we have made progress. Yes, there are still quirks. Yes, there is more to do. Come help!

As a YC alum, while I find some strong similarities between Digital Service and a small growing startup, there are indeed some additional hoops that we go through as a part of the US government and that's part of the tradeoff you get for being able to help government agencies that have huge impact!

It sounds high to me too and I'm was on the team that rebuilt the login system. From what I know about it - we knew how much it cost us[0] and didn't have the support and maintenance contract figured out yet so we reported out 4 to build and no more than 4 to maintain and the number has kind of stuck.

I would say that I (very naively) thought that it would be easy peasy to rebuild all of healthcare.gov. I was wrong. There are all sorts things that make it much harder than you'd expect in a private sector environment.

I do think that there are probably ways to get this cost even cheaper - however, from a software profiling perspective, moving it down an order of magnitude or two is a huge win and that it's a pretty clear choice between further optimization compared to an order of magnitude change on a VA, immigration, or other system.

[0] I'm not actually sure how exact this number is either - lots of help required from people from all different contracts - so this is an upper bound estimate

I'm not certain about the exact intricacies of the Schedule A hiring authority so I may be corrected by someone later - but my understanding is that this would not be possible to be rehired under the same hiring authority. People have joked about doing both a Schedule A and an Intermittent Consultant[0], which would allow you to do 4 years (2x2) - as we've just hit our 1st birthday, no one has tried it yet!

[0] There are some slight differences in how pay, retirement, and some other benefits are distributed, so they are mostly but not entirely interchangeable.

Sharp eye! We are indeed indebted to the work that the UK has done! They are a couple years more senior (USDS just had it's 1st birthday this Tuesday; GDS is coming up on 3) and while there is much tailoring for each country's unique circumstances, we've certainly benefited from their experience.

Yup. I want to give a shout-out to 18F who did a really great job with this. This project is really important for two reasons.

1. It brought analytics to the minds of agency communications and policy folks. Having 18F going around adding Google Analytics to as many government websites as possible also opened up lots of other opportunities to help people better understand their users.

2. It's an example of a shared service - one that doesn't require extra procurement, is super reliable, and doesn't require every agency to duplicate work. The sheer size of the government makes the fact that sometimes the left hand doesn't know what the right hand is doing is understandable - the way we've set up a federal system actually makes this an intentional feature. However, if we reduce rebuilding, either by sharing services or code would be a big win for the budget and for quality.

I'm currently at the US Digital Service so figured I'd add some color. There are many subtleties about how government hires that makes this a really interesting question.

We are brought in either under Intermittent Consultants or Schedule A hires. This means that in exchange for being able to hire you by name rather than going through a more structured process through USAjobs.gov, you can only work for 2 years. Therefore, no matter what the pay, it's probably not a viable long term career.

It's also important to note that this is probably a good thing - I've been at Healthcare.gov and USDS since Jan of 2014 and I'm finding that I'm getting adjusted to some of the bureaucracy. It's therefore actually key to balance experience with fresh perspectives.

I'd also say that the salary ranges - it's in part based on what you used to make in the private sector but has a hard ceiling. For everyone who comes, it's more than enough to survive but not a long term career move for anyone. And of course, there are no stock options or equity stakes :D.

For instance, they could easily jam or run out of product. They could erroneously dispense several bank notes instead of just one—all without the owner's knowledge.

I think this impresses me the most about ATMs. I've never personally experienced an ATM dispensing incorrectly. In fact, it wasn't till many years and many ATM transactions before it even occurred to me that mistakes could be made.

I presume they don't happen very often because I have no idea what I would do if something did happen? Who do I talk to if it debits 100 from my account but only dispenses 80? What if it's a bank error in my favor? Is it mine to keep?

Unrelated - I now use my phone to deposit checks rather then the ATM. Unrelated 2 - I now use ATMs to do currency exchange rather then dedicated currency shops.

I was surprised by how few OS/Browser combinations were tested. For instance, I'd think that the OSX + Chrome population is pretty big and I'd (selfishly) like that combination tested.

It feels like, just the fact that this competition and other bug bounty programs exist, means that the big companies here have gotten over reputation tarnish and know that the patch is worth it.

The group split was something that happened after S12, which was at the time the largest batch[0]. As the numbers of startups scaled, connecting all partners to all startups was order o(n^2).

I was in this batch - there didn't seem to be any negative impact on the companies given that most/all the founders figured this out and naturally distributed themselves to optimize for talking to the same couple partners week on week. It just meant that every once in a while, you'd be sitting across from a partner that you've never really interacted with and explaining what you do (probably good practice)[1]. Formally grouping partners sounds like a great scaling patch.

[1] Not sure exactly how they did it, but the partners had some internal communication. You'd be halfway through your pitch and then they'd be like "Oh, you're THOSE guys who did X, Y, Z". And then proceed to still give you great advice.