“A Cold War Every Day” inside Apple's internal tools group

https://www.buzzfeednews.com/article/alexkantrowitz/always-day-one-exclusive-excerpt-apple

by ttepasse • 6 years ago

334 174 6 years ago

rkho3• 6 years ago

A while ago I interviewed for a senior fullstack role with a very homogenous team at Apple -- every single person on the team had been converted from the same consulting company (I believe it was Wipro).

It was a bizarre interview with three of the engineers on the team on the same call asking me very specific programming language trivia and nothing about my experience nor ability to navigate complex political structures -- you know, the things that you would expect a senior fullstack engineer to have before advancing them to the next round.

Unsurprisingly, I wasn't advanced to the next round. In light of this article, I can't help but wonder if it was all a ruse to simply demand another nepotistic FTE conversion from their same consultancy.

timwaagh3• 6 years ago

Perhaps, but perhaps the role just wasn't fit for you. A lot of 'senior dev' roles are much more about programming language trivia and much less about leadership.

rkho3• 6 years ago

That could be the case, I just find it difficult to believe that an ICT3 role at Apple of all companies would be about programming language trivia, given that the specific team I spoke with mentioned how they were very cross-collaborative with other LOBs.

voz_4• 6 years ago

A lot of 'senior dev' roles are much more about programming language trivia and much less about leadership.

I am curious - where do you work?

Asking so that I can potentially avoid that company, as this is the most inane and borderline backwards thing I've ever seen written about senior engineering.

mabbo• 6 years ago

I can't tell if you're being sarcastic, or if you live in some kind of Kafkaesque dystopia I never want to be part of.

At my employer, the best senior developers are the ones who know they can google the answer to trivia so they don't need to remember it.

gwbas1c• 6 years ago

Some people just don't know how to run an interview

seemslegit4• 6 years ago

Is there any big consumer product company out there that doesn't deprioritize management and engineering talent for internal tools and systems ?

dcolkitt• 6 years ago

It's an interesting question. And if the answer is no, I guess the logical conclusion is that internal tools make very little difference to the success of consumer product companies.

That doesn't seem like an intuitive conclusion, but it's a hard circle to square. As someone who mostly builds internal tools (though in a very different context), it certainly seems like good systems are an Archimedean lever that acts as a force multiplier across the entire org. Yet, if good internal tooling was important, you think at least one company would have been able to achieve success partially on this basis.

The consumer product industry is extremely competitive, so the counter-hypothesis of entrenched sclerotic incumbents refusing to pick up $20 bills off the sidewalk doesn't really seem likely.

seemslegit4• 6 years ago

Internal tools are used by employees who unlike consumers will tolerate more usability friction, non-critical bugs and feature delays than consumers and will compensate for them with their own effort so the tools can still be critical in their core functionality but not polished or well-maintained.

Ironically and anecdotally, WeWork tried to build a strong engineering org for their internal systems before it blew up.

MereInterest• 6 years ago

And if the answer is no, I guess the logical conclusion is that internal tools make very little difference to the success of consumer product companies.

That is on the implicit assumption that businesses are reasonable, and accurately know how to manage tradeoffs. Your counter-hypothesis seems rather likely to me, if we look at the internal behavior of a company. When negotiating, the more dots that you need to connect before reaching "and then we get more money", the weaker of a negotiating position you have. It's the same reason why sales teams get bonuses for making sales, but developers don't get bonuses for enabling sales.

babesh• 6 years ago

Long feedback chains take longer to optimize so this is possible. Also one feedback would be to have the company fail. Another possibility is that it really is a tradeoff of where to focus.

For example, we are now paying for our lack of preparation and sclerotic government with the economic and lives lost to covid-19. It just took between 3-10 years to pay the piper.

jedberg2• 6 years ago

And if the answer is no, I guess the logical conclusion is that internal tools make very little difference to the success of consumer product companies.

Or perhaps no one has realized the power of good internal tools, and so crappy tools are the status quo and no one needs to do better.

But as soon as one company realizes the value of internal tools and it becomes a competitive advantage, other companies in that area will need to follow suit.

chillacy2• 6 years ago

To the contrary, some of the most successful software companies have strong cultures of internal tools (think FAANG companies). How much of this is cause and effect I'm not sure though. To some extent it's easier to make investments in internal tooling if you're profitable.

AceJohnny2• 6 years ago

Google?

seemslegit4• 6 years ago

What do we know about the quality of the tools used by say the chromebook sales analysts ? This isn't spanner or bigtable.

danans• 6 years ago

Googler here, but opinion is my own. Google's problem is that we have too many powerful internal engineering tools, many of which overlap in functionality, and so often its hard to know at first which one to use to solve a problem.

Thankfully, there is nearly always an overlap period when one solution gets deprecated and the replacement is launched.

The COVID19 office shutdown has really focused attention on our internal IT tools (most of which are also offered externally). So far, they've worked pretty great, in no small part because things like BeyondCorp [1] were designed from the start for distributed workforces in zero-trust security environments.

It wasn't always like this though. Corp-Eng, as it's called at Google was for a long time seen as a dead end career-wise - disconnected from Google's marquee products. However, a number of developments, including the rise of the enterprise and cloud computing businesses, have bolstered its internal importance, and with that have brought excitement and talent to the organization. It's now an important source of product feature ideas that eventually make their way to customers.

That said, a lot of things not core to Google's problem spaces do get solved with outside vendors' software, and some of that software is great. For example, I just used a site licensed version of the Chrome ScreenCastify [2] extension to make a video demonstrating a feature I'm developing. It worked perfectly.

1. https://cloud.google.com/beyondcorp

2. https://www.screencastify.com/

awa• 6 years ago

I am not sure about the funding level but the Google internal tooling for build/code review/search are the best I have seen.

twic• 6 years ago

The company i work for is not big or consumer-facing, so this is not an answer to your question.

But we do have a dynamic that i find interesting. The company is organised into many small business units - some very small, fewer than ten people. Each is accountable for its own profit and loss. They hire their own programmers to build the actual line-of-business software they need. But there are also internal tools and systems, shared across units or used by their developers, that are built by programmers hired directly by the top level of the company - their reporting line goes straight up to the CIO and CEO, rather than to a business unit manager. So, in a way, our company does the exact opposite of deprioritizing internal tools and systems! The programmers who develop the internal tools and systems are the elite!

tsomctl• 6 years ago

Sounds like Intuit.

Mindwipe• 6 years ago

Some larger video game production companies have started to do a lot more on their internal tooling, as it has a huge impact on their content production.

Most of them learned the hard way though. And some learned the hard way and still have awful tooling despite some efforts to fix them (Bungie...).

tschwimmer• 6 years ago

Yes, Facebook invests heavily in developer infrastructure. On my former team there were many staff+ engineers who were extremely capable. More speculatively I’d go so far as saying quality of talent is actually stronger than many groups working on public facing products.

Management of the group was also strong in my opinion.

lima• 6 years ago

Does Facebook maintain an internal fork of Phabricator or is it relatively close to mainline? It's such a great product but parts of it are quite clunky - I sometimes wonder if there's some internal secret sauce to smooth these out.

diebeforei485• 6 years ago

Apple IS&T isn't development infrastructure - while they do have some generic services, most groups at Apple develop and use their own tooling.

seemslegit4• 6 years ago

Developer infrastructure does not seem to be the right area to compare with here just like Google's internal engineering products aren't, more like - how polished are the tools used by the ad sales people, analysts, moderators etc.

voz_4• 6 years ago

+ 1 to Facebook.

jedberg2• 6 years ago

Netflix. At least when I was there.

Internal tools got just as much support as product. Engineers moved between working on both, and the hiring process was the same for both.

I did an interview recently about this on the retool blog [0]

[0] https://retool.com/blog/how-to-build-great-internal-tools-je...

K0SM0S• 6 years ago

Nice interview, great mindset.

Netflix is indeed becoming quite the sensation for its internal policies— the general intelligence apparently prevailing there. Seeing tools as first-grade citizens seems like a very sane approach to me— empower each other and watch people create marvels.

Papermill¹ notably is one incredibly seducing tool IMHO.

1: 2019 talk by Matthew Seal, ~40 min https://www.youtube.com/watch?v=3FmBJ847_y8

summerlight2• 6 years ago

I expect those tech giants which offer B2B solutions to have better situations. e.g. Google, Amazon, Facebook, Microsoft, etc. Many of their internal products eventually have made the positions in their cloud portfolios.

millstone• 6 years ago

Google and to a lesser extent Facebook invest heavily in internal tools. IMO this is because they hire for the company, permit easy transferring, and programmers sure love to scratch their own itch.

Google has a tool where interviewers can provide feedback and a numerical rating for candidates. But it also has a whole data mining workflow: you can visualize an interviewer's past ratings on a histogram, compare their hire ratio against average hire ratios, etc. It's cool and interesting, and sees a lot more love than many of their user-facing products.

Facebook has Bootcamp, their ~8 week onboarding process. There's a zoo of purpose-built tools just for Bootcamp, like the bot that sends you a question whose answer is written on the whiteboard just to confirm that you attended some Bootcamp class, or the Bootcamp Mentor rating machinery.

Some of that is excessive but Apple's underinvestment sin is far worse. The SWE org learned to run lean during the dotcom bust, and never unlearned it. Radar is your lifeblood, make it best in the world! Please!

Source: worked at all three!

danzig13• 6 years ago

Sounds exactly like government contracting.

larrik• 6 years ago

Is this the team handling iTunesConnect and other such "services"? I'd definitely believe it.

danpalmer2• 6 years ago

From what I understand, probably not.

iTunesConnect has had an overhaul in recent years to give it a much shinier and more reliable frontend. I believe the original was closer to the original WebObjects code, and now they've extracted out the user-facing bit into something a bit more modern. There's a lot of legacy WebObjects stuff around the iTunes backend, and my guess is that it's good engineers held back by old tech, rather than just bad engineers.

saagarjha11• 6 years ago

It's App Store Connect now!

captainredbeard2• 6 years ago

It’s their sister org for lack of a better term.

rllearneratwork• 6 years ago

saagarjha11• 6 years ago

No, services are run by Apple Media Products under Eddy Cue. They're not great, either, but I think they have more full-time engineers than IS&T does.

madeofpalk• 6 years ago

IS&T is purely internal. Basically no one outside of Apple uses anything IS&T makes.

nemo2• 6 years ago

IS&T is largely internal, but there are a number of IS&T owned external web apps for registration, surveys, and support related things. I worked in IS&T on teams developing them years ago.

messick• 6 years ago

No.

Waterluvian• 6 years ago

I could use some blog posts about quite functional teams.

Like someone on a team for a decade talking about why it works so well and discusses cultural and political issues and how the team overcame them.

leoc• 6 years ago

See my old comment https://news.ycombinator.com/item?id=19468090#19468699 . That kind of thing used to have a much higher public profile about 30 years ago.

ChrisMarshallNY3• 6 years ago

I managed a high-functioning C++ image processing group for 25 years. I kept senior-level engineers on staff for decades.

We tended to write "engine code." Like pipelines and whatnot.

The company had a very (very) long tradition of engineering (Like, 100 years).

There were many downsides to the way they worked (over-structured, mostly), but they always gave my team and me a great deal of respect.

They didn't use contractors very often.

To this day, design quality, code quality, product quality, documentation, and focus on deliverable are the major cornerstones of my software work, but I am shocked at how little that matters to modern development shops. It's been a really disheartening experience.

I guess I was in a silo of quality-focus, all those years. They would treat very minor bugs like extinction-level emergencies (not fun).

person_of_color• 6 years ago

Is this company still in NYC?

pm903• 6 years ago

but I am shocked at how little that matters to modern development shops. It's been a really disheartening experience.

Modern development shops operate in a hyper competitive environment with razor thin margins. Unfortunately, great documentation is often the last thing that's prioritized for such shops.

Personally, I think this is a HUGE mistake; talented technical writers are in good supply and should be hired by any development shop to improve adoption of their product. HOWEVER, as a developer, the shop will prefer that you work on features rather than documentation.

karatestomp• 6 years ago

Things I've seen that seemed to contribute to team happiness and productivity:

1) Process-focus. It's not "stop doing that, Bob", it's "let's make sure that can't happen unless we really, really want it to". And not just for technical stuff.

2) But also a willingness to kill processes if they weren't useful. Always OK to ask "who needs this meeting?" or, understanding its purpose, "can this be a few 10-second Slacks with follow-up when necessary, instead of a 15-minute team meeting every day?"

3) Project managers who were "on the team's side", at least so far as the team could tell. You come in taking the whip to people, you end up with bullshit information. You can't manage a project effectively with bad info. You want your team to tell you when shit's going sideways, or just that it might, early. They won't do that if they think they're taking on personal risk by raising issues. And they'll be stressed out and their work will be worse.

4) Semi-relatedly, estimation based on past (team) performance, on the same project. No guess-timates becoming commitments. No "so, can you have that tomorrow? Two days?" This is one part of the "agile" thing that, when done right, I've not seen anything surpass in accuracy or in keeping panic and confusion away. It does mean your estimates at the start of a project can't usefully go very far into the future and will best be expressed as large ranges until at least a few weeks in, and that you can't swap people around all the time and keep estimating accurately—but that's true anyway, even if you pretend it's not. I have seen people care a lot that their estimates often suck but not be willing to give up the "so, one day? Two days?" ambushes or accept that there will be times in a project when responsible estimation is very imprecise. These folks tend to get bad info on top of the inherent flaws in their approaches, because they have trouble with #3.

5) No "I have five bosses" crap.

That all may be more "in the weeds" and/or obvious than what you wanted, though.

RugnirViking2• 6 years ago

- Project managers who were "on the team's side", at least so far as the team could tell.

This. I've found this so imporant in my time. As far as I'm concerned, if a manager can't keep their own team on side, they might as well not bother. When the manager is popular and people unite around them in times of stress, then everything will work so much better. From there camaraderie and esprit de corps naturally starts to follow, especially if the product manager personally hired many of the members of the team and starts to build a tradition.

If I were to add something to your list it would be accountability, but not blame. If somebody does something wrong, both engineering wise but also in soft skills, its important that figures of authority take the time to both explain where the person went wrong with a view to trying to understand why the person made the mistake, but also visibly put into effect processes for the future to prevent this kind of thing, either by adding it to the onboarding training or with some kind of code review or even just make sure everyone in the team knows a key piece of information that person may have missed.

captainredbeard2• 6 years ago

IS&T shouldn’t be part of Apple. It’s culture and engineering talent aren’t representative of core engineering.

chillacy2• 6 years ago

In the old adage that "you get what you pay for", the internal tools at Apple aren't quite the best as a result. The Radar tool I recall was pretty horrible in particular.

Austin_Conlon• 6 years ago

Does IS&T also build the developer-facing Feedback Assistant?

astrange6• 6 years ago

Radar is the world's best bug tracker. The sole reason for this is that the originator gets to close bugs, not the fixer.

taormina• 6 years ago

Hey, Apple still hired them. Apple is chosing to dilute its core engineering brand. It's not like an end user can tell if a "real engineer" wrote the shitty webpage that's currently giving them grief.

saagarjha11• 6 years ago

Consumers don't interact with IS&T projects.

5cott0• 6 years ago

Body shop wars. What gets really interesting is working for a body shop with consultants from 2 other shops on the same team.

coleca• 6 years ago

I had a similar experience w/Apple. A startup I worked for was brought into Cupertino for a meeting w/their internal business teams. They wanted us to build them an app which seemed relatively simple involving their internal Cafe Mac cafeterias, data centers, and possibly retail locations. It was something that a company like Apple could build in their sleep. But the business folks we met with said that's how it is at Apple, all the engineering talent goes towards the product side and almost nothing is left for internal IT. They told us how they struggled to get anything done and there were almost no resources available, so the business teams had to go and hire their own IT if they needed things done.

ethbro3• 6 years ago

But the business folks we met with said that's how it is at Apple

That's how it is at most companies.

Split orgs (IT & business) result in only work of sufficient size, scope, and impact being able to cross the barrier.

Multi-year logistics management rewrite? You'll get two IT teams.

Frank in accounting needs to schedule a daily job to copy from one datastore to another? He doesn't have permissions to, and has been doing it manually for the last 5 years.

IMHO, every org along those lines would benefit from an independent IT tiger-team whose sole job it is to find business problems and apply technology to solve them.

dtech• 6 years ago

And god forbid IT allows tech-savy or power-users to do things, that's not secure.

7thaccount• 6 years ago

The political part of IT hates this though.

redhale• 6 years ago

IMHO, every org along those lines would benefit from an independent IT tiger-team whose sole job it is to find business problems and apply technology to solve them

You're describing external consultantcies. Some are good and do as you describe, but some are bad and work simply to find more work.

jldugger• 6 years ago

IS&T is run by the CFO and it shows.

riskneutral• 6 years ago

Sounds like the politics of every large corporate IT department, but with some twists given that it's Apple. Maybe this helps explain why Apple is not Amazon.

dehrmann2• 6 years ago

If there's a company CEOs should be afraid of, it's Amazon. They killed bookstores, mall stores, how many other retailers? They were the first major cloud provider. They tried their hand at phones, streaming video, and streaming music, and the bought a grocery store. There's nothing they won't try, and they have the war chest to do it.

ping_pong• 6 years ago

One of the world's most successful and wealthy companies in history shouldn't be bottom fishing for the best "deal" on employees. I wonder how much better they could be if they decided to hire at the top of the market like Netflix instead of the bottom.

That is probably why Apple software is among the worst in the industry, and it's a shame. I know IS&T isn't strictly software like iTunes but it fits the pattern I've seen at Apple. But I guess if it doesn't affect their sales maybe I'm wrong and it doesn't matter and they are right.

saagarjha11• 6 years ago

The quality of Apple's software engineering organization is head and shoulders above IS&T's; iTunes (…and Apple Music, News, etc.) is run under a different (services) group.

IST-Throwaway• 6 years ago

Fascinating to see an article about IS&T, definitely not something I ever expected to hear about on HN!

I was part of a team inside Apple that developed internal-facing tools for the retail teams. Our team was formed because of the extremely high cost in dollars and time that IS&T wanted to charge to develop some fairly straightforward tools. My team was taken from the departments that used the tools, and we developed things that were very custom-fit to what those departments needed.

Eventually politics overcame us, and IS&T finally managed to take over our team. We all became contractors in order to support the migration to their infrastructure, and continue development of the tools. A bit later we all became fired as our project was outsourced to contractors in India, managed by IS&T PMs who had no idea what our tool was used for and had never been inside a retail store. The tools all died shortly after that, some killed by IS&T, some petered out due to lack of use now that they no longer worked correctly or were a good fit for the users.

As far as we could tell, IS&T was run as a unique company inside Apple, which did it's fair share of price gouging in order to make itself money to keep going. Its purpose never appeared to be helping Apple customers or Apple employees, it was simply to get bigger, absorbing more money and power wherever possible, with no apparent reigning in from the parent org.

Although my experience is several years old, everything in this article rings true. The contracting companies they had us working for were taking a huge cut, the quality of the code they produced was dismal, (as soon as we were no longer allowed to re-write their code major things began breaking almost immediately) and people getting transferred around constantly and having no time to understand any one project was common. (rkho's comment about their hiring process seeming like it was simply a beard for a nepotistic contractor conversion was something we definitely saw a number of times.)

All in all it was an extremely eye-opening experience. Considering how "do it the Apple way" every other department we interacted with was, being in the IS&T buildings was like landing on an alien planet.

_bxg13• 6 years ago

When will companies learn that cheap development labor always ends up being more expensive than starting with quality? This same story has been playing out over and over for decades.

girvo• 6 years ago

Our remote team (in SEA) adds a tonne of value to our local Australian based team; but they mostly do maintenance work and are heavily integrated into our local team (they’re quite literally just treated as remote devs, rather than their own company). It’s worked really well for about 6-7 years now

hermitdev2• 6 years ago

Around 12-13 years ago, I was working (as a team of one on a highly technical upgrade, 32-64 bit upgrade, new compilers, OS changes, etc. Basically everything was changing). Naturally, I was falling behind debugging obscure issues largely due to 32->64 bit (this was C++ cross platform between windows and Linux).

My director offered to toss some offshore resources my way to help. Bluntly told him: I think I have 3 months work ahead of me. Give me offshore resources, it'll more likely be a 9-12 months, because by the time ive described the problem in sufficient detail, I could have just done the work. Instead, id have to review a broken "solution" that doesnt work, probably makes things worse.

Offshore is fine when you need grunt work, pounding out CRUD interfaces or already having a detailed spec. But, offshore doesnt work when you're flying by the seat of your pants.

Most of the big ossues I jad durong that release werent even my own. A few I remember: an x86/x64 inline ASM bug, use of an uninitialized variable in a DB lib that led to random failures dealing with NULLs, an overflow in a 3rd party datetime lib and ABI mismatch because of wrong datatypes used in an OSS ODBC wrapper that worked in 32-bit, but failed horribly on 64-bit.

None of the offshore talent I worked with at the time could have identified, let alone fix those issues. Several of those took reading the actual generated assembler to see what is going on. I dont write much x86/64, but I can read enough to know when shits going to hell.

The 2 big agencies I delt with were Salient and Accenture. I could work with Salient, most of their asserts were Indian and spoke English. Our Accenture resources were all Chinese. It was a pain in the ass because none of our engineering resources spoke or read/wrote English. Everything had to go both ways through a translator. Something was always lost in the translation. We ended up eating the contract and backing out with Accenture.

0x202020• 6 years ago

The first company I worked at had a great team in all departments until we got a new head of technology. Immediately all new projects went to outsource teams, some occasionally good and others bad, so much so that we (existing devs) began to “run out” of work. Eventually they tried to replace all the devs but me, the junior, and use me as the US based engineer for customer calls and maintaining legacy products. I said no and left on good terms with everyone but the new head who didn’t understand why I wouldn’t want to stay.

Within 6 months the new head was gone, the legacy code hadn’t been up kept and no new products had made full releases (of a planned 4-5). I did some maintenance and rewriting/launching one of the products I had originally started on as the only dev until they could hire someone else, but it did cost them 3x what they were previously paying me.

antipaul2• 6 years ago

I would like to ask _why_ does this nightmare exist, but is that the right question? What is the right question to ask here.

m463• 6 years ago

I wonder if it is because it lacks critical importance and sexiness.

So it lives out of the limelight.

Also expectations of internal tools are low.

Think about the sexiest jobs in tech. They are working with the newest technology, or developing the funnest resulting product.

If you have someone working on the latest machine learning technology, management has a vested interest in keeping those folks happy.

If you have someone working on computer games, the employee has a vested interest in working hard because they're doing something really cool.

But internal tools development? Expectations from the rest of the company are low, it is viewed as a necessary evil by management but not a revenue generator, and to many employees working there, it is a stepping stone.

Which is really a shame. Good internal tools can really foster a good culture. Shitty tools really take people's minds off the ball.

The worst is when some shitty off-the-shelf tool is site-licensed and shoehorned into an organization and people just suffer. I vaguely remember working somewhere where some sales tool was used as the bug-tracker or trouble-ticket system. gah.

csours• 6 years ago

This is the ad-absurdum[0] conclusion of the "Cost Center". Cost Centers don't make money, they cost money. Therefore any money you spend on them is lost.

Along with this you have the Iron Law of Bureaucracy - organizations tend (tend!, it's not absolute) to further themselves ahead of their stated goals or directives.

The right question everyone should be asking is how does the parent organization actually make money, and how to help them do that?

0. https://en.wikipedia.org/wiki/Reductio_ad_absurdum

1. https://en.wikipedia.org/wiki/Jerry_Pournelle#Pournelle's_ir...

motohagiography• 6 years ago

Sounds like a bank, a government agency, or any business over a certain size.

I tell people you only call it politics when you are losing. More accurately, it's a layer of literal stupidity above the competent to shield the money side of the company from the leverage that operations people would have if they had any information about how the money side worked.

Instead of a hierarchy, rethink a company as a hub and spoke model with concentric rings. The main differences are the implication in a hierarchy that there is "gravity," keeping people down and that they need energy and leverage to climb "up," which further implies there is a place to "fall," and that there is only one way "up," instead of many possible paths to the centre from all directions. There is no gravity, only gates and barriers, and even these are just information. Politics is how a middle manager runs interference and creates distractions to make sure you can't see over, around, or through them, and that the people behind them closer to the money can't see you. Tech is usually outside the main perimeter, mediated by contracting companies or middle managers whose job is to compartmentalize the value people create, and be sure it is replaceable.

Viewed this way, of course this demented political farce is how Apple works, because it's how everything seems to work when you have internalized the precise and specific mental model someone uses to take advantage of you.

Sorry if you can't unsee it now, but hopefully it will be funny and we can get good, competent people who value tangible skills into positions of power.

seph-reed• 6 years ago

This is just your own personal model? It's beautiful!

Is there a community where we could discuss something like this? "The spoke." I want to get obsessed with this.

mettamage• 6 years ago

I wouldn't know about it. I have seen more corporate political comments on HN, albeit far and few in between. But they are there.

I think a simple thing to do is to spot them and favorite them if you're interested in this sort of thing.

sabalaba• 6 years ago

https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-...

taneq4• 6 years ago

More accurately, it's a layer of literal stupidity above the competent to shield the money side of the company from the leverage that operations people would have if they had any information about how the money side worked.

This is a brilliant description of it. The money people aren't stupid, despite what the workers think. They just have more information and different motivations which, in most larger organisations, they actively avoid explaining to the operations people.

Edit: Changed 'an incredible description' to 'a brilliant description' because it is entirely credible.

coffeemug3• 6 years ago

Could you explain this more? Who are the operations people? (finance? HR?) Who is the money side? (product? sales?) How exactly would "operations people" have leverage if they knew how the "money side" worked? Like, what specifically would they do? Also leverage over whom? And ultimately, couldn't whoever all these groups eventually report to (the CEO but possibly lower) resolve all this by firing some people?

I don't have a good model of how these things work. I encountered it first hand, but couldn't develop a good model after a long time of thinking about it. Tons of things happened that I didn't understand at all, and I couldn't understand why higher-level people wouldn't just resolve it.

Help understand this better?

SuoDuanDao2• 6 years ago

Replying just so I can find this again. I'm going to have to think about it in the coming days.

colejohnson66• 6 years ago

You /can/ favorite comments

fishingisfun• 6 years ago

i dont understand this comment

chubot• 6 years ago

I think a summary of it is that some things are broken by design in a large org.

If you've worked in a huge org like Apple or the government, you've seen incredible inefficiencies. How can all these smart and often well-paid people be doing something so wrong (i.e. wasting time with political battles, etc.)?

But that's only true if you look at the "medium view" of the organization. If you look at the large view, how the money actually flows, then it might be beneficial for one part like IS&T to be kind of broken, as long as the rest of the company works.

I have seen this dynamic at other companies. Internal tools can sometimes hold too much leverage over the organization. They can almost "blackmail" people into getting their way because they have a literal monopoly over what they do. It might be better to let multiple teams fight battles amongst themselves, which seems inefficient if you're working as a regular joe, but could be efficient from the CEO's perspective.

----

Edit: A related idea is the principal-agent problem:

https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...

e.g. if you've worked at any large org, you know that not everyone acts in the company's interest (nor should they strictly speaking). A trivial example of this is putting extravagant meals and vacations on the company's card. (Although the nontrivial instances of this go all the way up to fraud/embezzlement.)

A less trivial example is making up fake engineering work that sounds good when review time comes, but doesn't actually advance the company's business.

So some things that seems mystifying on the ground level are pretty much a defense against this inherent conflict of interest.

In an adjacent domain, a lot of HR is a defense against employees acting in their own interest. (I see a lot of veterans on here saying: "remember HR doesn't exist for you", which is presumably a hard won experience.)

bitcoinmoney• 6 years ago

Can you explain this more?

danharaj• 6 years ago

It's as if there's three classes of people involved in economic production, workers, managers, and owners. hmm i'll have to think about that some more. It really hits the marks.

jml7c5• 6 years ago

This sort of sarcasm does not foster quality discussion.

extesy• 6 years ago

This deserves a larger blog post, perhaps with some more practical examples? I would love to understand this model better.

dehrmann2• 6 years ago

you only call it politics when you are losing

Gonna remember that one.

mattbillenstein• 6 years ago

Reminds me of the Gervais Principle - I think this has ruined me for working in larger companies: https://www.ribbonfarm.com/the-gervais-principle/

twoquestions• 6 years ago

I'm not good at seeing interpersonal relations, so it took me a couple years of working at a Fortune 50 to figure this out, and it's absolutely true. When I finally put it together, my political/economic beliefs shifted considerably. These organizations aren't there to create value, but to concentrate it, even at the expense of everyone else.

ummonk• 6 years ago

"Don't work in IS&T" has been the common wisdom in Blind for years now. Interesting how long it takes for the media to pick up stories like this.

ksec2• 6 years ago

Does Radar belongs to IS&T?

And I wonder if this IS&T power grabbing and politics happen and grow within the past 10 years. i.e Happen under Tim Cook's watch.

It surely doesn't seems very Steve Jobs approach.

nemo2• 6 years ago

Radar's owned by IS&T.

IS&T was completely broken under Jobs and hadn't changed much under Cook while I was there.

saagarjha11• 6 years ago

Does Radar belongs to IS&T?

Yes, but I think they have a dedicated team to it now?

apple20193• 6 years ago

I would be grateful if any Apple insiders here can discuss the risk of layoffs if one were to join the company now.

I have an offer to join an engineering team at Apple, but the chances of getting hit by layoffs is giving me pause. More details on my situation are here: https://news.ycombinator.com/item?id=22809769

sjg007• 6 years ago

Apple has a ton of cash. I think layoffs are unlikely. Many times new hires are protected as well for a bit.

apple20193• 6 years ago

Do you work for Apple? Might you have any insight into how the company decides how / who to layoff, when the time comes?

I would have thought that the newest would be the most vulnerable, since they haven't had the time to absorb institutional knowledge and make internal connections.

stormbrew• 6 years ago

IS&S is not the same as IS&T. Totally different orgs.

apple20193• 6 years ago

How does IS&S compare, relative to other orgs at the company? Does the org you work for (in contrast to your role or seniority level) matter, when it comes to job security?

Thanks for your insights.

Bahamut• 6 years ago

Apple didn't have mass layoffs when it was almost dead I have been told by engineers who were around at that time. I'd think Apple is one of the safest tech companies to be in as an engineer when it comes to times like this.

samdung• 6 years ago

The contract companies (Wipro, Infosys) hire graduates by the 100's from mediocre engineering colleges, slap on a little bit of training and advertise them as world class. The starting salaries are something like INR 25,000 (USD 350) per month. An onshore posting for these low skilled graduates is a quick way to make money. So they suck up to their managers who suck up to bigger managers. Their main criteria is to get hold of contract work by undercutting, knowing very well there is vendor lock in.

rclayton• 6 years ago

Lol. Just like Govt contracting.

PeterStuer• 6 years ago

"They’re just fighting for the roles,” Sabapathy told me. “That’s all they care about, not the work, not the deliverables, the effort they put in, or even talent. They’re not looking for any of those aspects.”

This is spot on for every large organization I ever consulted for. Moreover. Even in much smaller businesses, typically starting at around 70-100 people (sooner if management has prior large business experience) you can already see this pattern starting to seep in.

This is also why the whole private vs public sector efficiency debate is farcical. They are both identical in this aspect.

epberry• 6 years ago

For Apple, fixing its broken IS&T division would not only be the right thing to do from a moral standpoint — it would help the company’s business as well. If Apple is going to become inventive again, it will need to give its employees more time to develop new ideas.

I actually laughed. What kinds of applications do people think IS&T is building? Supply chain management, HR, support software (Apple forums anyone?)... I don't see how Apple sticking its core engineering resources or at all innovating within IS&T helps their business.

Steve was the one that coined the 10X rule, or popularized it. That idea must still hold within Apple's executive ranks. I guarantee Apple's top engineers never interface with IS&T and their time won't be saved by improving it. In fact I would bet that managing the deliverable of one of IS&T's projects is a sign that you're not one of the golden ones.

That said, I obviously think these contractors should be treated much better and overworking people for low pay is never okay. Maybe Apple could use some smaller dev shops that treat their people better.

xp84• 6 years ago

The part I don't get is, why does a company, especially one like Apple who would count software quality as a core competency, choose to pay $120-150 an hour to get contractors who aren't even good enough to command a salary (or hourly rate) of half that. I get why like, United Airlines does that. Or the US Government. They probably rightly assume that they can't properly manage a huge software development or IT org. But Apple? Really? What the hell are they gaining? If they paid a developer full time $120k a year plus benefits that still beats $120+ an hour. WTF.

chmaynard• 6 years ago

The days when the primary mission of IS&T was to develop custom software to run Apple are probably long past. The big exception is Radar, an ideosyncratic issue-tracking system which was developed in-house by IS&T around 1990. As far as I know, Radar is still used in R&D and elsewhere at Apple.

Tim Cook probably couldn't care less about the internal affairs of IS&T. As long as they do their job, play by the rules, and don't impose too much overhead on the company, it's all good.

“A Cold War Every Day” inside Apple's internal tools group

Related Stories