When Employees Use Software That IT Hasn’t Approved

https://hbr.org/2019/06/when-employees-are-using-software-that-it-hasnt-approved

by r0n0j0y • 7 years ago

247 318 7 years ago

hbr.org

tyingq6• 7 years ago

"Soon enough the CIO sniffed out the project and called her in to a disciplinary council."

Somebody has apparently lost touch with who the customer for IT is.

chrismatheson• 7 years ago

I've seen this over & over, i always think of it as a sort of a god complex from "admin" rights.

burfog3• 7 years ago

It's fear of copyright lawsuits and even having the Business Software Alliance convince federal marshals raid the business.

It's fear of malware.

It's fear of data being locked up in the format used by an employee's personal tool.

Above all, it's fear of being held responsible for any of the above.

It's also a fear of being not needed.

spydum• 7 years ago

No, CIO role often carries responsibility for security. VP violates policy is like skirting regulation - yes it cost less money, but for all you know they are not compliant with policy and aren’t doing the whole job.

However it does often seem like IT doesn’t consider SaaS solutions - they always want to build something their selves without doing cost analysis.

tyingq6• 7 years ago

Sure, you need security. I would, though, expect to be summarily fired if I proposed something like a "disciplinary council" for when I had a disagreement with my customers.

If you need rules to force the business to engage with you, you've failed.

thisisnico3• 7 years ago

IT Does consider SaaS solutions. When the business executives see the cost of the solutions, the business leaders decide to roll your own. SaaS isn't the end-all be all for everything. It's all about value add and achieving a goal at the end of the day. Trust me IT would much rather roll a SaaS solution, far, far less of a headache and less overhead for the department.

burfog3• 7 years ago

I have to use SaaS solutions for work, and the security situation terrifies me. I have to put my corporate password, with access to all sorts of important stuff, into a sketchy 3rd-party web site. This looks mighty bad.

TheCoelacanth• 7 years ago

The security triad is confidentiality, integrity and availability. If a security expert doesn't make sure that their security policies give users access to the things that they need, then they are only doing two-thirds of their job.

toyg3• 7 years ago

I have the opposite experience - most IT I know would rather outsource as much of their job to "the cloud" as they can, and go feet-up.

The problem is typically that cookie-cutter solutions don't necessarily map what the leadership requires: either the cost is too high, the knowledge gap is massive (e.g. the tool can do everything, but requires specialized knowledge of an obscure DSL and implementation details only three people in the world have actually mastered...) or the security implications are nontrivial.

To be fair, I do know also people who will always prefer to build their own anyway, because it makes them feel more in control (which they are). It's the CEO's job to rein in these tendencies when necessary, though.

thisisnico3• 7 years ago

To be honest, I find it odd when you treat it as if everyone else that you work with is a customer. I don't believe in this philosophy. The business is my customer. The business is what IT is trying to protect. If you have individuals that are not following policies, they would be disciplined like HR would discipline for not following policies. It's all in place to protect the business and what's best for the business. Sure you'd like admin rights to your own machine, that will help you individually, but will it help the business as a whole if we get hit with cryptowall again?

mc323• 7 years ago

Agreed. Everyone is beholden to the company and its principles, not the CEO or manager, though there should be alignment, but when there isn’t, then it’s the company.

kyriee• 7 years ago

Did you miss the line where it was written "before we started the program, we were losing revenue. Now we increased it by 1MM$ / month"?

It seems that the whole point of the article is that while IT thinks that it is serving the customer (the business), it actually lost track of what it was needed to: helping it succeed.

bob33212• 7 years ago

It goes both ways. Is the business user taking inappropriate risks to get his own work done quicker? Or is IT denying access to vendors to minimize his personal responsibility?

scarface743• 7 years ago

I find most “IT security policies” that hamper developers to be mostly security theatre. No matter how many policies they put in place, since they aren’t developers, one junior developer can write:

  var sql = “select * from Customer where firstname = ‘“ + firstname + “‘“;

And thwart all of your security “best practices.”

I was the lead dev at a medium size non tech company, and the hoops I had to go through to get anything done dealing with the “security team” was ridiculous and of course I didn’t have access to production to troubleshoot for awhile.

I had ultimate control of all the code that did go through the process. If I were to do something stupid or purposefully malicious, while I didn’t have access to the environment - my code did.

As far as someone mistakingly installing a “crypto wall”, if a user can download a program that doesn’t require admin access, that program has access to the user’s files. The system can be restored much easier than the user’s data.

jjoonathan• 7 years ago

Sure -- if we also discipline IT when their policies fail to meet business needs, because they ultimately serve the business, not themselves.

It's really easy to secure / fix / support a system by making it nonfunctional and redefining that as success.

woofie113• 7 years ago

I don't think it's reasonable to treat everyone you work with as your customer, but that's not what's being proposed.

IT's role is generally to support the organization. The organization is its customer. For the most part, it doesn't "work with," but it supports. In any organization, there's a complex network of who is a customer, who is a client, who is a peer, and so on.

There are places I'm not IT's customer, but they're the exception rather than the rule. If IT isn't providing a service I need, then that's a failure of IT. At the end of the day, the fallback is to purchase the same service elsewhere. If IT needs to know about that (e.g. for audits or security), it's fine to have a process for that (I report to IT, IT verifies what it wants to), but if that process becomes an unnecessary roadblock (IT doesn't want to compete for my business, rather than a core security issue), either people will circumvent that process or the business will take a hit.

The customer-provider networks vary on business. In some cases, engineering is the customer of marketing, and in some cases, the other way around. You have companies where marketing decides what to build based on customer conversations, and engineering builds it. In other cases, engineering decides what to build, and marketing sells it. And then you have all sorts of cases in between, from synergistic peer relationships to all sorts of balances where one drives but the other informs.

That doesn't change the gross organizational dysfunction being described in this article.

awinder• 7 years ago

Implicit in protecting a business is that the business continues to exist, i.e., that it's run competently and can hit revenue targets, it can grow, etc. Focusing on rules & decorum is playing from behind, rather than thinking about how IT can become a trusted partner from inception (so that you are out ahead).

BTW -- if IT's goal is really to protect the business, then you should find & discover the ways people are getting around your fences, because the first thing that a malicious actor is going to do is find & hop those same exact fences. These people finding security holes should be lauded as whitehats finding your mistakes, not people to be punished for not following rules.

mc323• 7 years ago

It depends. If the company has instituted SSO and MFA and someone goes out and uses a solution that is outside of that, they could be exposing the company to liability.

tyingq6• 7 years ago

It's the fact that something like a "disciplinary council" exists, and that it was the first tactic the CIO went with that bothers me.

There's a reason they went around the CIO. I suspect if the CIO had met with this person they could have learned and helped.

imtringued• 7 years ago

The "customer" doesn't care about IT and wants to do things behind the back of the CIO. The "customer" goes to extreme lengths to hide the fact that he is violating company policy by purchasing SaaS with his own credit card. If that employee leaves one day, all the data inside the SaaS is gone because nobody else knew about it. A malicious employee could also use it to extort the company.

adwww• 7 years ago

how about trust your staff.

geekamongus2• 7 years ago

What could possibly go wrong?

pjmlp• 7 years ago

Unfortunely many prove that they aren't worthy of such trust and open the company to liability issues, data loss, lawsuits and other security issues.

jcims• 7 years ago

Domain Admin for everyone!

Trust is always contextualized. I trust my mom to watch my kids but not to remove my appendix. The purpose of technical controls and security policy is to wall off areas where employee capabilities or motivations are too uneven or complex to safely expose them without a risk of loss that's incompatible with the appetite of the business.

justinclift• 7 years ago

Maybe, "empower your staff as they each prove competence", instead?

burfog3• 7 years ago

I've been amused by VMWare being on the strictly-enforced official software list, and the VM being considered data. Nothing in the VM counts as software! It's not even being sneaky. Official policy is that the VM is data.

geekamongus2• 7 years ago

It sounds like the author recommends embracing the Agile philosophy of letting your teams choose their tools, then working with IT/Sec to make sure implementation is sound. I like that philosophy.

FooHentai• 7 years ago

Great until you have five teams, each having chosen a different tool, and now you're wondering why the IT support costs are out of control.

Still possible to support but requires a different model e.g. one where IT delivers a new, unconfigured workstation to your new team member and it's up to them to build it. If it breaks, their loss of productivity is their problem and not ITs.

Authority for something (e.g. software selection) must go in conjunction for responsibly for consequences arising. Those things must always move in lockstep to avoid perverse outcomes.

closeparen2• 7 years ago

Our IT does not support development environments. Just network, backups, printing, client certs. Dev tools support is all essentially peer to peer and ad hoc, with escalation to the internal owner of the tool (another engineer) sometimes possible. If you mess up in an unrecoverable way, IT will give you a loaner to work on while they reimage your machine. It works fine.

S_A_P2• 7 years ago

I see this a lot in consulting. When a new CIO (or CEO or other C level) arrives, they want to make their mark with a digital transformation intiative. This usually just means that the new C level employee is coming into a medium to large business and would like to add a bullet point to their resume and get that new shiny object everyone is talking about. Tableau, Salesforce, Data lakes, blockchain, ERP, Identity Management and "cloud" projects are often the result. It seems to also stem from the new C level employee having a close relationship with a sales rep/partner/C level employee at the vendor. Left a project a couple years ago that had Hadoop interfaces from every system. The user count of all this data? exactly 0.

One somewhat disturbing trend I've seen at some of the largest corporations- cut/outsource IT support staff to near egregiously low levels to "save money". At the same time kick off 7-9 figure ERP/consulting projects that at best provide fractional value to the organization.

Of course there are counterpoints to this. One of Houston's major pipeline operators pulled off a digital transformation and actually ended up with well designed, highly integrated and easily maintained systems. It took about 5-7 years and had a few reboots, but it eventually landed. That brings me to my final point. These projects often have a timeline that is divorced from reality. Whatever time frame you think a major IT project will take. Double it. twice, then add 50% and you are close. It also seems that C level folks are hesitant to hire boutique/small shops that have industry experience and years of experience in favor of big consulting. Nobody every gets fired for hiring Accenture/Deloitte/PwC. What usually happens in the non trivial niches is that these big shops sleeve the boutiques through them to get things done...

maxxxxx22• 7 years ago

All you are writing sounds exactly like my company.

ct520• 7 years ago

Ditto. Working in the enterprise industry I can relate to a lot of what SAP is talking about.

lzol2• 7 years ago

This resonates so much and seems to be a major trend in non-traditional tech companies. I've mostly worked in the financial industry and the executives' knowledge of technology is almost always horrible. As you said, a couple buzz words and very set opinions on the ways to do things. It's like they get pet projects in their head from reading an article in a magazine and get locked into it.

I don't really have an issue with CXXs being ignorant about a subject. No one knows everything. What I do have an issue with is when they act like an expert ignoring all the people who are actually experts in a particular area. It'd be like me going into a room full of IT people and saying EBITDA a bunch of times claiming to be an accounting wizard ready to lead a major initiative. It's frustrating but I've learned all I can really do is smile and watch the show.

noir_lord2• 7 years ago

It's like they get pet projects in their head from reading an article in a magazine and get locked into it.

It's not like that, it's often exactly that.

I'm extremely fortunate in that while my boss sets the goals he never specifies how they should be achieved.

That means I get to implement them as we need them.

I'll never underestimate the value of smart management :).

alexhutcheson2• 7 years ago

You'd probably really appreciate this comedy sketch video: https://www.youtube.com/watch?v=BKorP55Aqvg

bt3• 7 years ago

Nobody every gets fired for hiring Accenture/Deloitte/PwC. What usually happens in the non trivial niches is that these big shops sleeve the boutiques through them to get things done...

To provide a prospective as someone who works for a consulting firm like the ones you've mentioned... Hiring the "big" firms versus boutiques is a lot about a perception of risk, maintaining partnerships (procurement with new vendors is a nightmare everywhere), and leveraging experiences across other F500. For big implementations, think any ERP, our consulting teams can number 100+. Overkill? Probably, but there are few boutiques that have those kind of resources. And each of those 100 have more experience doing implementation work across a range of companies. It's a bit of a vicious cycle for boutiques where they will ultimately struggle to be competitive in these types of bids without seriously underpricing their services, which ends up meaning fewer resources or a less comprehensive scope.

As a side note, when projects go south, the company CIO isn't getting fired, but I've know many leaders (from the consulting side) getting let go for botching multi million dollar contracts.

S_A_P2• 7 years ago

Agreed. There is a perceived risk with the F500 so they may spend more to do something than is required because they feel they have recourse with the big shops. There are some good teams of folks out there, but there are also some shops that are happy to send a dozen folks @250-300/hr that generate process maps in visio and power point slides instead of delivering/implementing a project. I have seen these big shops deliver in a timely manner and I have seen boondoggles that waste millions. I don't think they are by default "bad", I just think they are not necessarily the right choice in some circumstances, and the right option is to hire the boutique niche firm that specializes in what they need. This lack of awareness is exactly what the article brings to light. CIO shoots down VP who needs timely solution for long protracted big rock implementation.

lasereyes136• 7 years ago

Sure, they get fired but are hired somewhere else because of the contracts they made botching a multi-million dollar contract. There isn't much of a downside to them for over-promising and under-delivering.

wdb• 7 years ago

My experience with working with those consulting firms is that you start with 2 of them and at the end of the year you end up with 6 while still wondering why and having the test team somewhere offshore not doing what they promised so you end up doing it yourself as boutique firm

argd6782• 7 years ago

One somewhat disturbing trend I've seen at some of the largest corporations- cut/outsource IT support staff to near egregiously low levels to "save money".

I see the opposite too, they just staff up on tons of IT people thinking they have a resource shortage, and end up with massive departments that deliver just a little as before.

It also seems that C level folks are hesitant to hire boutique/small shops that have industry experience and years of experience in favor of big consulting.

The reason this makes sense is because they need to work with companies that have enough resources that they can be really inefficient and have enough capital that they can run for long periods of time and not go under. It’s more of an insurance policy, the quality of the work would be better at the smaller shop of course but they likely couldn’t complete it due to bureaucracy.

diveanon• 7 years ago

I'm convinced that the majority of Salesforce implementations are done so that C level execs aren't the odd one out that isn't using Salesforce during their next round of golf.

snowwrestler4• 7 years ago

We've got a Salesforce implementation going at the nonprofit where I work. While there was some debate about which big CRM we'd buy, the need to consolidate was blindingly obvious.

Why? Because our organization has been quite forward thinking about allowing managers and executives to source the technology they think they to succeed. As this article advocates for, IT was largely consultative rather than dictatorial, and a lot of business units were able to pick what they wanted.

But what this has left us with is dozens of places where customer data was being stored, some of them now past their end of life. No central visibility into customer experience. People getting multiple copies of the same email from different departments using different email platforms. Poor deliverability. Subscriptions on random credit cards that suddenly turn off because the person left and no one knows how to get into the admin account and update the card.

We hired a boutique shop to do the Salesforce implementation; we're not scared of doing that. Unfortunately this time it did not pay off... their performance fell off, to the point that they couldn't even reply to emails on time. As sometimes happens with small firms, they grew too fast and exceeded their ability to operate. We can't wait for them to figure it out... so here we go with a big dog firm. Let's see how that goes.

Maybe I'm lucky in who I work with, but I find the "add a bullet point to the resume" take to be maybe a bit too cynical. Tableau, Salesforce, data lakes, ERP, identity management, and "cloud" infrastructure each seem like useful tools if implemented smartly. (Note that I took out blockchain...)

lmkg• 7 years ago

One of the side-effects I'm seeing of GDPR is a stronger incentive to consolidate systems under central management. Companies that allowed different departments the leeway to control their own systems now find themselves literally not knowing how many different places a customer's data might live.

peteradio• 7 years ago

Whatever you want implemented smartly probably doesn't need each of these buzzwords. Implemented smartly could live on a desktop in a closet.

ownagefool5• 7 years ago

Your problem statement does make it sound like you need a CRM, but I do wonder why is has to be a big CRM with a big consultancy, and why IT aren't delivering it?

Who's going to run the thing afterwards? Will the bigdogs deliver something that you can maintain, or is that generally against their own interests?

Finally, who's gonna secure all this customer data? Are they taking that on as part of their remit? They rarely do.

ownagefool5• 7 years ago

Digital transformations can be great. The problem is, anything great will be sold to you by consultancies as a way to give you more consultancy services, which is kind of the opposite of what a digital transformation should be.

maxxxxx22• 7 years ago

This is exactly extremely common. In my company there is this constant battle about the devs having admin rights on their machines. We need admin rights to do our job. We have had dozens of meetings explaining the situation but IT can’t come up with a solution so the devs go around security because they have no alternative if they want to finish their work . Same with Dropbox. They block it but we have suppliers who use Dropbox. So the result is that people download confidential files from Dropbox on their home computers or phones and transfer them to their work machines.

In my view security shouldn’t be isolated at corporate headquarters but they should be close to the end users so they see what users need to do and help them to balance security with getting things done. They can’t just block stuff without providing alternatives or they will either hurt the business or they will be circumvented.

77483947736392• 7 years ago

I have my own PC strapped under my desk connected to public Wi-Fi. I use it to do all my work. My company pc is left turned on but disconnected, sitting on top of my desk to dissuade suspicion.

maxxxxx22• 7 years ago

I have a WiFi router hidden so we can do some specific network testing. We also have a consumer DSL line so we can test on a network where IT doesn’t block random stuff. It’s not even much of a secret. Everybody local knows this stuff and every six months a guy at IT headquarters throws a fit, doesn’t provide an alternative and gets ignored. I have thought about sending them recordings of previous meetings so we don’t have to repeat the series of same meetings every six to twelve months.

esoterica4• 7 years ago

Does your company know you have their IP on your personal device (and are sending company communications over public WiFi)?

megaremote• 7 years ago

I have a VM on my mac to go on the work network, using ethernet, while I do everything over wifi.

brokenmachine6• 7 years ago

Don't you need shared drives, etc?

We can't connect to anything internal from outside, there is VPN access but it seems a step too far to use that all the time.

ConfusedDog2• 7 years ago

I entirely agree with you. This is the same idea in the Greatness video by David Marquet (https://www.youtube.com/watch?v=OqmdLcyES_Q)

Try to control everything is just futile. I think hiring and retaining best people is the best countermeasure to this sort of things.

rb8083• 7 years ago

I think most of the reasons for admin rights are no longer valid. Its easy to change user environment variables and lots of applications can be installed as a user. Why would you need admin rights?

Dropbox/googledrive is a huge security hole that is definitely blocked at most companies I work at.

beart• 7 years ago

One example off the top of my head: It used to be (may still be) the case that you needed admin rights to install and run the Windows Subsystem for Linux. Sure, you might not need this to do your job, but IT is not really in a position to decide that. It could be that WSL greatly increases your productivity.

maxxxxx22• 7 years ago

This is about Windows desktop development. I need admin rights to install sql server, I need them to customize my machine so it’s similar to our target environment. I need to change user permissions all the time yo see how things behave under different conditions . There is a ton more I could walk you through and have done multiple times. Comments like yours come repeatedly from people who don’t know about the work we do. I have offered them to demonstrate doing our job without admin rights but so far nobody has even tried. They just keep sending the same email about not needing admin rights which has repeatedly been showed to not work.

alasdair_• 7 years ago

Running things like wireshark or certain debuggers without admin rights is often difficult.

Also, lots of stuff simply cannot be installed as a regular user, especially stuff that needs unfettered access to network cards or memory.

wernercd2• 7 years ago

Why do you need admin rights?

IIS development - Visual Studio needs Admin to actively debug IIS.

Memory tools like dotMemory.

Dealing with Windows Services.

Shit... dealing with Windows.

TheRealDunkirk6• 7 years ago

lots of applications can be installed as a user.

Because most of the non-insignificant ones still CAN'T be, under Windows, to this day. So special people get a completely separate account with pseudo-admin rights. I have to enter those credentials several times a day.

Then I spoke to a help desk guy, who said he had to enter his domain admin account password 40 TIMES a day.

What a waste.

exabrial3• 7 years ago

I really wish some big shots in the security world would write an ISO standard or something stating how harmful blanket 'block Dropbox' policies are for the reasons you list.

maxxxxx22• 7 years ago

And they should commit to talking to my VP every time the VP commits us to working with a supplier who uses Dropbox and also commit to finding solutions that allow us to get our work done within deadlines.

ubermonkey3• 7 years ago

The "dumb" here isn't even limited to "block Dropbox." Lots of my customers have blanket "block everything that could plausibly be used for file sharing" policies, and explicitly include services literally AIMED at corporate/B2B data exchange like Citrix's ShareFile.

No, we don't have an internal FTP site. No, I won't set one up for you. We use Sharefile for distribution so we don't have to do that. Your IT blocks it? Yeah, that's dumb. Go talk to them; it's not my problem. We're not going to do customized delivery channels just because your halfwit CIO decided to block every site with an upload button.

esoterica4• 7 years ago

There’s nothing wrong with a block Dropbox policy. The problem here is a failure to establish a standardized method of transferring files in and out of the company.

protomyth6• 7 years ago

I'm still not sure why developers aren't on their own network for development. Have a red box / blue box type system at the developers desk. Given modern networking, it wouldn't actually be that hard to setup and keep development / integration / system tests (or what names you use) away from a locked down production would not be such a bad thing[1]. Having some dual homed file shares wouldn't be that hard either.

1) I would probably still use different terminal color schemes, but being in front of a different physical box might be quite a good thing.

maxxxxx22• 7 years ago

I have proposed that. Give us a network where we can do what we want do and protect the boundaries of it. So far no luck.

lightbritefight2• 7 years ago

Developer data can still be confidential/sensitive, so you still need to monitor and control this second network with many of the same restrictions as the main one. You still have most of the same risks to compensate for, like data exfiltration and cryptolocker, etc.

It doesnt introduce that many positives for lots of admin overhead, not just in maintaijg two distinct networks, but also in ensurijg interoperability when needed.

pmorici2• 7 years ago

Shouldn't this kind of thing be a problem for the managers to address? If you just circumvent this kind of nonsense instead of addressing it head on it just proliferates and allows the people who promote it to think they are doing an acceptable job.

At minimum you should inform your direct manager of the situation so they can address it or accept the consequences that the work that depends on the restricted resource won't get done w/o circumventing company policy.

maxxxxx22• 7 years ago

Management knows.

lnanek2• 7 years ago

I've heard security management say it is their job to say no all day. They definitely don't care about preventing work getting done. They will only get fired if a data leak occurs, etc.. Preventing work won't even ding their promo outcomes.

jnosCo2• 7 years ago

I'm one of those assholes that makes security policy. I deal with the same requests. The problem is, I write up a proposal identifying the risks associated with the exemption, along with minimum and recommended compensating controls. This then gets discussed among IT Management, where it is usually decided it's too much overhead, and to just deny the request or if the user can scream loud enough, allow it outright and get some director to sign something. The third oft-used response is ignore the problem and hope the user finds their own work around so we can get back to the 13 projects we're somehow expected to complete this quarter.

maxxxxx22• 7 years ago

I feel you. The whole setup is dysfunctional by design.

AstroJetson• 7 years ago

Me too, we try, but it's too easy for people in the management chain to push a fix into the "Too Hard Bucket" (tm) and it dies.

novok5• 7 years ago

It's an incentive misalignment. IT is evaluated in 'how secure things are' or 'how easy is it to maintain' or 'does this give me more headcount'.

Not letting people do their jobs, or in how fast they can do their job. Employees are a captive audience, and if there was competition they would probably chose something else.

oceanghost2• 7 years ago

I worked at a very bad company, where IT was aggressively incompetent, and mean about it. At best they were negligent, at worst they actively interfered with anyone who they thought was a threat, which included anyone who was more intelligent than them, which was virtually everyone since the company was full of senior EE/ME/RF/CS folks.

And I mean incompetent-- our network would go do down for hours, every day, and the problem lasted almost a year. It got so bad our "source control" became yelling "Whose got the latest main.cs?!" and walking it over with a USB key.

I realized we needed to disintermediate the IT group, and my boss was supportive. I ended up establishing our own, parallel IT department with a business cable line, some routers, and NAS devices. I outsourced any task to any cloud service (this was over a decade or so ago when that was somewhat unusual) — eventually hiring our own IT person even. People would ask for access to our network because it was more reliable :-)

Then things got vicious. They actively interfered with our group, trying to get us fired (they had some success with this previously). They were able to get our IT guy fired. Things just went on that way for years. They'd find some way to make trouble, and then we'd route around it. Once they even took my computer for two days, denied they had it, then returned it but locked me out of it. I was certain this was part of some scheme to get me fired, so I turned the computer off, zeroed the hard drive, and req'd a MacBook to work on.

They all eventually got fired when someone took note of their gross incompetence. One of their replacements was eventually fired for embezzling, proving that despite thinking we had the worst IT department anyone could imagine-- we really didn't.

user59944616• 7 years ago

Why would you stay for years in a company like that? It's so dysfunctional and toxic.

fancyfish• 7 years ago

I've consulted for a big bank which blocks most file sharing sites and also blocks you from attaching scripts, server logs, etc to emails.

Luckily, S3 is not blocked. I set up a bucket and have them upload to the bucket, which I then download on my own computer. Mission accomplished.

Just getting free Windows applications procured and installed on the company laptop takes several layers of approval, emails back and forth, etc. Also the lovely forced password resets every 2 months.

This is true for the banks I've worked with/at. Security systems actively block you from the tools you need to do your job, let alone get the job done in an efficient manner.

padraic7a• 7 years ago

I work for a local government department. Must try your S3 workaround. I swear sometime I think our IS dept are playing jenga, they just pull random services at will. Today I couldn't update our website because the proxy settings allowing me access the login page somehow changed without notice. Last week they blocked USB access to machines without telling anyone who backs up to external 8tb drives. Tomorrow who knows what they'll decide. And it doesn't make for a secure environment! Everybody tries to figure out workarounds. Staff actively try to undermine security policies. It's a total disaster.

BonesJustice• 7 years ago

Be careful. I work for such a bank, and even secure traffic is MITM’d when possible. There are data loss controls in place to analyze outbound traffic for things like source code.

Just because you’ve circumvented IT’s blocks doesn’t mean you won’t land yourself in hot water.

OkGoDoIt• 7 years ago

I used to work at a place like that, it was incredibly frustrating and time-consuming. So I came up with a solution that works even if S3 is blocked: I built https://github.com/OkGoDoIt/UploadAndPaste and set up SCP file hosting on my own server that listens on port 443. (They blocked most outgoing connections to non-standard ports and did MITM sniffing on any port 80 traffic, so this was the only way to get through.) Then I could just easily "paste" a file to a my remote server and download on the other machine via a url.

TheRealDunkirk6• 7 years ago

Outgoing SSH is blocked at my company, even to non-SSH ports. Even to virtual machines I had already setup in Azure before the block.

Sure, guys. Of all things to block, let's block the most secure one. That'll really improve our security posture.

At this point, I'm continually surprised they haven't superglued the USB ports.

Shivetya• 7 years ago

I have rarely found that they need admin rights on a day to day basis unless the tool is badly designed. I have one software delivery platform that requires full admin rights, it cannot write just to the user configuration!

However developers are very good at presenting it otherwise. Myself, I have run into issues where admin rights were needed, again always because of some poor installer. USB has been blocked as well and you can guess it, cannot do their work.

If they need admin rights all the time then put those particular machines on a protected network and not allow any other business work to occur there.

mcshicks• 7 years ago

I think it really depends on what kind of development you are doing. I worked at an IC company, and among other things we developed (and tested) USB drivers for our devices. We installed them on literally hundreds of windows machines for testing before they were signed/ whql approved. This happen all the time. So really no way do our jobs without admin passwords. We did have the machine on isolated networks, but even on the regular networks many of the team member frequently would need to hand install drivers. This is just one of many examples. If you do anything even remotely hardware related, it can really be a totally different problem. We had IT installed USB filter drivers (we were not told about for security reasons) that actually broke a lot of testing in our labs and it took us months to figure out.

Aeolun• 7 years ago

Even if you need admin rights only once a week, if it takes a day to get them you waste 20% of your time.

twunde• 7 years ago

The problem may actually be compliance requirements. SOC2/HITRUST/SOX all mandate the removal of admin rights from computers, mandate an approval process w/ manager approval. Regulated industries, especially banking have more security-related compliance requirements causing a lot of the pain.

Unfortunately from a security perspective devs and system admins are probably the highest risk targets since they typically have access to servers and admin rights. At the very least they have source code an attacker could analyze, and likely have access to external services.

The reality is that compliance, security and usability are often in direct conflict that can only be solved to make everyone happy with significant work.

richk449• 7 years ago

SOC2/HITRUST/SOX all mandate the removal of admin rights from computers, mandate an approval process w/ manager approval

I’ve heard this before, but never with any detail. Can you explain further, or point to a resource? For example, clearly SOX doesn’t say that nobody can have admin rights - because IT does. And I highly doubt that the law says that only departments with IT in the title can have admin access. So what does it really say?

m463• 7 years ago

In my company there is this constant battle about the devs having admin rights on their machines.

One company I worked for had two lans. The admin lan and the engineering lan. Your admin machine would always work. The eng lan could go to hell, and it/management wouldn't know/care.

I think it worked well. It was kind of like having the common areas of the house neat and tidy, but giving the kids creative control over their rooms (and being able to close the doors when things got out of hand or guests came over).

orionblastar• 7 years ago

Or they block Youtube, and you got clients using Youtube to describe how they want things done. Watch it on your phone or at home.

vikingcaffiene• 7 years ago

In my company there is this constant battle about the devs having admin rights on their machines.

I ended up leaving my last job over this and more stuff like it. They had a url filter in place for instance, that would randomly block access to our network resources. It would never be the same one so every now and again your stuff would start failing and you would lose a few hours debugging until you realized.... GAH! Then have to email and lose the rest of your day waiting for them to fix it.

Never. Again.

brixon• 7 years ago

Let's ignore the SaaS security issues for a second. When IT says "No" it's not like the area asking is going to go away and not try to solve their problem. Organizations are going to find ways to solve their issues and IT can either help from the beginning or help clean up the mess later. I try to take the stance of offering the right solution and a lot of the times a now solution at the same time. There is no saying "No" in the long term, either help them now or get stuck with the shadow solution the magic macro guy cobbled together that became a critical business function.

rosege• 7 years ago

In my last job I tried to handle this in a similar way. The issue we ran into though was that often these managers would not properly evaluate the software. They would get wowed by the sales guys and sign up for huge contracts without, sometimes, even checking with IT or testing other vendors.

tvanantwerp2• 7 years ago

Whenever I'm wearing my IT hat at work, a big part of it is being a detective looking for clues that somebody, somewhere, is about to do this. Then I can insert myself into that process. It would be preferable to be included in the beginning, but one must work within the reality of the situation.

x38iq84n• 7 years ago

I have seen IT being unaware of and unwilling to meet requirements of highly specialized technical teams, such as network engineering. You cannot have a TELNET client because the use of TELNET is prohibited by corporate policy, test TCP connections another way. You don't need vim when you have vi. You can't have admin rights but we don't support drivers for RS232 dongle so nope. Sometimes it's quite a challenge to get some work done.

user59944616• 7 years ago

Use netcat.

Telnet is pretty hard to procure since it's not included by default since Windows 7.

basch9• 7 years ago

There can be a lot of steps between "help me fix this problem" and fixing the problem. They include qualifying the idea, scoping the request, possibly transforming the request into an abstract form and searching the organization for other people with forms of that abstract problem. Then you get to procurement and you need to figure out if you are getting a specific tool for a specific task or some kind of kitchen sink. Now that youve learned what the kitchen sinks do, you go back to scope and decide if other requests or projects are moving into this one, or if perfect is becoming the enemy of good. Then once your implementation project is done, you need to redesign old processes around it, communicate the change, and offer training.

Saying yes and fixing the problem quickly, without analysis, will often UNDERSERVE your company in the long run, because you only fix a specific problem for a specific person, functional group, or division.

ativzzz2• 7 years ago

You're essentially suggesting "properly" going through the slow and inefficient bureaucracy machine, when the underlying issue is exactly that people tried to do what you suggested but ended up getting nowhere.

You realize you need a kitchen sink, it turns out there are 2 other teams who already created their own semi functioning kitchen sink which they want you to adopt, but doesn't fit in your kitchen, and the CIO is working with Home Depot to create standardized kitchen sinks for the entire company which will be ready in 3 years (realistically 5 years, or maybe never), but your current sink is leaking and is flooding your kitchen now, so you do what you can to fix it.

Basically a principle of asking for forgiveness rather than permission. Does it cause issues? Of course, any solution to today's problem becomes tomorrow's problem. Now there are 3 semi functioning kitchen sinks in your company, but at least they are functioning

drevil-v2• 7 years ago

I still shudder thinking about my time working as a developer on corporate IT locked down IBM leased laptops. Every time I did npm install I needed to request admin access to Windows which took 2-3 hours to action by IBM team sitting on the other side of the world in India.

One day a grey beard took pity on me and installed a Linux VM where I was admin, copied the security certs from the Windows host and I could access all corporate resources at my leisure. Never logged a single IT Helpdesk ticket after that.

abledon• 7 years ago

wow. that sounds horrible. did they ask you fizzbuzz or mergesort in the interview?

maxxxxx22• 7 years ago

Probably not. The interviewer didn’t know how to do fizz buzz himself.

flurdy• 7 years ago

Yup, back then my way around corporate locked down Windows machines was to only get permission to install VMware Workstation or Virtualbox.

Then fullscreen Ubuntu in a VM from then on. Slower but no restrictions. Even better when Workstation supported multiple screens.

Macha2• 7 years ago

Honestly on my work machines, though we have root, the difference in FS performance tilts in favour of VMs/containers due to the slow endpoint protection affecting native FS access, and for a lot of tasks we do that outweighs any virtualisation overhead.

thrower123• 7 years ago

I'm curious, were you usually going through Sametime to make requests into that helpdesk?

raxxorrax• 7 years ago

If you don’t think this is happening in your organization, think again

That story probably never happened anyway. But the essence of the article is very true. I never have been in a corp where IT enforces 100% conformity anyway (apart from medical industry).

Sure, there are actual successful attacks, but that is mostly not the fault of unsanctioned programs.

But there are systems where people should not just start to use any system, because information gets lost on the way. That would include CRM and ERP in my opinion. That a company can exist without a CRM is questionable to begin with and solutions are plentiful. If they did not have anything like that...

If the story were true, it would not be the fault of Chief Input/Output.

cannonedhamster• 7 years ago

I've been in corporate IT where this happened. All company apps were built internally. None were able to run on anything past Windows XP. On top of my regular help desk, asset management, software project, and lease refresh program I was also somehow supposed to make the software work with Windows 7 as they had let the developers go. This is the same company that refused my sane security requirements and ignored just about everything until too late. I hear they have since outsourced IT and networking and it's failing dramatically, but they are saving money right?

philipodonnell2• 7 years ago

The CIO admitted that he had been approached and explained that he had informed the VP that IT already had a project with SAP to deliver what the VP needed. “Yes, but that won’t be ready for me to use for three years, and I need something today,” retorted the VP. The CIO was silent. Then the CEO asked the VP, “I’ve known you for ten years. You don’t seem like someone who would do something to harm the company. Why did you do this?” The VP hit right back: “Since I started this digital customer acquisition program, we’ve increased revenue $1M per month. Before we were losing revenue. If you want, I can shut it down right now. What do you want me to do?”

Shut it down right now and ask the VP to tender their resignation. Any company doing a 3-year SAP implementation is a very large company. That $1M in additional revenue pales in comparison to the risk introduced by sharing company or personal customer data with a vendor who has not passed the required security auditing. Data is no longer a thing to be thrown around in search of additional revenue and "but I made money" or "I had to because IT is slow" is not a post hoc rationalization for the behavior.

Regardless of the merits of large enterprises acting this way, this is a VP who clearly cannot function within the enhanced risk-controlled environment of one and should find a position with a smaller company where they have more freedom to pursue personal initiatives at the VP-level. Those companies exist. Go find one.

enraged_camel• 7 years ago

>Any company doing a 3-year SAP implementation is a very large company.

Not at all. I have a client with ~100 employees who are past year 2 of their Salesforce implementation because the director of technology keeps changing priorities and project requirements.

philipodonnell2• 7 years ago

That's fair, but just using the time-frames in the article, 6 months had passed before the VP got caught and still had 3 years left on the implementation. I think a director of technology who take 3.5+ years to do a CRM implementation at a 100-person company... isn't doing a very good job. :-)

arkades• 7 years ago

“My unit is bleeding money for lack of a CRM solution so cheap I can finance it on my personal credit card. Help?!”

“Cool, we will get back to you two years after you’re out of a jo- I mean, we will have a solution ready in three years.”

Yes. Someone should be tendering their resignation.

SerLava2• 7 years ago

If your company can't replace a doorknob in a decade, destroy that company. It has no value.

TheRealDunkirk6• 7 years ago

What does it mean, to you, to do a "required security audit" of a company like Salesforce, or any cloud provider, for that matter?

ConfusedDog2• 7 years ago

This is my current situation. Being a SWE, IT and security are always putting out fires with networks or upper echelon cybersecurity violation complaints (mostly people downloading software without authorization). They have very little time, almost none for investigating new software, and all software must be installed by them. End of the day, nothing gets done on our work computers. I once waited two months for them to say no for a piece of solution we as the team approved. It's absolutely frustrating.

noonespecial• 7 years ago

Its the same dilemma companies face with their legal team. The "safest" thing to is nothing at all so sometimes the overabundance of caution hamstrings business growth.

That's what the CEO is there to figure out.

exabrial3• 7 years ago

IT is a service to the rest of the company. If you don't approach it with a servant's heart, people will go find their own solutions without you and you'll be part of the cleanup crew.

jmkd• 7 years ago

Joining Google was an eye-opener for me on this. Was the first time I encountered an IT department (TechStop) that didn't act like a police force and instead had your back, helping you get where you needed to be. Was always the first thing I would show guests on a tour of the campus.

repolfx4• 7 years ago

But you're probably a developer?

TechStop is/was great. But Windows users had locked down workstations where IT whitelisted binaries. I assume the approval process sucked about as much as normal.

Many Google employees use desktop Linux which is basically unheard of outside the tech world. That by itself simplifies things quite a bit. Not many people writing viruses posing as screensavers for Google's in house Linux strain. Anyone who cracks that is probably an APT attacker and those require different approaches.

alexhutcheson2• 7 years ago

CorpEng also does an amazing job building in-house apps & tools to let people get their job done.

tvanantwerp2• 7 years ago

The ideal IT team is one that proactively learns the needs of others in the business and works with them to solve problems. It's no wonder that so many companies end up with shadow IT when so many IT teams are just people who tell you "no" whenever you ask for something. Doing it right is harder in the near-term, but much easier in the long-term as you're not putting out so many fires or going to "disciplinary council" meetings.

eithed• 7 years ago

To me it reads like this - VP didn't care about the consequences of utilizing their solution and didn't care about IT; they simply wanted their stuff done, without acknowedgling prioratization of tasks.

The proper way this could have been resolved is by VP utilizing people's skills they've hired. Does this solution look good and will accomplish the task that was prioritized? Excellent! Pass it to IT to evaluate. If the task has specification - excellent, have somebody in IT look for a product that ticks all the boxes and let's choose it together.

fphhotchips• 7 years ago

How many years since The Phoenix Project and this conversation has barely moved an inch?

CIO probably wins this battle and gets the VP fired, but will be mystified when they're reporting to the CFO or a Chief Digital Officer when it happens 3 more times by the end of the year.

scarejunba2• 7 years ago

It is wonderful. Each crack in a behemoth’s process is a point of leverage for a startup. We will outcompete by being better.

finnthehuman• 7 years ago

How many years since The Phoenix Project and this conversation has barely moved an inch?

People are still fighting the lessons in The Mythical Man Month; it's gonna be a while.

hartator• 7 years ago

Isn't IT something from the past? I would expect people knowing how to use a computer and what they need to do their job.

kleborp• 7 years ago

As a lowly help desk technician perusing this thread, you couldn't be any further from the truth. The software devs never open tickets, everyone else does and its for the most banal problems

rhinoceraptor• 7 years ago

I've opened plenty of tickets as a developer, mostly the tickets are IT's garbage software or horrible network setups preventing me from doing my work.

silverfox17• 7 years ago

A lot of people who post here don't seem to realize that a lot of companies still have IT departments.. a lot of people here are also developers who don't realize they can easily do things to compromise data, even if they think they know better.

Macha2• 7 years ago

When MacOS borks itself in an update because the endpoint management software does something funky with partitions, I'd rather delegate that to a team that's handled three cases of that this week and get on with my job description than diagnose the exact steps to resolve a problem that'll often require access I don't have anyway.

jillesvangurp• 7 years ago

My thoughts exactly. It's been a while since I last saw a company with an actual IT department and even longer where they had an actual clue. The reality is that IT in most SMEs simply sucks and no longer requires a college education. Working in IT for an SME is not a career plan. You're at constant risk of being outsourced and essentially all you do is done better by a gazillion companies as a service that probably cost less than a few months of your salary. Frankly, most SMEs would be better off doing exactly that. Most startups I work with do this from day one for obvious reasons.

Once you hit a certain scale different dynamics may kick in but even then, outsourcing is an option.

As a freelancer these days, I bring my own laptop and am granted access to stuff for the duration of the project. It's understood and expected of me that I do such things as encrypt disks, use 2FA, and don't use "secret" as the password. Most stuff I access for these projects is SAAS based. I'd probably walk away from projects where that wasn't the case.

dragonwriter2• 7 years ago

My thoughts exactly. It's been a while since I last saw a company with an actual IT department and even longer where they had an actual clue

Let me guess: most of your recent experience is with relatively new and/or small companies, probably in the tech industry.

dragonwriter2• 7 years ago

Isn't IT something from the past?

No.

I would expect people knowing how to use a computer and what they need to do their job.

Yes, but their job generally isn't maintaining the computer or it's supporting infrastructure (networks, shared servers, etc.) or, outside of dedicated programmers, programming the computer even as an incidental task. IT exists to do (or coordinate contracting out for) those things, and the last point—restricting programming to dedicated programmers, has pretty much monotonically increased since the 1980s, when incidental programming was both more common than now and frequently projected to become increasingly common (lots of people said all good jobs would require some.)

protomyth6• 7 years ago

Its the user who downloaded a program they "needed" which had malware which sent out a lot of spam email because this was a user that did announcements which basically got an e-mail server listed on blacklists that creates these IT policies.

You want to treat people like responsible adults, but they aren't the ones who have to deal with the fallout. Developers know the score for the most part, so full privileges are expected with the caveat, if it all goes bad, we are wiping the machine[1], not doing a recovery.

IT dreads the moment we are called to account for something some user decided they needed to do.

1) most developers understand backup tools and code control - those that don't, well...... with great power comes great responsibility

dx87• 7 years ago

Yep, a company I worked at hired a tech writer that downloaded some cracked version of software that included ransomware on their first day of work because they said they didn't want to wait for the company to get them a legitimate copy.

toyg3• 7 years ago

Well, that used to be common practice... in the '90s.

Thank $deity for the rise of opensource.

thomasjudge• 7 years ago

I work in an IT organization & I see (in the sense of witness) both sides of this. We are over-tasked and under-resourced and new projects/ideas/initiatives that come in the door go into a backlog of requests. So I see business/end users signing up on their own for SAAS solutions to solve their problems.

argd6782• 7 years ago

Right, the CIO is also being held to a lower standard than a P&L. The CIO could have planed out an interim solution to meet the business needs quicker.

If the tables were turned, say the CIO needed to deliver a service and didn’t have a big enough budget, then what?

sgt101• 7 years ago

Well, that cleared that up then! Gosh I had no idea that the solution would be so simple.

It does shock me that the people who've had their whole infrastructure compromised and held to ransom by viruses and the people who've been held over a barrel by suppliers or had vast amounts of money burned by being locked into a dozen vendor contracts for the same service are so silly and hysterical about it when the solution is as simple as "identify when you need to be best in class and stay small everywhere else".

thisisit• 7 years ago

As a born and bread corporate (mostly banks) corporate IT guy, I used to frown upon this behavior. Then I got one of my bigger career breaks because the finance team went behind IT, bought a software and installed it in a machine which kept under their desk. They further hired people from an IT service company to configure the machine.

The configuration was so bad that it exposed the company's network to whole wide world. Google contacted the company and after searching high and low IT security managed to track the pc down and take it away. Finance team promised to hire someone with skillset required to run the software in a closed environment. And that's how I ended up getting my job.

lapnitnelav• 7 years ago

Really interesting story, especially if you put it in the context of being corporate world and a finance team, both of which aren't the type to rock the boat.

It says a lot about the struggles of enabling change in structures with strong silos that (what are probably among the most risk averse type of) people would go to such lengths.

vinay_ys• 7 years ago

Does your IT team use key loggers or other employee monitoring software at your company? I hear some big trillion dollar companies do this. Is that true?

jrjarrett2• 7 years ago

This thread hits home. I switched jobs a few years ago because the IT policies on workstations were being ratcheted down to make my job as a developer difficult to impossible.

Now, the company I work for, ostensibly a _software_ company, got its ISO certifications, which meant policies and procedures that make developing hard or impossible again.

How does a software business _successfully_ implement stringent access controls while still allowing for efficient software development? I'd like to see/hear what works.

CrossWired• 7 years ago

I'm heading down this path right now. How do I obtain my certs while also allowing enough freedom for the dev teams to operate.

We have to deal with the fallout when they screw something up, there has to be a happy medium somewhere.

user59944616• 7 years ago

What's the issue specifically?

Developers don't need admin rights for much of anything in this decade. No need to bother with that.

Common software has to be made available in self-service, so developers can install development tools like notepad++ or visual studio.

Deployment is usually the challenge because you have to store binaries somewhere, copy it to some random servers and finally execute it, each step causing numerous security headaches, so there has to be some approved tooling to handle that.

miqkt• 7 years ago

Alas, I'm in a similar situation with my current stint and looking for an exit.

The most maddening part for me is to literally sit around helpless and unable to do any development because you need to wait for your IT support ticket to be looked at. Then having to explain to your manager why work is behind schedule.

However, idle time alone doesn't seem like strong enough reason to open discussion on changing IT policies.

davvolun3• 7 years ago

The CIO admitted that he had been approached and explained that he had informed the VP that IT already had a project with SAP to deliver what the VP needed. “Yes, but that won’t be ready for me to use for three years, and I need something today,” retorted the VP. The CIO was silent. Then the CEO asked the VP, “I’ve known you for ten years. You don’t seem like someone who would do something to harm the company. Why did you do this?” The VP hit right back: “Since I started this digital customer acquisition program, we’ve increased revenue $1M per month. Before we were losing revenue. If you want, I can shut it down right now. What do you want me to do?”

Maybe not for this particular project, but another interpretation of that is "who cares about security if we're making money" which is a very dangerous argument as well.

yebyen2• 7 years ago

When a person says "we need this infrastructure project" and a project is commissioned, acknowledging the need, it is in my experience that unfortunately that person's job function is rarely placed on hold until the appropriate infrastructure has been made available.

"Who cares about your pie-in-the-sky infrastructure project, my boss continues to measure our real performance with basic accounting, and is expecting to be able to report on growth each quarter, which I can't help without tools" seems to be a bit closer to the argument posed here, IMHO.

davvolun3• 7 years ago

Of course that's the real question with the actual story in the actual article -- could they have implemented their security on the temporary infrastructure, or "good-enough" security, if the CIO knew about it?

In that story, is the blame on the VP for going ahead instead of getting dialogue started between CEO, VP and CIO? Is it on the CIO for just saying "no" instead of recognizing the need and the value? Is it on the CEO for failing to empower the VP and CIO to get that conversation started themselves?

And then, it's all well and good to worry about the bottom line first, until you're sitting in Equifax's shoes right.

scarejunba2• 7 years ago

Yeah, but because each “who cares about money, we’re doing the secure thing” will be naturally outcompeted by the “money over security” guy since money is the measure of success and is the unit that lets you expand. The hard part is rapidly reacting to a realistic threat model for each situation. That’s why good security chiefs are so expensive.

They know when to move that risk control dial in each direction.

davvolun3• 7 years ago

Exactly, well stated. In the end, the only reason a business cares about security at all is because if they don't, it will come back and bite them, hard; e.g., Equifax.

rsuelzer• 7 years ago

Our IT security department was incentivized to deny everything from new tools to new internal applications.

We had an outside firm making security decisions and if there were any security issues it would end up being on them. So as long as they did not allow us to release any products and or install any software they could not be held responsible.

I made friends with a lower level contractor who told me off the record to use my judgement on what to install to get the job done, because the security department would never approve anything new unless directly instructed to by the CEO.

Fast forward three months, there was a major security flaw on our website (also built with outsourced labor) which allowed anyone to access private data without a login.

A few of us had reported to the security department that the code running the website was so poorly written that the odds of being insecure were close to 100 percent. We suggested upgrading the website and rewriting the code, and management was on board with this but security department refused to allow us to use any new frameworks since they were not approved. Of course in a matter of a few months the site was hacked and millions were spent as a result.

I quit this job after we were unable to release several products after a year even though we jumped through every hoop we needed to. That department killed all innovation.

closetohome• 7 years ago

I think it really depends on the company. If you're something like a nontechnical non-profit, sure, turn that decision making over to IT. In that case IT is performing a vital, skilled function.

But in most software shops, the workers are probably more qualified than the IT department to be making decisions about what applications to use, and what kind of security they need. IT is just there to make things run and fix them when they break. They don't really need to offer guidance.

la_barba2• 7 years ago

Hehe, if you think this is nuts, come to pharma. We can't do jack shit with our machines. If you so much as change the time on your machine, that is a 'data integrity breach', and if your actions are determined to be malicious it can result in a firing.

GuB-42• 7 years ago

To be honest, changing the time on a machine is a very serious concern. Accurate timekeeping is crucial in security, that's how you connect events together.

la_barba2• 7 years ago

Well, all the rigid policies like no dropbox or no FTP or no whatever, also arise from serious concerns. I just wanted to point out another seemingly innocuous one. Most of our equipment is not internet connected, and we need to manually change the time for daylight savings or other corrections. We have a company policy and procedure to do that periodically so that our audit trails are accurate. Sometimes folks get busy and the shop floor guys take matters into their own hands.

alkonaut4• 7 years ago

Security training focuses way to much on email phishing and not enough on this kind of stuff. Actually getting your work done, managing your own computer. Of course people can't be trusted if they havem't been trained. How to handle USB drives. What and from where you can download and run programs. What actually IS a program and what isn't. Many of us learned this the hard way by playing lots of cracked games in the 90s. But not everyone did that.

Try explaining to a non-technical person how how a desktop background image isn't a program so it's basically safe to grab from anywhere, while a screen saver is definitely a program and usually unsafe to get from most places, and a word document is some times a program that might eat your computer. Training could involve things like "which of these 5 webbpages would you consider it safe to download and run executable from"?

Having too cumbersome rules around security just means it's ignored or circumvented, increasing risks.

JakeTheAndroid• 7 years ago

It's tough, I give security awareness trainings myself and I completely agree with what you're saying. However, that's a lot of information to give to a group of new employees that can span any department and technical understanding.

I actually was talking today with a customer during a logical assessment about if I talked about downloading malware in the training. I dedicate an entire section to downloading documents, but I don't really give people the information you're talking about. I tell them how to avoid ever having to download anything, and if they must do it, how to try and do it properly. All of this is ended with the process on how to report incidents because eventually something bad will happen.

As a company you kind of expect this to be solved at a number of layers. Endpoint management should hopefully help resolve this issue. Restricting web access where it makes sense can help. Sec Awareness Training helps keep people aware. Etc, etc, etc. You hope your controls are what save you from incidents, because there is no way you can effectively train your entire company on security topics to a degree that they can make good, security conscious decisions. That said, many of these SAT's are really just checking compliance requirements, because thats the real need. I put my own training together starting with what I know needs to be covered for compliance (pii handling, passwords, acceptable use policy, common threats, security incident response reporting, etc). Anything else that makes it in is purely because I have extra time and I know it to be important.

alkonaut4• 7 years ago

Concrete example happened just this morning: I needed some documentation that exists on archive.is, but has been taken down from the original site. I navigate to the cached content on archive is, and archive.is is DNS blocked when going through my VPN by Cisco Umbrella because apparently it's an "anonymizer" service.

So I change my DNS settings to use an 8.8.8.8 dns first, and my company dns second. Now I can access both archive.is and sites on the company network. Excellent. But in doing this I circumvented all the DNS filtering, not just for this site. The reasonable thing would have been a warning like a https-style warning "Are you sure you want to continue to this site"? Or a way of whitelisting, perhaps temporarily, a single address. Instead my options were to ask an administrator or disable the whole security feature entirely. (Or connect/disconnect the VPN temporarily every time I needed something blacklisted, but that didn't feel like a good solution).

holy_city• 7 years ago

This assumes that a company has a culture that allows training to happen and affect change, instead of being pushed off, ignored, or laughed at.

And training on different file/executable types isn't effective. Many high profile phishing attacks have been carried out using malicious attacks embedded in innocuous files like word documents or PDFs. The only way to actually prevent malicious code from a download is to prevent the download in the first place.

noja• 7 years ago

There's a difference between using software that IT has approved and shipping customer data outside of the company.

TheRealDunkirk6• 7 years ago

I just witnessed a very similar situation, on a smaller scale, but there are many of these in my company, and they add up.

Boss: "We need access to the database of our primary application that you wrote for us so that we can pull the data into this new tool to track progress."

IT: "No. Not only can we not give you access to YOUR data in YOUR application that we wrote on YOUR dime, we will not allow you to have this new application written by someone who isn't in our group. If you wanted something like this, why didn't you just ask us? We would have written this for you."

Boss: "We had a meeting about this over a year and a half ago, and you told me that you didn't even have the time to discuss it further."

IT: "... Well, we're still not going to let you do this."

IT is effectively holding the rest of the company hostage, and the corporate technical debt is becoming epic. So skunkworks solutions will continue to be developed.

dingo_bat• 7 years ago

In our company people just started using free slack en mass, boycotting the horrible IT approved Skype for business. When it was discovered that thousands of employees were using slack, the CTO had to step in and tell IT to fuck off, and started paying for the full version.

jmspring• 7 years ago

This brings back a memory. The only time I was fired "for cause". Summer after my freshman year at college, I was temping and got an assignment doing real estate purchase comps with a company in the East Bay. At the time, there were laser printers, but often printing sucked up CPU time and let's just say multitasking was still not a widespread thing.

I found myself tired of sitting around. I found a TSR / print spooler that would use RAM and offload the process of printing. This allowed me to keep working. My productivity (as a temp) was higher than many others including the person I was "reporting to" at the company.

They found the print spooler, labeled it "unapproved software", and I was walked out the door.

The funny thing is, a friend at the time (and I didn't realize it) was higher up in the management. He reached out to me on a multi-line BBS that was popular in the area and offered me a full time job a few days later. I was in school and obviously declined.

Working the rest of the summer for a Chemical Engineer in Martinez/Benecia ended up being incredibly more interesting. So it was a net win.

jccalhoun• 7 years ago

I just wish I had a decent computer at work that didn't have 3rd party antivirus that would just slow the software to a crawl.

analog312• 7 years ago

Something that's crossed my mind is John Gall's observation that complex systems operate in failure mode 100% of the time. I understand "failure mode" to mean that built-in guards have been bypassed in order to enable the system to do anything at all. Germane to this thread, the "guards" are IT approvals.

I suspect that if a business is complex enough to have IT policy, that policy is always being bypassed in some way, at any given time. Somebody is using unofficial software, or using official software in an unofficial way.

When Employees Use Software That IT Hasn’t Approved

Related Stories