Enforcement by Software

https://lemire.me/blog/2022/02/24/enforcement-by-software/

by ibobev • 4 years ago

148 60 4 years ago

kfarr• 4 years ago

Money quote:

People seem to think that if the software requires some document, then surely the rules require the document in question. That is, human beings believe that the software must be an accurate embodiment of the law.

bob1029• 4 years ago

Of course, you can achieve the same result with bureaucracy. You just change the forms and the rules. But it takes much effort. Changing software is comparatively easier. There is no need to document the change very much. There is no need to train the staff.

Herein lies the central value proposition of our B2B product stack. With enough validations and required inputs throughout, it is possible to put very complex business processes 100% on-rails.

The value to our customers come from the fact that they can basically hire people directly out of high school and have them accurately manage complex financial transactions for customers on the front line while having virtually zero domain knowledge.

If you are stuck wondering "well how far could we take this?", look to semiconductor manufacturing. I've worked in facilities that are nearly 100% automated. If we can make an entire building that keeps people from doing the wrong thing, there isn't a whole lot that is off the table.

I do see dark patterns that can emerge from this style of software, but at the same time it is impossible to argue with the standardization of outcomes you can achieve if you are willing to go to the ends of the earth in detailing your particular theme park rides.

The biggest complaint is usually from power users, but they typically get to retain some access to more powerful legacy tools anyways.

cwillu• 4 years ago

Reminds me of the retail management system from the story https://marshallbrain.com/manna

morelisp• 4 years ago

Broader ethical issues of deskilling aside, it makes no sense to do this for university professors (virtually the definition of an extensively-trained professional) creating course material (a key task they were hired to do).

autoexec• 4 years ago

All the variations of "Click yes if you are over 18" were probably enough to get most people used to the idea that enforcement by software is a joke and that lying to software in order to get what you want is easy and profitable.

teddyh• 4 years ago

“Computer says no.”

smoyer2• 4 years ago

If any here are unaware of the origin of this phrase, one of my former bosses worked in the U.K. and said this anytime someone asked for something unreasonable. He was also nice enough to introduce me to Little Britain - here's one to get you started: https://www.youtube.com/watch?v=qNDS4kVwA68

Hokusai• 4 years ago

"Learned helplessness is behavior exhibited by a subject after enduring repeated aversive stimuli beyond their control."

The key part is beyond their control. As a society we are being trained to see software beyond our control. "The algorithm" decides. Many non software engineers fail to see that behind each software decision there is a company that can change it.

Learned helplessness is dangerous even just because improvement becomes impossible when people just accepts software like if it was the physical laws of the universe.

sumtechguy• 4 years ago

It is amazing how this sort of thing sticks too. One cell company I used to work for has terrible GUI's. I mean they are actually infuriating to use. It has been that way for at least a decade. I was reminded of how bad they were yesterday when I had to use that GUI again to fiddle something in my account. I told my wife 'no wonder the line is 20 people deep at the stores no one wants to mess with it'.

Does no one do usability testing anymore? Sit down and make someone use the software. Not 'do x and y happens, call it a day' I mean end to end 'hey here is a task figure out how to do it' then sit on your hands and take notes while they try to do it? Bad interface design makes people feel powerless. That powerless feeling makes people give up and just accept that 'computers are complex'. Then once you are in that mode little changes become a huge deal.

nerdponx• 4 years ago

Does no one do usability testing anymore? Sit down and make someone use the software. Not 'do x and y happens, call it a day' I mean end to end 'hey here is a task figure out how to do it' then sit on your hands and take notes while they try to do it? Bad interface design makes people feel powerless. That powerless feeling makes people give up and just accept that 'computers are complex'. Then once you are in that mode little changes become a huge deal.

The company probably sees usability testing as an unnecessary expense that doesn't improve their bottom line.

j-pb• 4 years ago

This reminds me of some of Startrek's greatest quotes:

  There can be no justice so long as law is absolute. Even life itself is an excercise in exceptions.
  - Jean Luc Picard

and

  When has justice ever been as simple as a rule-book?
  - Riker

silon42• 4 years ago

I love the TOS episodes where Kirk turns off the central AI.

skohan• 4 years ago

This is a concept I think about often in an engineering management context, and I think it applies not only to software, but to processes in general.

I think there is often an impulse to systematize workflows, because it makes work quantifiable, and it helps us to justify to others that we're working in the right way. In other words, if you fail without a system in place, it's your fault for not planning enough. If you fail with a system in place, there's always something outside of yourself to blame it on (as long as you were following the agreed upon process).

But sometimes rigid processes create their own problems. At various points you have to decide if something goes in column A or column B, either of which conclusion can have knock-on effects in terms of how it has to be handled by the system. Sometimes this creates problems down the line, because the system wasn't designed to handle something that isn't really A or B. Or else you end up adding columns C, D, E... to handle all these unfit cases until you have a system which is so complex and cumbersome it ceases to be useful, or you are spending all your time dealing with the system rather than solving the actual problem or creating value.

In the absence of a process, it's usually up to common sense and human discretion to solve a problem. People can see what the situation is, what the goals are, and decide the best course of action to take. And the best thing about that is, human beings can take things into account like empathy or external factors, or creative opportunities which a system could never be designed to account for. That's why it's a really good thing to find places within the system to create space for people to figure things out for themselves.

So one problem with systems and processes designed around using software is that it can make it difficult to include that human element. If the process is not designed correctly, it may be impossible to progress unless the software is satisfied.

klodolph2• 4 years ago

Surprisingly, it appears that no other department has been routing around the damage that is this new tool.

I recommend a different strategy when you encounter annoying processes like this—insist upon following these processes, to the letter, and going through appropriate channels until the system collapses under its own weight.

For example—a place I used to work had an internet filter that was too restrictive. Rather than admit that the internet filter was poorly designed and never should have been put in place, the bureaucracy provided us with a form to fill out to unblock web sites when we were blocked from visiting something necessary for work.

You could obviously try to work around the filter, and some did. If they were blocked, they tried a different site or filled out the form, and just tried to get work done the best they could. This was, in my opinion, a mistake. Fight bureaucracy with bureaucracy. People started filling out forms every single time they tried to access a website for work that was blocked. Not just once per site, but every time they wanted to access a site they’d fill out the form. The IT department got a thick stack of forms to sort through every single day, and the entire web filter was scrapped not too long after.

“Silently routing around bureaucracy” is a good plan if what you’re doing has some real importance to it. But if you want to solve bureaucracy long-term, make it painful for everyone.

The other danger with routing around bureaucracy is that when something goes wrong, you end up dangerously exposed because you can’t CYA properly if you’re routing around bureaucracy. If you route around bureaucracy and shit goes south, bureaucracy wins and you take the blame.

voiper1• 4 years ago

Sounds like malicious compliance.

But electronic forms can be easily ignored. If so, I'd follow up with printing it out and handing it to the person responsible for handling it. If it's not handled immediately, then also deliver it to their boss. And so on. It has to actually be painful to the bureaucracy so they want to change it.

Applied to this example of course changes, you could do something simple like changing the title or intro and then physically going to each other person and telling them they need to approve the change.

Then innocently change it again right after approval. ("What? It's a WIP! I have a responsibility to make updates to teach the course in the best way.")

Bonus points to get other co-workers in on it so you're not the problem-person.

CoastalCoder• 4 years ago

I think "malicious compliance" is sometimes a misnomer. I.e., when the goal is the common good rather then malice.

codecutter• 4 years ago

I agree with your overall sentiment that fighting beurocracy by dumping more paperwork is a good strategy.

However, that should be last resort in my opinion. Based on my experience, people (especially IT staff) are reasonable if you explain the issue nicely. Remember, honey attracts more bees than vinegar.

klodolph2• 4 years ago

If it’s always your last resort, then you’ll run out of political capital at work. IT staff are also generally not responsible for these policies in the first place—IT staff wouldn’t install an internet filter, for example. That’s something that a VP would ask for. Getting into fights with a VP over policy can end badly if you ignore office politics.

skinkestek• 4 years ago

"The Practice of Systems and Networks Administration"-book has this somewhere:

There are basically two strategies for getting what you want from slow bureaucrats:

- either: be the one nice and polite person they meet during the day. Chances are you will stand out and you might get preferential treatment. After all bureaucrats are humans too and they (the ones at the edges of the system) can face a lot of abuse during a day.

- or: follow the rules to a t. Not necessarily malicious compliance, just don't care that you overload the system. At some point someone will sit down and look at it and corners will be cut.

AnIdiotOnTheNet2• 4 years ago

IT might not even have any real control over the situation and you could just be making their lives more miserable anyway. Or you might be helping them provide evidence that the policy is garbage, but you really need to actually talk to someone in the department to find that out.

Our company president has mandated and refused to budge on a wildly stupid policy regarding email. The users know it's stupid, we know it is stupid, but nobody can convince the guy who's name is on the building that it is stupid, so we have to deal with it. Our solution is to straight up tell new users how they can get around the policy on a technicality.

WhyNotHugo2• 4 years ago

I recommend a different strategy when you encounter annoying processes like this—insist upon following these processes, to the letter, and going through appropriate channels until the system collapses under its own weight.

Yup, this usually works. If lots of stupid bureaucracy is required for things that should be straightforward, then my daily summary will often be "yesterday morning I had to deal with stupid bureaucracy".

I can waste time if you pay me to do so, I'll just remind you that you're asking me to waste time where I could have been doing something useful. (Tip: I won't do this indefinitely, usually a short time is enough to make the broken processes evident).

projektfu• 4 years ago

Yeah but the Author is the one who has to process the requests and has no power to change the software. They’re routing around to avoid being inundated with requests and the subsequent emails, phone calls, etc.

brudgers• 4 years ago

In the Graeber model of bureaucracy [1], a functioning bureaucracy will not collapse. It will expand. It will add one or more people whose job is to badger professors toward compliance...and whose function is to expand the empire of the bureaucrats above (for this is why many requirements exist at all).

"Silently routing around bureaucracy” is a good plan if what you’re doing has some real importance to it.

If that's not the case, there's a chance your job is toward badger-professors-toward-compliance end of the spectrum.

[1] https://www.youtube.com/watch?v=kikzjTfos0s

jrochkind12• 4 years ago

It works if the people responsible/authorized for imposing the annoying process are the ones you can impose the burden of compliance on.

In OP's story, OP was the department head, the burden was imposed on faculty (requesting any change) and department head (approving the request), but the department head was not the one who could change the process (other than by the un-specified manner they apparently did, and described as "routing around the damage")

I still agree it's better to make noise and make it known you don't like it, but if "following the letter of the law" does not in fact impose any burden on those who could make a change, it alone is unlikely to be effective.

Of course... now that I think about it... effective "political" (one definition) organizing/action itself is basically always a means of imposing a burden on those in power that can be lifted if they do what you want. It's just a matter of finding the lever that can be used by the amount of counter-power you have managed to build, to impose a burden on those who can make the change you want.

dangerface2• 4 years ago

I assume the department head had other work to do besides approving requests if the number of requests are significant to the point that the department head can't do any other work then the system still failed to bureaucracy. Either the higher ups are fine with department heads doing nothing but approve requests or they change the system.

BeetleB• 4 years ago

I recommend a different strategy when you encounter annoying processes like this—insist upon following these processes, to the letter, and going through appropriate channels until the system collapses under its own weight.

This is usually not what happens. Case in point: Our company had an easy means to share files with anyone at the company. You could specify exactly who is allowed to see it.

Suddenly IT took that means away, citing security reasons (people were not using it properly). However, all the alternatives they provided were unworkable - and I mean that literally. They only worked within people in the same department, but you couldn't send something to people of other departments.

The files (text) I needed to share were classified as top secret, so sending them via mail attachments (unencrypted) was a no-no according to IT. Encrypting them made them not work on the receiving end (the decryption process was somewhat faulty - although you could view them in the tool, it wouldn't save it properly to disk, so the tools that acted on them failed.

I made a fuss about this and for a few days complied with IT policy and work came to a halt.

But only my work. Everyone else just said "Screw IT" and sent the files unencrypted over email in violation of company policy.

My management isn't going to be sympathetic with me if none of my teammates are on board. And in general, this is how it plays out. Those who find non-kosher workarounds will utilize them, and you'll be in a small camp fighting IT. At your annual review, you'll be the least productive and will get the boot.

antihero• 4 years ago

I have a friend that works on the video desk of a major U.K. newspaper, and obviously has to receive and send video files on a regular basis. IT took a policy that blocked large attachments and attachments for most filetypes, unless you filled in a form. Of course now that my friend could literally not do his job, he had a lot more time to fill out forms. Result was similar.

nfriedly• 4 years ago

That works sometimes. Other times you get fired for poor performance

Cthulhu_• 4 years ago

But then you would have literal paperwork to explain why your performance is not up to standards - because the websites you need are not accessible, because your time is spent filling and handing in forms instead of your job.

adolph• 4 years ago

But if you want to solve bureaucracy long-term, make it painful for everyone.

Insist on doing everything through channels. Never permit shortcuts to be taken to expedite decisions.

https://workcompass.com/the-cia-guide-from-1944-on-how-to-sa...

etothepii• 4 years ago

"There is no negotiating with a computer." --Simon Biggs.

At its core I think it's the main reason executives are so bad with technology. Nothing else in life is like a computer. You can not argue, you can not infuriate. It is pure logic.

Unfortunately those of us who know about computers know that the logic is almost always flawed.

3pt14159• 4 years ago

It cuts both ways. The very system that forces people to use four digit years and unambiguous date formats[0] is the same system that leads us all into one giant Year 2038 problem.

I mostly agree about executives, though that's fading. The world got the message that being good at technology is something effective people increasingly need to know. I'm an expert in Excel (Solver, UDF, arrays, custom formats, etc) and even I've picked up some tricks from CEOs.

[0] What fresh hell is the date "12/11/10"?

pjc50• 4 years ago

https://twitter.com/swiftonsecurity/status/13855657371677245...

"A COMPUTER CAN NEVER BE HELD ACCOUNTABLE

THEREFORE A COMPUTER MUST NEVER MAKE A MANAGEMENT DECISION"

(IBM slide, 1979)

trasz• 4 years ago

Back then it was assumed that the management could be held accountable.

moritonal• 4 years ago

I generally believe that all business logic such as approvals and what-not should be implemented in the user-land portion of an engine. If it's hardcoded that x-process should be followed, then you've now attached the price of a developer to changing that process.

I'm getting to the point where when friends ask me for a project, I implement it in something like Node-Red (a "not-much-code" solution) then give that to them as a dockerfile. I know it's not perfect, but at least they "could" change it with just a mouse.

WhyNotHugo2• 4 years ago

There's a lot of this in Argentina (we call it "Legislation through it system").

For example, for many years it was legal to invoice in foreign currency, but you MUST inform the tax agency of all invoiced emitted in real time. Their API required that you specify the currency, and only the value for local currency was accepted -- nothing else.

So while it was technically legal to do something, the IT system would block you from doing it.

undecisive2• 4 years ago

I've had my first run-in with the UK's test-and-trace system yesterday.

I did an LFT on Tuesday, and registered it - but hit the wrong box, said I was positive, clicked send, panicked, didn't notice that there was a link to fix it, tried to use the back button, etc.

Long story short, I ended up registering another LFT just so that I could mark myself as non-contagious - but that wasn't enough for the Test and Trace system. The next day I was hounded by text messages, and then called by an operative at the end of the day.

I told them the mistake, and they told me there's nothing their system could do - I needed to call 119 (they didn't tell me which phone system options I needed - It's a horrendous phone system) and get them to mark it as a mistake.

But before I abruptly ended the call with this operative, she told me that until I do this, I'm not allowed out of my house because my phone number was being monitored. I was not polite about the end of that call (I was not rude, mind - not her fault they she has a script to follow - but I was a bit angry)

And once I finaly got bounced around 119 and found the right person to talk to, she said: "Well, self isolation rules end tomorrow - so don't worry about it".

I picked the right time to make this mistake, apparently. But yeah, software has far too much control and extra-legal real-life consequences.

dijonman2• 4 years ago

What an awful system! What happens if you refuse to participate?

I would give up my phone and go back to landlines.

_dain_• 4 years ago

What happens if you refuse to participate?

Luckily, nothing.

undecisive2• 4 years ago

Yeah. So given the choice, I'd never register a single one.

Unfortunately, the agency that runs the scheme won't allow me to visit my bed-bound mother without "evidence" of a clear test, despite the fact that the system only checks that a new test id is given, not whether you actually used the test, or what the result was (you're expected to be truthful on that form - you're not allowed to just be truthful when you enter the building).

Which completely defeats the point.

smoyer2• 4 years ago

"I for one welcome our new robot overlords" [0]

More seriously, does this article mean that the societal decline caused by misinformation is stronger because it's showing up on a computer screen (e.g. Facebook?)

0. https://www.youtube.com/watch?v=OGxdgNJ_lZM

bsedlm• 4 years ago

what about enforcement's softer sibling, herding?

I think most institutions are doing this knowing full well how it works and with a clear intention of where they want us to end up

I recall reading about how google would arrange the food in the cafeteria to softly nudge people towards healthy food, hence I assume that when they make software that everybody uses it they carefully and intentionally consider which things to make convenient and which to make pain in the ass. That way they can always plausibly claim to not have forbidden anything you're just being lazy by not dealing with all the 'incidental' inconvenience.

In then end, if you wanna do something which they don't want you to do then you must really really want to do it for it'll be a pain in the butt to do it. On the other hand, if it's something they want you to do then click and done.

I wish I could make more people see why this makes it important that software (and technology in general) to be free and open source, because the alternative of closed and proprietary software systems (which by being inscrutable) works as tools for nudging, restraining, limiting, and in more extreme scenarios, enslaving people who must use this systems without making them.

userbinator• 4 years ago

And there lies the scary point. People are easily manipulated by computing.

I've seen "tools turning people into tools" happen with developers too, and I think that's even scarier --- when it happens to the people who should know more about how computers work than anyone else, it is really disturbing. I suppose some of it might have to do with the general erosion of freedoms and individualism that's happening to society in general; it seems governments and corporations have slowly discouraged thinking for oneself and rewarded docile obedience.

ryandrake• 4 years ago

Reminds me of an article by Venkatesh Rao, who points out we're moving toward a world where you are either above the API or below the API. You either tell robots what to do, or are told by robots what to do.

1: https://www.ribbonfarm.com/2017/08/17/the-premium-mediocre-l...

RajT88• 4 years ago

The flip side of this is when software intentionally does not implement adherence to the rules in play.

As an example, in what I call the "License Compliance Shakedown", the DRM software used does not prevent usage outside the license usage terms. A year, two, three go by, and an audit is conducted and "true up" fees are demanded at threat of lawsuit.

Flexera pioneered this model, and works with their customers to adopt it. I would say it is a small but not insignificant part of what is pushing companies towards open source.

drewg123• 4 years ago

At a previous employer, we had an expense reporting system that required that every expense have an image of a receipt. So lots of people would loose receipts, and would then just include a photo of a napkin with "I lost the receipt" scrawled on it..

908B64B197• 4 years ago

However, the staff that built our course revision system decided to make it so that every single change should go through all layers of approvals. So if I want to change the title of an assignment, according to this tool, I need the department to approve.

I have to wonder if the rule wasn’t put in place a long time ago after someone (tenured) tried sneaking in something truly controversial in the curriculum he planned to teach. Dr. Lemire doesn’t teach a controversial subject -at least having an opinion on vectorized instruction sets won’t get you on the news- and sounds like the type of researcher that values teaching a lot so I assume he frequently updates his course material and must be annoyed by this.

Keeping in mind that recently a university close by (lower caliber but still) fired a professor for saying the N-Word in a classroom as part of a required reading during a race and literature class [0]. Maybe it’s the king of backlash they want to avoid.

An actual policeman can be intimidating. However, it is a human being. If they ask something that does not make sense, you are likely to question them. You are also maybe more likely to think that a policeman could be mistaken. Software is like a deaf policeman. And people want software to be correct.

It’s ironic because the Province he’s writing this from had policeman enforcing a mandatory curfew (because of Covid of course) and would slap fines on people being outside after the designated hours. No matter if they were simply walking alone or their dog.

To anyone with high school level biology that should sound a little ridiculous. Walking alone you can’t really transmit or catch an airborne disease. Interacting with a policeman sure.

I think that there is great danger in some of the recent "digit ID" initiatives that various governments are pushing. Suppose, for example, that your driver's license is on your mobile phone. It seems reasonable, at first, for the government to be able to activate it and deactivate it remotely. You no longer need to go to a government office to get your new driver's license. However, it now makes it possible for a civil servant to decide that you cannot drive your car on Tuesdays. They do not need a new law, they do not need your consent, they can just switch a flag inside a server.

Again, the author’s country just demonstrated the slippery slope by arbitrarily freezing assets and bank accounts on suspicion of being friendly to a group of (first amendment protected) protesters. Instead of following the legal pathway of demonstrating a reasonable reason to a (non-political) judge to get an account frozen, the government just gave themselves emergency powers to do as they please. No oversight. No reasonable doubt? Doesn’t matter.

What’s next, someone has a black sounding name and too much money in his account? Freeze it, most likely crime money.

[0] https://globalnews.ca/news/8450947/quebec-university-academi...

Enforcement by Software

Related Stories