The Paradox of Source Code Secrecy (2019)

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3409578

by BerislavLopac • 6 years ago

71 18 6 years ago

papers.ssrn.com

Today, the government relies on machine learning and AI in predictive policing analysis, family court delinquency proceedings, parole decisions, and DNA and forensic science techniques, among other areas, producing a fundamental conflict between civil rights and automated decisionmaking.

Most of the methods used for this are basically large matrices with no human understandable way to represent them. Getting the values would only let you recalculate the results, not discover why the results were produced.

phh• 6 years ago

Getting the values would only let you recalculate the results, not discover why the results were produced.

It also allows you to calculate new results, which can be used to detect when the system is actually wrong, and to detect bias.

And actually, yes you can discover why the results were produced, because models leak a ton of data about the training set.

michaelmrose• 6 years ago

You presumably can't get anything at all without very expensive expertise on top of the expensive legal expertise that a lot of people already can't afford.

The simple and affordable solution is to ban the use of artificial stupidity in legal decision making.

krona• 6 years ago

Climate models and recent epidemiological modelling (outliers in terms of their effect on public policy making) are far simpler than say the ML this article seems to suggest. But you don't need a human-understandable representation to understand their limitations.

p_l2• 6 years ago

Climate models don't use ML, similarly, most sensible epidemiological modelling doesn't.

ML is, afaik, considered too much of "curve fitter" to actually simulate or analyze anything novel instead of rehashing of past events, making it useless in cases like this.

Gunax• 6 years ago

Humans all ready are black boxes.

IMO, if you're worried about a black box AI juror, we all ready have that--they are called (human) jurors.

gerland• 6 years ago

Putting aside the topic of how easy is it to understand a trained ML model, I really hope we won't end up in a SEO-like situation.

umvi• 6 years ago

Putting aside the topic of how easy is it to understand a trained ML model,

I wouldn't say it's easy to understand a trained ML model. If you don't have access to it, it's a black box that can be hard to penetrate. But if you do have access to it, it's fairly easy to figure out how to trick it into doing what you want.

Example scenario: in a dystopian 2030 America, judges are replaced by ML models where you just input a bunch of parameters and the model spits out the judgment. Soon:

Divorced Husband: "Why do you keep calling the non-emergency police line to concern troll?"

Divorced Wife: "So the ML judge will give me custody of the kids"

(number of calls to police as an input to ML model is being gamed to alter outcome in one party's favor)

TheOtherHobbes• 6 years ago

You've just invented a new service industry: AILegal, a combination of legal expertise to play the human side, and ML expertise to play the machine side.

Probably available by subscription for a monthly fee that corresponds to the probability of a positive result.

lliamander2• 6 years ago

The Anglo-American common law system, by virtue of its emphasis on legal precedent as a way of shaping law, is a sort of trained ML model, implemented in a social institution.

gazzini• 6 years ago

The gist:

IP law lets software have the best-of-both-worlds. Source code is patentable & protected by copyright, but is also a trade-secret (because it's not publicly disclosed).

This is problematic when these black-box algorithms are used to assist in the processing of legal cases.

-----

My 2 cents:

- The overall concern that our lives are being governed by the invisible hand of black-box abstractions really resonates with me. Richard Stallman has been warning us about this for a while [0], and I've personally started considering my role in this as a software engineer [1], although I don't have any actionable advice yet.

- I don't feel as strongly that all of this code should be open-source. I think it might be sufficient to just strictly define a series of tests -- measure the inputs & outputs, describe the requirements... that kind of thing.

It's inherently difficult whenever there's a constantly-evolving ML model behind the curtain. It reminds me of the issues with using AI in the aviation industry, for instance [2]. As I recall from my undergraduate classes, aviation control systems have been unable to use "fuzzy" logic to pilot planes due to strict deterministic testing requirements.

[0] https://www.gnu.org/philosophy/free-software-even-more-impor...

[1] https://gazzini.com/essays/posts/tools/

[2] https://www.nlr.org/areas-of-change/introduction-non-determi...

jka• 6 years ago

For your consideration: https://en.wikipedia.org/wiki/Diesel_emissions_scandal

Cerium• 6 years ago

For actionable advice, I believe that the government should make it a first choice priority to use and improve open source software. Open software is digital infrastructure and critical in the success of our society.

ChuckMcM• 6 years ago

I largely agree with this. I have suggested in the past that the mission of the US Digital Corps be amended to include the development and maintenance of infrastructure critical software in a transparent way.

Imagine for example if the software used in voting machines was managed by such a system. Always published, always analyzable, maintained and authenticated by people whose job is only to make sure it does what it is supposed to do.

D13Fd• 6 years ago

In my view, the fundamental problem is that software doesn't fit neatly into either the patent or the copyright regimes.

Patents are not supposed to cover abstract ideas. In reality, all software is abstract.

Copyrights are supposed to cover an expression of an idea. It is not meant to cover inventions or functional devices.

Patent terms are relatively short, 20 years from filing, because technology needs to advance. It's great that you invented a drug to cure cancer, but you shouldn't be able to own it forever.

Copyright terms are functionally infinite (currently the life of the author plus 70 years, or 120 years for corporate works, but likely to be extended going forward).

It doesn't make sense that software is most similar to inventions protected by patent, but gets the essentially infinite term of protection of copyright.

I would propose that Congress pass a law limiting the term of copyright protection for software (object code, source code, or otherwise) to 20 years from date of creation. That puts it in a middle ground between patent/copyright, where it should be.

Trade secret is fine as it is.

x3blah• 6 years ago

For those not using Javascript (SSRN uses third party scripts), link to PDF: https://poseidon01.ssrn.com/delivery.php?ID=5550640311191050...

The Paradox of Source Code Secrecy (2019)

Related Stories