nickvincent

I think the reaction to this protest is definitely a very bad look, but that it's too early to say how that partnership panned out. Still hopeful!

I think it's pretty reasonable that many users might perceive this as a betrayal of an implicit social contract. People posting things ten years ago, or even a year ago, shouldn't have had to make projections about future modelling advances / how data might be used in the future.

IMO: Renegotiating for "public approval" will always be something computing organizations have to engage with when making advances in tech that's downstream of publicly created records, knowledge, etc.

(That being said, I personally think this announcement is overall a win for healthier data flow, in particular because these kind of deals make the value of data explicit.)

Yeah, that's what I'm basically thinking: for scientists who already like Python because it's readable and mostly matches the mental model of what they care about, being able to catalyze the switch to a different, faster language that still reads nicely could really drive adoption. But it's a stretch for sure!

I'm quite hopeful that language models for code might actually lead to more people getting comfortable with multiple languages and especially some kind of hopping back and forth. I think the prospect of starting off your script in Python (which is a "comfort food" for many, including myself), using LLM to translate over to Julia, probably do some minor debugging, and then taking off from a starting point instead of a blank file could be a nice workflow for people who want to dip their toes in.

I think it's related to Reddit users who posted (very frequently!) on a counting focused subreddit (people literally post "1", "2" , "3" in sequence so usernames appear 50k+ times). Some screenshots and links in this Twitter thread: https://twitter.com/SoC_trilogy/status/1623118034960322560

Plus additional commentary here: https://twitter.com/nickmvincent/status/1623409493584519168 (in short: I think this situation is comparable to a "Trap Street" https://en.wikipedia.org/wiki/Trap_street that reveals when a map seller copies another cartographer)

I hadn't seen the Twitch plays pokemon hypothesis though (from another comment here), I wonder if it could be both!

Basically agree -- feels off-putting, but not technically a wrong detail to add. An additional reason it rubs me the wrong way, however, is that I believe open-source software code is especially critical to ChatGPT family's capabilities. Not just for code-related queries, but for everything! (e.g. see this "lineage-tracing" blog post: https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tr...)

Thus, I honestly think firms operating generative AI should be walking on eggshells to avoid placing blame on "open-source". Rather, they really should going out of their way to channel as much positive energy towards it as possible.

Still, agree the charitable interpretation is that this just purely descriptive.

IMO - this kind of tool cuts across a debate that currently involves a lot of people yelling past each other.

If you're an artist, of course you can make changes to your process or your content that modify how it's used. To argue otherwise is analogous to arguing we must always be mindful to stay in the surveillance cameras' view when walking about the streets.

Yes, it's a never-ending back-and-forth game that the obfuscater will probably lose in the long run (though abstractly, an obfuscation technique with >50% adoption could "win" long-term) . And yes, it's important to stay apprised about how effective such tools are.

But in the short term, the existence of these tools provides a critical counter-measure to the current narrative, which is basically that everything that can be scraped will be scraped. Returning to the cameras & streets analogy, obfuscation tools are maps that tell us about routes out of view of the cameras (even though these routes may often be blocked off or inconvenient).

Whether you hate AI art or love it, I honestly believe both sides can get behind understanding obfuscation and poisoning and making tools available: those opposed will use the tools, those who want to improve generative AI can learn from the counter-measures, etc. This kind of thing can be part of a healthy deliberative process around these emerging technologies.

It's an open challenge, this preprint proposes a solution for robust data valuation. https://arxiv.org/abs/2205.15466

However, I'm not aware of anyone actively trying to adapt these techniques for large generative AI models (though would be great to see).

Yes, this doesn't use attribution techniques like influence functions or Shapley values that are popular in machine learning research, but I am pretty convinced that even a nearest neighbors search is better than the current baseline offered by "AI art systems": shrug our shoulders and say nothing about the role of human-created training data in producing the outputs.

As far as I know, nobody is even thinking about doing the very expensive experiments needed to get ground truth data for formal attribution techniques in the generative AI context (for a given prompt, retrain your model so you can see how the output changes when a particular training example or group of examples is omitted or added), so we're nowhere near building true attribution systems for these very large models. Centering the training data will be net good for public discourse on the topic.

That said, I see why people want to push back on some of the language used here.

Appreciate the distinction in the above comment that they are two distinct questions, but also agree the two questions are very connected.

I should've been more specific: I was thinking mainly of the artists v. stable diffusion lawsuit which makes the specific technical claim that the stable diffusion software (which includes a bunch of "weights files") includes compressed copies of the training data. (Line 17, "By training Stable Diffusion on the Training Images, Stability caused those images to be stored at and incorporated into Stable Diffusion as compressed copies", https://stablediffusionlitigation.com/pdf/00201/1-1-stable-d...).

I expect that if the decision hinges on this claim, that could have far reaching implications re: model licensing. I think this along the lines of what you've laid out here!

This is a great point.

Not a lawyer, but as I understand the most likely way this question will be answered (for practical purposes in the US) is via the ongoing lawsuits against GitHub Copilot and Stable Diffusion and Midjourney.

I personally agree the creativity is in the source images and the training code, but think that unless it is decided that for legal purposes "AI Artifacts" (the files containing model weights, embedding, etc.) are just transformations of training data and therefore content and subject to the same legal standards as content, I see a lot of value in trying to let people license training and code and models separately. And if models are just transformations of content, I expect we can adjust the norms around licensing to achieve similar outcomes (i.e., trying to balance open sharing with some degree of creator-defined use restriction).

Yeah, that's a fair critique, I think the short answer is depends who you ask.

See this FAQ here: https://www.licenses.ai/faq-2

Specifically:

Q: "Are OpenRAILs considered open source licenses according to the Open Source Definition? NO."

A: "THESE ARE NOT OPEN SOURCE LICENSES, based on the definition used by Open Source Initiative, because it has some restrictions on the use of the licensed AI artifact.

That said, we consider OpenRAIL licenses to be “open”. OpenRAIL enables reuse, distribution, commercialization, and adaptation as long as the artifact is not being applied for use-cases that have been restricted.

Our main aim is not to evangelize what is open and what is not but rather to focus on the intersection between open and responsible licensing."

FWIW, there's a lot of active discussion in this space, and it could be the case that e.g. communities settle on releasing code under OSI-approved licenses and models/artifacts under lowercase "open" but use-restricted licenses.

Thanks! Looks like a bunch of changes were made to page structure and CSS (in a way that makes scraping a bit more inconvenient). Filed an issue for myself, but looks like the other suggestions in this thread may be better approaches to avoid this kind of fragility.

Super cool idea, thanks for building and sharing!

One possible suggestion for future iteration: I think it's a bit of a shame that everyone's been sharing ChatGPT outputs almost exclusively through screenshots. I threw together a (very quick and rudimentary) browser extension I've been using to save some of my more exciting transcripts in JSON files. I think something along these lines could be really great for sharing purposes, especially if people want to study these outputs more systematically (e.g. for research, or just a kind of crowd-sourced audit).

Here's the barebones extension, just as an example: https://github.com/nickmvincent/chatgpt-exploration/tree/mai...

There's also a longer discussion to be had about the best way to do this: ideally, we might save some of the formatting information too and/or more metadata (exact timestamps for responses, etc.), but I think the json with plaintext is adequate (at least for personal retrieval use).

Also, based on all the public info about InstructGPT (the closest ChatGPT "family member"), all of StackExchange is definitely in the training via OpenAI's "filtered Common Crawl", if it isn't also included as a special over-weighted training set (English Wikipedia, for instance, was over-weighted in GPT 3 training).

There's a big concern in the long run: if ChatGPT seriously reduces the number of people who visit StackExchange domains, there will be no dataset no for GPT 4 / ChatGPT 2. E.g. what if a brand new programming language gains popularity, or new libraries with very different patterns of use?

This "paradox of reuse" is a really big deal, IMO (blog post on the topic: https://nmvg.mataroa.blog/blog/the-paradox-of-reuse-language...)

It's actually a separate concern from the spam / content bloat described in the linked post, but they complement each other in creating amplified harm for StackExchange-like platforms: some fraction of users may stop visiting the site (because ChatGPT answered without links) and some other fraction of users will submit spam to try and earn points.

Also curious... I spent a while trying to get system to tell me directly, but no dice: https://twitter.com/nickmvincent/status/1598478685019189248?...

It gives a generic answer that it's some proprietary combinations of "books, articles and websites". I'd guess Wikipedia is in there for sure (English and maybe other editions as well), something like "BookCorpus" (https://huggingface.co/datasets/bookcorpus), probably a large scrape of news articles up to 2021. And definitely a full scrape of pretty much the entire academic/scientific literature (just based on poking around). Overall, probably very similar to GPT-3 (which is also a bit mysterious still!)

The official post (https://openai.com/blog/chatgpt/) also describes that some pretty rich human feedback data was collected as well, for the reinforcement learning component. I think this probably the real secret sauce for why this feels so qualitatively different than a lot of the LLMs that came before.

Yeah, I think this is spot on. The version of WMF Enterprise being described here is formalizing / codifying an informal relationship that already existed (Google using Wikipedia content all over the place, and making various one-off donations to WMF), and as you say, formalizing has benefits for both parties! Certainly ways this could have some long-term negative impacts, but WMF is obviously thinking pretty hard about mitigations, it seems.

The discussion linked in a comment above (https://meta.m.wikimedia.org/wiki/Wikimedia_Enterprise/Essay...) also provides a pretty nice FAQ to the negative responses.

In practice, I believe the reason is historical: that's how Wikipedia started and it hasn't changed. But there's pretty compelling evidence that it provides Wikipedia some unique benefits relative to an account-locked alternative:

Hill, B. M. and Shaw, A. (2021) ‘The Hidden Costs of Requiring Accounts: Quasi-Experimental Evidence From Peer Production’, Communication Research, 48(6), pp. 771–795. doi: 10.1177/0093650220910345.

Crazy impressive! A question about the training data: anyone familiar with this line of work know what social media platforms the "conversation" data component of the training set came from? There's a datasheet that points to prior work https://arxiv.org/abs/2001.09977, which sounds like it could be reddit, HN, or a similar platform?

Yeah, that's fair! My view is not that everybody should use Rally, or even Firefox. Rather, it's that even though there's going to be massive filtering going on with regards to who will even contribute data to Rally, it's still going to benefit the public to have biased quantitative estimates of parameters of interest, and to surface qualitative findings from the group of people who do contribute.

It's always possible in the future that Rally, or other initiatives, will find other ways to incentivize contributions (more projects that align with specific interests, more "visualization" of how your data contributions might impact the world, etc.).

As to whether the cost-benefit accounting (producing new knowledge, informing design vs. scaring users off Firefox, creating new privacy harms) will come out in favor of the project remains to be seen, of course!

I understand there seems to be a lot of pent-up frustration in these comments, some of it related to this specific initiative and some not.

I'd like to highlight a few reasons why this could be exciting and, indeed "pro-user".

* The core value proposition in my view, though not spelled out on the linked page (I think because it's targeting a broader-than-HN audience), is that data collection can produce answers to empirical questions about "Big Tech" that would otherwise likely never be answered by anyone except in-house Big Tech data analysts. E.g., empirical questions about the impacts of search engine design choices, how newsfeeds influence behavior, etc. Of course, these studies will face issues with selection bias, but that's not insurmountable, and probably better than nothing. For instance, publicly shared data about Pixel tracking, even from a biased sample, is better than having no data on that topic, IMO!

* The collaboration with academics and journalists is important because it means the results of each study will be shared. It's in the incentive systems of both groups to publish results. Best case, individual users, people who build browser extensions, etc. can use findings about search engines and newsfeeds to change their behavior or build new tools. Policy discussions would also benefit from some quantitative grounding.

* The alternative ways to answer these questions: academics and journalists run scrapers, data-collection browser extensions, etc. on their own. Harder to do, especially post-Cambridge Analytica and more recently, NYU's ad observatory getting shut down (https://cyber.nyu.edu/2021/08/21/facebook-disables-ad-observ...). Or, the state forces tech co.'s to publish answers. Or, we just never know!

* A weaker point: This project is trying to set new standards about data management, e.g. providing a true "one click to opt out" option. I see this as an experiment, that other projects can follow if it works. This is a bit more of a stretch than the previous two points.

My bias: I'm an academic who wants to see the results of the various studies being conducted, so I'm very hopeful!

Definitely a challenge for many mechanism design approaches to intervening in the world. As I understand, the only really solution is to experiment in the real world, see what holds and see what doesn't, and tweak accordingly (e.g. test out different interfaces that communicate how the mechanism works!)

Worth emphasizing that, as the article states, there's a lot of ongoing experimentation with quadratic voting. Wikipedia has a good overview (https://en.wikipedia.org/wiki/Quadratic_voting#Applications), but users include Colorado's House of Reps and several civic platforms in Taiwan.

The core argument for x^2 is basically just that it's the only function with linear derivative, so for rational voters who calculate marginal benefit of additional votes purchased at cost x^2 (in terms of likelihood of changing the outcome of the "election"), you get a welfare maximizing outcome.

There's some assumptions underlying the theoretical result, but there's also a lot active experimentation in the real world, a few links: https://www.radicalxchange.org/concepts/quadratic-voting/, https://en.wikipedia.org/wiki/Quadratic_voting

Here's what I believe is the key paper on the topic https://www.aeaweb.org/articles?id=10.1257/pandp.20181002 (appendix: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2790624)

Lalley, Steven P., and E. Glen Weyl. 2018. "Quadratic Voting: How Mechanism Design Can Radicalize Democracy." AEA Papers and Proceedings, 108: 33-37.

IMO, the best mental model is that this gives a more formal way for Google or Apple to donate to Wikipedia. Wikipedia's mission of free knowledge is unchanged, but hopefully tech companies can provide financial support in a structured, reliable fashion. Ideally, it's even positive sum (i.e. the Wikipedia-dependent services can perform better, so that the efforts of Wikipedia volunteers impact more people).

Worth emphasizing that this is described as basically just adding new ways to consume Wikipedia at scale, not taking anything away (not that that's impossible in the future, but unlikely IMO). In theory, systems that use a lot of Wikipedia data (search engines, "AI" assistant/question answering systems, language models) can keep doing what they're doing with Wikipedia dumps, but new APIs could make these services better (e.g., faster reaction time to current events).

A big upside of testing this out is the opportunity to keep better track of what services are Wikipedia-dependent. While search results often send people to Wikipedia (see e.g. https://diff.wikimedia.org/2021/09/23/searching-for-wikipedi...), voice assistants or Q&A systems might be (i.e. likely are) surfacing Wikipedia content in more subtle fashion. Same with large language models. Under status quo, it will be increasingly hard to track these relationships, so exploring alternatives is well worth it.

It just isn't true that "The point of ethics is to shame people or practices" or that ethics has "no value production".

In general, a primary factor reviewers in computing conferences are asked to consider is the degree to which a submission make a "substantial contribution" to the community. What is or isn't a substantial contribution is subjective and entirely dependent on the prevailing ethical perspectives in various communities. Papers -- a key unit of academic progress (for better or worse) are entirely subject to concerns of ethics. Certainly, there are interesting argument around how much time and paper space should be spent on speculating about negative impacts, and people are having those conversations.

The fact that ethics is "trending" is because more researchers would like tackle ethical challenges explicitly, instead of falling back on the default of a given community. For instance, here is a paper that quantitatively (with reproducibility!) analyzes the values in ML papers. https://arxiv.org/abs/2106.15590 This is one way to have a very empirically-grounded discussion of the topic.

IMO, many researchers can and do debate topics of ethics in AI, and in doing so move the field forward (and increase likelihood that computing will have more positive impacts than negative ones).

Can't say I agree this is "plain and simple", as the way you're defining value here seems to be "might makes right".

I guess if that's what people believe, it may be tough to convince them otherwise... but IMO, it's really valuable to think about how the "winners" of history navigated tough moral challenges, and often did things that we can now identify as harmful.

In this case, it's not about making historical missionaries feel bad about themselves for "winning" -- this article (in my read, etc) is concretely about trying to avoid repeating these mistakes in modern tech, specifically NLP and translation. We can (hopefully) do better as a society than "might makes right"!

I didn't read this as "slamming" missionaries so much as asking people to reconsider the framing of language-related actions under the frame of power. That is -- many people do see what Christian missionaries did as a uncomplicated social good, for reasons you laid out, but we should also consider outcomes like the decline of a language, and most importantly the role of power/consent. I thought the missionary example made for compelling comparison w/ modern day NLP work.