Stop measuring community engagement

https://rosie.land/posts/stop-measuring-community-engagement/

by rosiesherry • 4 years ago

296 116 4 years ago

Even though it’s the energy they’re after, not the screams, it’s the screaming that was their primary metric, and everything about their company was organized around increasing that.

Missed opportunity to bring it full circle back to this.

Otherwise, not bad.

jpster• 4 years ago

This sounds great, but I see a challenge. The author recommends measuring e.g. user happiness instead of engagement. How to measure user happiness? IMO, some type of survey could be used. But the “best” metrics are based on observable behavior. Not what users self-report, because they may not know how they actually feel about the experience. In general it’s better to see what users do rather than what they say.

Bjartr• 4 years ago

Observables are better, but self reported metrics do still have value. It's an aggregate version of the problem devs encounter when doing usability testing. Users will tell you what their experience is, but it's up to you as a software designer to distill that feedback into actual needs. Which may not at all be what they actually ask for. Survey responses are similar in that they don't tell you exactly what to do, but they do contain actionable information if you have the skill to interpret it.

ItsMonkk• 4 years ago

You must have an expert or experts. That expert will then make decisions, and use metrics and surveys and intuition to drive their new world view, but it's the expert that is always at the top of the stack.

If you then take that model and try to use them in a new domain, that is never going to work out. The environment adapts to the incentives that the current system produces, and will instantly make your pre-created model worthless[0]. The only way to work out long-term is by having an expert stay ahead of the pack creating what they think are systems with better signal to the noise.

[0] https://dilbert.com/strip/1995-11-13

mhall11910• 4 years ago

I'd recommend reading The Community Engagement Trap[0] by Rosie Sherry (which should also be shared here on HN), she talks about Community Discovery as the process for getting these deeper understandings.

[0] https://rosie.land/posts/the-community-engagement-trap/

burlesona• 4 years ago

Good article, though I think it missed explicitly making the point that the reason "engagement" is the metric is because that's what platforms monetize. Engagement is a euphemism for attention, and social platforms exist to sell your to advertisers.

Since the article is written for "community builders," understanding that engagement is a metric to quantify the advertising potential of a platform should make it obvious that people who aren't in the business of selling attention to advertisers don't need to copy social media and optimize for engagement; instead they should target better measurements of value creation for their community. Of course Goodhart's law[1] still applies.

1: https://en.wikipedia.org/wiki/Goodhart%27s_law

(edited for readability)

cyborgx7• 4 years ago

Exactly. It took me a while to understand that this article was written for community managers of brands, not people developing social networks. They keep saying engagement has no value, but to social networks and to "influencers" who both make their money off of ads, that is exactly what has value.

phkahler• 4 years ago

> Good article, though I think it missed explicitly making the point that the reason "engagement" is the metric is because that's what platforms monetize.

In the case of "community" around a product, the purpose of the community is usually to provide free support. That's not so much monetization as it is cost savings. For that to work, people need to be "engaged" or kept around so they are present to field those support issues.

sfink2• 4 years ago

Engagement is still the wrong metric for this purpose, because flamewars rate much higher than single comprehensive answers. In fact, partial or incorrect answer rate better, because the questioner will have to come back or someone will need to post corrections or clarifications.

Engagement is a better proxy for value destruction, if you define people's time and sanity as valuable. (It's not perfect; a thriving community will rate highly.)

throw10920• 4 years ago

the reason "engagement" is the metric is because that's what platforms monetize

...and the implication of that is that if we change the monetization model, then the metrics also change.

For a company selling a paid product, metrics become things like the effectiveness of the sales funnel, how much support the average user requires, the rating of the product on app stores, etc - which are not ideal, but still far better than "engagement".

crazygringo• 4 years ago

You might be surprised to discover that even for paid products, engagement is often still the #1 metric.

Simply because people who are engaged with a product/app/whatever are by far the most likely to be repeat customers -- to renew their subscription, to buy the new version, to adopt the next product.

Once engagement drops, future revenue dries up. A drop in engagement is the canary in the coal mine -- it shows where your business is going wrong months/years before it shows up in sales stats or renewal rates.

So actually, the metric doesn't change at all. E.g. Netflix is obsessed with engagement even though it's subscription-based rather than ad-based. Because guess who winds up cancelling their subscriptions?

hinkley2• 4 years ago

Engagement is just a more sophisticated version of 'eyeballs'.

For community building you want successful events, you want interesting questions to be asked, and interesting answers to be offered. Someone could write a book on how to make communities work. Probably several books.

mhall11910• 4 years ago

There are several books on precisely this.

Theodores• 4 years ago

You are completely right on Goodhart's Law. I know this from experience.

I put the customer first in the belief that 'the rain follows the plough' and, if you put the customer first, then all else follows.

I see metrics as there to be cracked. If you are measuring the right things then you can see where the problems are, fix the problems and move on to measuring different metrics that are more specific and focused on the remaining problem areas. For me it is a campaign where initial reactive firefighting gives way to calm, proactive problem solving, fixing problems before they are of consequence. Along the way many unexpected things are learned about the customer.

I do all this with a focus on the customer, meanwhile my colleagues in marketing are out for themselves. They are the dead weight keeping progress back as they have to do things like buy traffic and hold on to keyword optimised URLs that are cargo cult SEO. Their reports for their micromanagers are always measuring the same stuff, e.g. 'engagement', yet they could be engaging customers solely for the purpose of pissing them off for them to never come back.

My favourite is the pop up shown as someone leaves a website begging them to stay, or the sign up to the newsletter popup. These could get one extra sale at the cost of pissing off a million people but if you only measure the former then it all seems good.

In the capitalist world only one metric matters to the shareholders. Once I worked for a highly successful company that sent out sales newsletters with no measurement whatsoever of open rates or click throughs. None of that was needed. The problem was emptying the warehouse too quickly to have no stock left. Customers would be waiting for our email, not ignoring them. We had plenty of engagement with the customers as they actually bought stuff.

Subsequently I have worked for agencies where they spend a fortune on some email service and they get these fancy graphs to show the clients of these open rates the world over. Which had wow value at the time and 'engagement' stats. But I had come from somewhere where we had no need to measure that stuff, we weren't looking at these charts, we were coping with a deluge of orders and had no time for that.

bluGill• 4 years ago

Unfortunately what is really needed here is a suggestion of something else to measure that is better. I fight this a lot with bad metrics (code test coverage), nobody wants to give them up unless I can suggest a replacement to measure.

mhall11910• 4 years ago

Here's a couple:

Connections between people: https://www.savannahhq.com/2022/09/06/measuring-your-communi...

Support: https://www.savannahhq.com/2022/09/08/measuring-your-communi...

fullshark2• 4 years ago

I enjoyed the Monsters Inc reference and the turning of it on its head central to this piece. But we've said/argued this for years and at the end of the day engagement = ad reach = profits and until the economics there change any for-profit social media will gravitate towards engagement. Hell ANY media gravitates toward engagement and outrage bait. An actual community built around Mastodon seems possible, and there's community discussions around private group chats but they will always remain niche and not profitable.

I don't get what https://savannahhq.com is and what the value add is there but that seems to be what he's interested in, niche + small communities.

snarf21• 4 years ago

Well said! Outrage sells and it is easy to keep it stoked. Everyone is addicted to free. If we get rid of ads, how does (even the barest bones) Twitter keep the lights on?

I wonder if there is a cross-over between Patreon, Substack and Twitter. This would probably work for niches but not writ large. But at the end of the day, we need to find a way to pay for what we value because that is how we get more of it. Interested in new ideas in this space....

ethbr03• 4 years ago

The Medium/Spotify model -- per-user constant subscription fee, apportioned to content creators on the basis of their views + some time-viewing / read-to-end adjustments to discourage clickbaiting -- seems fairly decent, but really requires a global platform to work.

But product companies don't like it because they can't scale revenue user-transparently by repackaging data (e.g. building new / better ad products), and raising prices for users sucks.

The original sin is that the web was free, and the consequence of that, once post-content-scarcity in a free market system, has been continual sleight of hand to make money from "free."

If everyone has gotten in the habit of paying for quality content... we'd have a very different world now.

(And this said as a hard-charging "information wants to be free" type. But the end commercial result nowadays is terrible)

kevincox• 4 years ago

I think the problem is that while engagement may be the source of profits that is short-term thinking. If joy based engagement drives profits and keeps users on your site long-term it may be worth it to take the short-term revenue boost that anger-based engagement brings you if it slowly drives people away from your site.

Of course it isn't completely clear if anger-based engagement actually drives most people away. In that case it is actually Facebook's fiduciary duty to capitalize on it which seems to be a failing on our society.

marcinzm2• 4 years ago

Facebook is still not shrinking in the US in terms of users. So even if there is a long term cost it seems well beyond the time horizon companies care about.

mhall11910• 4 years ago

Community isn’t social media.

You don't advertise to your community. If you do you're actively killing it.

Community is made up of people making connections and building relationships with other people. Savannah CRM helps you manage those connections and relationships.

fullshark2• 4 years ago

Yes I agree, I think this is basically the point I was trying to make distilled.

puchatek• 4 years ago

This. And also this:

We still want our members to be happy, right?

... we might want that, but do the people running the platforms?

lucideer• 4 years ago

Nicely written post but can't help noticing the elephant in the room being tiptoed around, which really makes me question the sincerity of the author.

The monsters need energy to power their civilisation. Why do social media companies measure engagement? What are they extracting? There is not one single mention of the "p word" in the article.

The last two sections are delivered with such heavy doses of naivety that this ends up just coming across as Orwellian in tone.

“We should measure what we value, not value what we measure”

Who is "we"? Is "we" the shareholders? What do "they" value do you think?

forbiddenvoid• 4 years ago

I like the premise of the post. I think it paints a rather overly idealistic view of social networks, who exist to make money first, not to create value (if they could make money without creating value, they certainly would do that).

alexb_2• 4 years ago

Now imagine if this played out the other way around. Imagine that the monsters originally focused on maximizing energy by making children laugh as much as possible. Then one day, in a horrible twist, someone discovers that a child’s scream was a much easier and more powerful way of meeting their goals. Their company, which built a platform that gave them instant access to children around the world, suddenly realizes that they could reach their goals by spreading fear rather than joy.

Then they would instantly start using screams? Nobody making business decisions in social media companies will prioritize "user happiness" over increased production of their main product (your attention) for their customers (advertisers).

egypturnash• 4 years ago

Congratulation, you grasped the intent of that analogy!

alexb_2• 4 years ago

Maybe I'm reading the intent wrong, but no social media company will ever take this suggestion. This line in particular:

Let’s bring this back to community building. We still want our members to be happy, right?

If it doesn't make them more money... no they don't!

catherd• 4 years ago

Some will and some won't. In the absence of rules that penalize that type of behavior, usually the ones that will end up winning. It has nothing to do with morals, those just seem to be the rules of the game as we currently play it.

efitz• 4 years ago

The article dances around a topic that is very relevant across technology companies- toxic metrics.

I worked for many years at a company that prides itself on being metrics driven. To the point that they celebrated having many significant figures after the decimal point when measuring length of negative events in seconds.

The good side of metrics driven culture is that having a metric gives you a concrete goal.

The downside is that really good metrics are often very hard. By really good, I mean metrics that measure the desired outcome.

It’s much easier to measure, for example, how many operation X your service performed, rather than the value the service delivered to your business or your customers. So we assume that each operation X contributes a uniform, positive quantum to those outcomes and we count X’s.

godelski• 4 years ago

I don't think they dance around the topic so much as outright state that it is bad. They mostly talk about engagement because it is the primary metric but even state that we should not value what we measure but measure what we value. Which it is harder to do that, but hey, you got billions of dollars to work with, it seems like the resources are there.

JohnFen8• 4 years ago

The software industry has made me very aware that being metric-driven can be a terrible thing as easily as a good thing. Software has been getting substantially worse over time (in my opinion) in large part because of the use of metrics.

FalconSensei• 4 years ago

Goodhart's law basically also.

I've seen ppl on Twitter posting 'unpopular opinions' that they don't even agree, just to get replies. And then admit on the conversation that they don't agree with that.

hinkley2• 4 years ago

Toxic metrics are a subset of perverse incentives. Necessary, but not sufficient to stay out of trouble.

slackerIII• 4 years ago

Early on in a community, there often aren't enough data points to cover other possible metrics. With that in mind, starting somewhere is better than not starting at all — measuring engagement is a useful way to begin understanding what's resonating in your community. As you continue building your community, there are many different angles that you should use to evaluate its health.

Measuring engagement then becomes one piece of data that's valuable, but shouldn't be the only piece. When engagement data is combined with qualitative community surveys, enthusiasm from members to contribute to a community, clear and timely responsiveness to community needs, membership growth over time and geographies, depth of member interactions (whether across community channels or within specific channels), what's topically important to members and why, and overall sentiment and change in sentiment over time - that's when community builders can begin to better understand the health of their communities and their impact on their business.

Engagement is an important piece, but just one of the pieces, that helps paint the full picture.

Disclaimer: I’m a co-founder of Common Room [1] and a we’ve invested a lot of energy in solving for this exact problem. You’re welcome to check out the product (it’s free to sign up) and would love to hear of ideas or feedback.

1: https://www.commonroom.io/

nmilo• 4 years ago

I don't get it. What is a "community manager"? Is it someone that runs a company's social media page? And do they think that people care about their brand so much that they would join a community of people whose only common trait is liking the same brand? And in the first place, how do you even build a "community" manually and artificially? It's almost a bastardization of the word, as if a community is not a set of people with similar goals but instead yet another way to convert people's time and desire for social interaction into profit.

The article implies that community building should be some kind of altruistic purpose, where your only goal is to maximize the amount of "meaningful relationships" created? But building an artificial community in the first place can never be altruistic because the end goal of it all is to guide people to your product or conference or whatever. If you were an actual altruistic community builder, you would be telling people to get off the Internet and go make "meaningful relationships" with people in real life.

mhall11910• 4 years ago

I don't get it. What is a "community manager"? Is it someone that runs a company's social media page?

No, that's a social media manager, completely different role.

And do they think that people care about their brand so much that they would join a community of people whose only common trait is liking the same brand?

Yes! It happens all the time. Most of the biggest brands have a community of people who share that common trait.

And in the first place, how do you even build a "community" manually and artificially?

Again yes, just like you build a garden. The gardener doesn't make the plants grow, but they do make sure they have the right environment and resources to grow in.

The article implies that community building should be some kind of altruistic purpose, where your only goal is to maximize the amount of "meaningful relationships" created?

That's not altruism. Those meaningful relationships help the company/project the community was built around. They provide feedback, help improve processes, make connections outside the company/project, provide support to other users, there are too many things that communities do to list them all here.

JohnFen8• 4 years ago

That's not altruism. Those meaningful relationships help the company/project the community was built around.

Which brings up another aspect of this whole issue -- we're talking about companies figuring out how to exploit the human need for community in order to increase profits. The companies themselves are not encouraging real community, they're farming people. Companies are not people, do not have the best interests of people in mind, but pretend like they are.

It's very distasteful at the least.

wsb_mod2• 4 years ago

The profit motive is being blamed here for optimizing for engagement.

But how do you assess progress when you remove all desire to monetize?

I run r/WallStreetBets and "quality" is an extremely nebulous term.

I look for things like "novelty", "thought-provoking / well reasoned commentary", "original content", "authenticity", "self-awareness" or "primary research". But these are human assessed metrics.

Some more easily measurable metrics might include, "length of submissions", or "number of outbound links excluding blacklisted domains". Or even "number of tickers or quality-correlated keywords mentioned".

All these metrics have very clear downsides, and if generally well-known, become useless. Interestingly, a score too high can also result in something being unlikely to be authentic.

Another challenge is your relationship with users. Surprisingly, moderators are not innately adversarial to users, they can also promote content through other channels (discord, twitter) or sticky threads for a viewership boost.

So, even without a profit motive, what do you do?

hammock• 4 years ago

Hard to scale, but farming out content samples to a panel of your target market for them to rate on the attributes of "novelty", "thought-provoking / well reasoned commentary", "original content", "authenticity", "self-awareness" or "primary research" might be an approach

empathy_m• 4 years ago

Historically, the immortals on MUDs used to optimize for thing like "fun" and "clout" - how much do you enjoy interacting with the players and being seen as the person in charge of their game? How much do you enjoy telling people you volunteer your time to work on it?

If WSB members were doing bad stuff like, making mean jokes about the leadership of companies Ryan Cohen divests from in; or coordinating real-life harassment of family members of employees of the Depository Trust Company or something; you'd probably want to put a stop to that and focus them somewhere else. Because it wouldn't be fun.

If Fox News interviewed you and you sounded silly and all the community members got mad at you, you'd probably also quit the volunteer job.

But as long as you make choices that make the community a fun and engaging place to hang out and feel like you're part of a big secret treasure hunt, that seems awesome, and you would probably want to make choices that maximize that.

TBH most MUDs were not data-driven and kinda govern by feeling -- they'd sort of add steering they thought was interesting and keep it in place unless there's a backlash. The tedious administrative stuff (bans, moderation, player requests for item reimbursement) would always have a backlog and you'd recruit junior imms to help out and they would feel like they were part of the fun too.

JAA13372• 4 years ago

Just want to give you a shout out for using the acronym MUD. I havnt seen it used in like 25 years :). Thank you!

runnerup2• 4 years ago

Measuring community engagement makes sense when your primary mission / core goal is to build a strong community. When I was building a large online third-party Discord community for a university during COVID, I was very glad that Discord exposed a lot of metrics for admins/mods to track. It's obviously important to not "game" the metrics by falling into short-term gain traps like over-pinging users, or encouraging high-volume but low-quality participation, etc. But as long as you are laser-focused on "providing real value" to members, then long-term or multi-cyclical trends of community engagement is a useful benchmark.

But it doesn't make sense to focus on these metrics if your mission is to create a great product -- focusing on community engagement should be just one small aspect of your overall marketing strategy, which should be a relatively constrained part of your overall budget (both in labor and money). And even within marketing, there are many other important metrics besides community engagement.

Soylent had great community engagement but not enough focus on scientifically-guided product engineering. Tons of businesses that get this wrong.

Generally I interpret overcommitment to community engagement as a signal that the founders are overly narcissistic and usually apply a discount the expected future value/evolution of their product.

JohnFen8• 4 years ago

Measuring community engagement makes sense when your primary mission / core goal is to build a strong community.

But measuring community engagement doesn't tell you anything about the nature of the community. Only that it "engages" people. If your goal is to build a strong community, isn't it more important that the community be a good one?

JohnFen8• 4 years ago

But how do you assess progress when you remove all desire to monetize?

If you ignore the profit motive, presumably you are doing it for other reasons. You assess "progress" based on those motives. Or perhaps you don't assess "progress" at all, beyond "I like what I'm doing here and where this is going".

I've run numerous internet services over the years without the goal of them generating an income at all. I've run MUDs, websites, discussion groups, etc. I didn't objectively measure anything about any of them, because my goals were not ones that could be objectively measured. And honestly, even if there were some metric that could be used, I would have avoided using it because then it becomes about maximizing the metric rather than the purpose I started the activity in the first place. If I was happy with how they were going, that counted as "success" to me.

So, even without a profit motive, what do you do?

I honestly don't understand this question. Without a profit motive, you do it for other motives. If you have no other motives, then why are you doing it at all?

the_snooze• 4 years ago

because my goals were not ones that could be objectively measured

This is something that technically-minded people can take to heart more. There's a lot of value in things that can't be objectively measured. For instance, I'm in a bunch of group chats with various friends. I don't always interact with it (i.e., my "engagement" is fairly low), but I derive a lot of enjoyment from those. Why? Because I like those people. Can I put a number on how much I like those people? No, because I'm not a socially stunted robot.

scythe• 4 years ago

I didn't objectively measure anything about any of them, because my goals were not ones that could be objectively measured. And honestly, even if there were some metric that could be used, I would have avoided using it because then it becomes about maximizing the metric rather than the purpose I started the activity in the first place.

There is a middle ground here. Choose a basket of metrics, impose the natural partial ordering, and intuit your way through pushing up one or the other. If this sounds crazy, keep in mind it's roughly how the Fed (inflation + unemployment) runs the monetary system.

nonameiguess• 4 years ago

I don't think you're taking this the way it was intended. The person who posted is a moderator who does not have a profit motive. Reddit doesn't distribute profit to subreddit moderators (as far as I know?). But even without a profit motive, he is still left without any simple way to evaluate the quality of submissions and decide what should be promoted and what should be killed. The point being "engagement" is often used not because of profit motive, but even in the absence of profit motive, it is a good default metric that is easy to measure in trying to figure out what your community actually wants to see.

He's not asking what motives you should have if you don't have a profit motive. He's asking, even in the absence of a profit motive, what better measure of content quality is there than how much the community engages with it? You gave an answer. In communities you ran, you just asked yourself if you were happy with them. That's an acceptable answer. But assuming someone who runs a community wants the community to be happy as well, that is a lot harder to assess, as you no longer have direct access to their mental states the way you have to your own. Now you're left again with the need to estimate their happiness via some proxy measure. If not engagement, what should that be?

I'm not saying there is no answer and all communities should throw up their hands and either just use engagement or use no measure at all, but I do think it is legitimately a hard question.

AndrewKemendo• 4 years ago

So, even without a profit motive, what do you do?

I think there are two options here:

1. If the community that you have does a "foom" and grows like crazy BEFORE it has a set of values, then you're at the whims of the mob and where certain vocal and rallying members decide they want to take the community. Usually this ends in internecine conflict and the inevitable "break offs" into other smaller groups

2. If your community starts with a value vector then you structure, nudge and contain the community to a MDP (Markov Decision Process) with the value vector as the reward, for which the moderators are the critics and the community members are the actors in the actor-critic model.

It seems like the vast majority of organizations/movements etc... fall into category 1

For me the learning here is, if you are a leader in your community it's never too early to nail down what the core value and benefit of your community is and stay obsessed with just that.

tstrimple• 4 years ago

All these metrics have very clear downsides, and if generally well-known, become useless.

Good ol' Goodhart.

"When a measure becomes a target, it ceases to be a good measure."

wolfofcrypto• 4 years ago

Mark Cuban recently launched a community at https://biztoc.com as sort of a WSB for business people. I wonder if that will/could work without an umbrella topic.

RealityVoid• 4 years ago

It seems to be a news aggregator rather than a community.

jrochkind12• 4 years ago

But these are human assessed metrics.

OK, and? I could guess what the problem with this is, but let's spell it out... maybe that they are subjective so different assessors can disagree... and this is a problem why?

What if there are some cases where it's actually important to use human-assessed metrics as one component, or even as the entire thing, cases where no appropriate 'objective' metrics are available? (In scare quotes, because these quantitative metrics are seldom _quite_ as 'objective' or independent from human judgement as assumed. There is usually human judgement involved in how the metrics are defined and measured, where different people might define and measure a metric different ways resulting in different numbers...)

zinglersen• 4 years ago

There are definitely some cases where it's important to use human-assessed metrics, like cooking a meal.

But if you are the size of Facebook, Instagram, Tiktok, etc. it is too risky (and expensive) to let individuals define what success looks like - you need something measurable that you can spread to multiple projects or even across the complete company culture.

I fully agree with your point that metrics are typically not objective, like any data once it's used to communicate something.

So the question should be what metric(s), if not engagement in itself, should we measure for to create a healthy online community?

I wonder if it's less about metrics and more about principles; how does HN keep the level of quality so high? Is it because we share an interest? Because of the quality control? The UI/UX of the site? Something else?

groby_b• 4 years ago

So, even without a profit motive, what do you do?

You acknowledge that human interaction and metrics are not compatible, and apply actual judgment. You're a human being, not a paperclip optimizer.

Your goal isn't to stripmine anything of value out of a community, but to provide a vibrant place people like coming to. So, for feedback, you ask them what they like, what they miss, what they wish for.

Qualitative assessments instead of quantitative ones. Lean into the power of the humanities.

civilized2• 4 years ago

without a profit motive, what do you do?

We could pose the question even more generally. Apart from making money, what is the value of a human life or community?

jamesknelson• 4 years ago

This is an issue with more than just the tech industry. It’s an issue with modern society itself.

Take the GDP, the number that when it’s first derivative goes negative, we shout “recession” and use it as an excuse for all manner of harmful (and sometimes beneficial) behaviors. What is GDP? It’s not a measure of wealth; it’s a measure of production.

Take the boots paradox, for example. A society making $50 boots that need replacing after a few months, will have a higher GDP than a society making $100 boots that last for years. And the shitty boots society, despite spending more on boots - to paraphrase Terry Pratchett - will still have soggy feat.

It’s a hard problem, and I can’t see any obvious way to solve it. Both in social networks, and in economics. For whatever reason, consumption seems to be much easier to measure than wealth.

mikkergp• 4 years ago

I think people enjoy this argument because it's counter intuitive, but I also think it's counter intuitive because it's wrong. If you build a playground and everyone avoids the teeter-totter, there's probably something wrong with the teeter totter. Oddly the monsters inc example seems counter to his point since it seems like they were measuring something to specific and the proper solution would be to something more general (like engagement) instead. It seems to me like trying to start with something other than engagement would be premature optimization.

Of course engagement is insufficient and of course if you want to know something else, measure that other thing, and of course you should interrogate your metrics, and maybe it's just this thing that bugs me about modern writing where you have to take this ridiculous extreme stance to get eyeballs but yeah.

JAA13372• 4 years ago

Couldn't agree more.

I believe this falls into the 'Results Oriented' mindset of delivering value. This means figuring out your desired outcome ahead of time, then measure it over time. In addition, overtime, you may determine that your desired outcome has changed and you need to measure differently. Yes, iterating is not just for software.

IMHO pure quantitative metrics are of little value for customer experience. I believe metrics should be a little squishy and subjective. The value in the subjectivity is that it spurns conversation and actual thought instead of simple counting.

felipelalli• 4 years ago

But... How? It's missing how to do it (measure value) in the article.

mhall11910• 4 years ago

I've started writing those answers out as separate blog posts:

Measuring Connections: https://www.savannahhq.com/2022/09/06/measuring-your-communi...

Measuring Support: https://www.savannahhq.com/2022/09/08/measuring-your-communi...

robust-cactus• 4 years ago

Lately - it seems we're all caught between 2 worlds:

1. If you over focus on impact the quality of your product gets torn to shreds. You've built great direction but do people really use/like your product?

2. You build for engagement, people use the thing, but what impact did it drive?

This article is great, but it's unbalanced potentially. I think directionally, some average in the middle is likely the way - so why not both? And while we're add it, add in other indicators. Success metrics that balance all of that, plus just a vibe metric for good measure.

photochemsyn• 4 years ago

'Stop making decisions based on engagment measurement' might be a better approach, if what you care about is maintaining the quality of your online social group. Some social media sites have tons of high-quality original content submitted by people with expertise in their domains, and some are 90% fluff. If you want to attract people who'll make high-quality contributions, getting rid of the fluff is the most important goal.

swyx• 4 years ago

when community engagement is measured badly, its because community is being treated as a product like any app where MAU and time on site is being prioritized, yet companies dont dedicate even 1/100 of the resources to community “product” that they do their software product (perhaps because it is not viewed as a real job, as perceived in other comments here)

that said, you do need a way to justify to others why you are a better community manager than the next, and whether you are doinng a good job. that part went unanswered here and the article would be better if it had a constructive replacement instead of just arguing to be unaccountable to the rest of the org.

ChrisMarshallNY• 4 years ago

That's a great post, and I plan to share it with others.

Sadly, it's pretty difficult to measure for some of the "values" that my communities run on, but "engagement" isn't even on the map.

I suspect that "engagement" does, indeed, provide real monetary value for advertising-based sites, as clicks == money. Since that isn't a factor in my communities, we don't worry about that. In fact, we try to minimize "engagement."

tamsaraas• 4 years ago

Originally they were there to connect people and let them share in each other’s joy.

No, they looking for a ways how to easily and faster to fuck.

They couldn’t measure joy, but they could measure engagement, and so they did. That measurement became their goal, and they focused on maximizing it.

Because this is social networks profit. Ads, and more people attached to some topic where can be shown an ads to these people. That's why and only that's why it is works like that. Nobody cares about principles. Only money. Power in money, a money is a power.

What they didn’t understand was that negative, divisive, hate-inspiring posts get more engagement than positive, supportive, kind posts do.

For what do your country has intelligence and different defense structures? Because it is very easy manipulate by people and do wars. Extremely easy to organize that and involve a lot of people to mass destruction events. Extremely easy. And i do not know (the same as you too) who is behind some kind of negative behind.

. The algorithm didn’t care whether the posts brought joy or anger, it only cared about whether they brought engagement.

Because this is about ads and money. More people will watch ads more money social network will get. Easy.

So in order to maximize engagement, the algorithm actively encourages and elevates posts that cause unhappiness among the platform’s users.

You said that? Is a bots with fake votes, SEO thing is not about manipulating data, opinions? Its easy to make fake accounts and promote hate as a wide interested topic, while it's not. Only social network know that.

They join looking for support, or connections, to give back, or find meaningful relationships.

No, they looking for themselves reflected in the community.

Measuring engagement doesn’t tell you if your community is happy, or healthy, or even accomplishing any of the things you were hoping it would accomplish.

Depends on a case. Some measuring engagement can prevent negative business processes.

Knowing how much engagement you have doesn’t tell you how much value you’re getting

For that you have another metrics. Depends on a case. Registration, profit, grow, etc. Final countable results.

Let’s bring this back to community building. We still want our members to be happy, right?

Who the fuck am i to let people happy? Happiness is subjective thing. I have no clue, no ability to make someone happy. The same as you, the same as anybody else. This is very agile thing depends on many factors and the most important wish to be happy and objective situation.

So what should we value?

Does your community member even read or know about "values" at all? Is it some kind of cult, or what?

----

I have a well known secret. To get maximum engagement need to say that the white color is black color. Water - is not a liquide, and planet earth is flat.

And then your "engagement" will go above the sky. Just always need to split people, and say that one thing is not the thing what you used to think, it's another thing. totally vise versa.

Oh... Darling, exactly this you have done by saying: "stop measuring engagement". And by this title invited to engagement a lot of people who forced to read all of your nonsense to one more time confirm that white is white, black is black.

pdntspa2• 4 years ago

Hot take: stop building communities around everything.

People are highly tribalist and banding together like that produces a lot of ugly outcomes. Communities form and likely won't go away, particularly the more people internalize them emotionally and integrate them into their identity.

It is exhausting to see a community for literally everything, even the smallest products.

Perhaps we should let products simply exist.

BrianOnHN• 4 years ago

The other day I was think, it's actually the platforms where most of these communities that benefit directly from their existence.

And we know those platforms are tools for manipulation.

How much of the desire for these communities might be manufactured in the first place?

pdntspa2• 4 years ago

That's the problem... so much of both the communities and desire itself are manufactured. When they grow organically... cool, for the most part. But the minute money becomes a focus things go to hell.

neilk• 4 years ago

You're raising an interesting point, but you're coming close to saying that communities are bad, and in my experience they don't have to be. Even ones based around a product.

Near where I used to live in the Valley, there was a real-world club for enthusiasts of a particular weird antique car. Not toxic, just eccentric. They just liked getting together and drinking in a nearby pub and showing off how they'd restored this or that aspect of the car.

I think our goals should be to replicate that kind of success.

I think the other posters in this thread have it right, that the reason why we usually can't is because our metrics are all about selling audiences to advertisers.

yieldcrv• 4 years ago

It’s because investors aren’t discerning either

They chose to pay higher prices for shares, divorced from actual revenue

The companies, in-turn, found non revenue metrics to show quarter over quarter

and thousands of startups copied the model

Just don’t rely on the A/B test, be aware of it, but don’t rely on it. Put the human control back into the process. Users aren't staying on your platform 5 seconds longer after you hid the escape hatch because they love it, they are frustrated and lost! The A/B test doesn't tell you why the engagement is longer, only that it is. It takes an empathic human to say “okay, B got us that result but not for the reason we want”

dfabulich• 4 years ago

This article completely undermines itself by arguing in the conclusion ("Measure what you value") that we can and should measure "meaningful relationships" instead.

They’re not as easy to measure as engagement, sure, but they can be measured.

I'm afraid that a citation is sorely needed here. There are no real measurements for "meaningful relationships."

In fact, the only proxy metric we have for "meaningful relationships" is engagement!

And, yes, engagement is not a very good proxy metric for meaningful relationships, but since there's no alternative, it's all we've got, so this article is pointless.

scanr• 4 years ago

I wonder if measuring engagement and optimising for engagement should be separated. I’m glad that there is lots of engagement on HN but I’m also glad that HN chooses not to implement a number of engagement optimisation features as well as downranking topics that result in high engagement flame wars.

mikejulietbravo• 4 years ago

engagement takes many forms. If you value people connecting with each other, measure how many messages they're sending or connections they're making.

Saying stop measuring community engagement is a good clickbait headline, but it's absurd advice for community managers that even your own article doesn't agree with.

RicoElectrico• 4 years ago

So please tell me, what should I optimize for when moderating a local OpenStreetMap community. I personally feel it's crucial people internalize that's a project with a bottom-up organization. A "civil society", if you will. Alas, despite 200-250 daily active mappers in Poland [1] only a fraction ever posts on the forum, even if just to ask something. Meeting IRL is also futile except 2-3 largest cities. We have a forum, Facebook group and Discord.

Whatever I'd come up with, I'd like it to be a self-healing organism that would continue to thrive even if some important members become inactive. And to feel it's not a Sisyphean task.

[1] https://osmstats.neis-one.org/?item=countries&country=Poland

scoofy• 4 years ago

I'm confused. I see OSM as a huge success. Firms like Mapbox and Strava have an incentive to update that wiki with you. The maps are amazing. Any local sub-community will be challenging to organize, but maybe the metric should be making it fun for you, and if you're having a good time others will join.

I am also starting a wiki, a golf course wiki: https://golfcourse.wiki

It's still very early for me, but I see myself in this post. It's going to be extremely slow going, as the main consumers of the content created have no desire to contribute, but since golf courses are mostly static, as long as i plug away, adding a little bit all the time, the site will be a success. It costs almost nothing to operate. Generating traffic is challenging, and i've been trying to find more people willing to contribute, but again, I see my product as best placed if i do exactly the opposite of what most of the golf course info sites are doing. Don't base my success on engagement, no aggregation, no overly-prodding to contribute. Just one day at a time, with the assumption that the task is endless anyway, having people share their passion on their own time.

If i can make that profitable at all, and it shouldn't be too difficult as the site grows, i'll see it as a total win.

Sophistifunk• 4 years ago

Twitter isn't a community, it's a platform. There's nothing at all wrong with community engangement. Platform engagement and the advertising model that requires it are the problems.

swayvil2• 4 years ago

Maximizing community engagement, trolling and advertising are related. They're different heads of the same multiheaded demon.

All 3 are concerned with attention. Are hungry for it.

If social media could be optimized to deliver engaging conversation to everybody then the demon would be slain. Because the hunger would be satisfied.

Does that sound right?

actionfromafar• 4 years ago

Engaging conversation is not as intense as raging. The hunger can never be satisfied.

sfink2• 4 years ago

Seems reasonable to me, if you reduce it to putting the demon to sleep.

The demon is always there. A forum filled with engaging, productive conversation can and will slide back to toxic crap if there isn't something keeping it in check.

It may be "optimized" for engaging conversation, but what does that mean in practice? If it's fully or mostly automated, then it's relying on metrics, and humans will always always always find a way to maximize the score :: effort ratio of any metric. Low effort is incompatible with engaging productive conversation.

If it's done fully or mostly manually, then that means actively keeping that demon snoozing.

civilized2• 4 years ago

Here's the little discussion I have in my head when this issue comes up.

"Engagement isn't a good metric. Optimize for having a good community."

"But engagement is easy to measure AND it means people are using the product. If people are choosing to use the product, doesn't that mean the product is giving them value?"

"Not necessarily. Drugs like tobacco and heroin have fantastic 'engagement' but we consider them bad for us."

"But how do you tell the difference between something that engages because it's good and something that engages because it's some kind of drug?"

"I guess one important signal would be when people know something is bad for them but they're doing it anyway?"

"Is that the case for social media? Do people feel addicted - feel that it's bad for them but they can't stop?"

I think we've heard a lot of anecdata on this last question, but I can't recall seeing any big survey on it.

EDIT: according to this, people will pay to have their social media restricted: https://www.washingtonpost.com/outlook/2021/07/19/social-med...

bawolff• 4 years ago

Is engagement really value neutral?

In the extreme, if you spend all your time "engaging", something toxic is happening

swayvil2• 4 years ago

Advertising is a kind of trolling. I think that's an important connection to ponder. It could lead to something.

nitwit0052• 4 years ago

If you can find a way to easily quantify the quality of human social interactions, you're probably on the short list for a Nobel prize. It'd have applications in every field related to humans.

joelthelion• 4 years ago

I don't think that's completely unachievable with the current state of NLP tools.

You could (roughly) measure the diversity of discussion topics, the frequency of positive and negative emotions, etc. It'd be work to do it well, but I think it could work decently.

nitwit0052• 4 years ago

You're suggesting that you can find things you assume correlate with what people want out of a community, but several people here commenting they can't define what they want. There's low odds of it somehow matching what they want, but struggle to express.

Sohcahtoa82• 4 years ago

As an end user, fighting back against using anger to drive engagement is easy.

Stop engaging with content that angers you.

Don't follow politicians and news outlets you despise. If someone you follow retweets the latest garbage being spewed by Fox News, don't interact with it. No retweets, no replies, no likes, no matter how well they refute the bullshit or how well they "own" the original messenger.

The algorithms feed you content they think you'll engage with. Train the algorithm to feed you better content.

Stop measuring community engagement

Related Stories