Logging Everyone Out

https://lists.wikimedia.org/pipermail/wikitech-l/2020-October/093922.html

by edward • 6 years ago

313 143 6 years ago

lists.wikimedia.org

aerovistae• 6 years ago

Geez, that's a new one.

punnerud4• 6 years ago

The same happened in Norway on a governmental page a couple of years ago. It then was a memcache problem that made other people see one persons tax report.

It also happened a couple of days ago again. The cause is not known: https://translate.googleusercontent.com/translate_c?depth=1&...

rmetzler• 6 years ago

I have seen the same issue with caching in varnish when cashing rules were naive or some proxy stripped relevant headers like Cookie.

leoh• 6 years ago

Fun fact: income for every citizen is publicly available in Norway.

trynewideas4• 6 years ago

They've done it a few times this year alone: https://lists.archive.carbon60.com/engine?list=wiki;do=searc...

The other times just weren't noticed, posted, or upvoted on HN.

s_dev• 6 years ago

This seems all very reasonable. Look forward to the post mortem.

Given their size on both the web in terms of employees this unusual for Wikimedia. They typically fly under the radar. How many times has Wikipedia ever been down?

I recall AWS, Google, Microsoft having more outages -- mind they probably are considerably bigger but still they're doing something right.

jshen5• 6 years ago

it's not hard to have high uptime for content that is largely static. Has less to do with size than static versus dynamic, and accurate (think bank transactions) vs fuzzy (search results)

edit: fixed a typo

outworlder• 6 years ago

it's not hard to have high uptime for content that is largely static

Largely static? There are edits happening all the time.

hackonr• 6 years ago

SSR would be more accurate definition than static.

aidos• 6 years ago

I’ve not looked for a long time but the last time I edited it, it was close to realtime*

*in the web sense - don’t want to offend any of the real real-time people :-)

kashif• 6 years ago

it isn't static - just wordy. You are confusing lots of text for staticness.

dheera3• 6 years ago

There are enough copies of Wikipedia that if it has ever been down you could just get the same content elsewhere, so people don't usually make the same fuss about it that they would about AWS/Google/Microsoft.

AWS/Google being down for even a minute or two is a big deal though.

middleclick2• 6 years ago

How many people read Wikipedia other than from the main website? What's the source on that?

fireattack• 6 years ago

What do you mean by "Given their size on both the web in terms of employees this unusual for Wikimedia"?

(I knew this likely just a typo, but I genuinely didn't figure out what you meant.)

donatj• 6 years ago

I believe the Steam Web Store had a similar issue of people being logged in as the wrong user a couple years back.

gizmo385• 6 years ago

That was a caching issue if I remember correctly, as opposed to users be "logged in" as a different user.

zapt022• 6 years ago

Isn't that the same thing? Say that you forget to specify that "my_login_cookie" should vary the response from the backend. User A logs in. Now user B logs in, but because the page was cached from user As visit, user B sees the content for user A. B is not "logged in" as A, but can still see their content.

cheeze• 6 years ago

This is a very common 'advanced failure scenario.' I've seen it on a handful of sites, session objects and caching are difficult and sometimes overlooked during migrations.

punnerud4• 6 years ago

Love MediaWiki as a platform. I host several of them. Hope the interactive editor in PHP soon get out of beta.

gerdesj• 6 years ago

Do you mean the Visual Editor? That's been available for several years now. Getting the back end for it running can be a bit fraught.

I wrote this: https://www.mediawiki.org/wiki/Intranet and keep it up to date every now and then. It's probably time for me to look at 1.36. Anyway, the current Parsoid based thing works fine.

bawolff13• 6 years ago

I think the grandparent means the new version in 1.35 where the backend should just "work" (i think it is out of beta now)

punnerud4• 6 years ago

Backend/Parsoid also using PHP I was thinking of.

znpy• 6 years ago

The Visual editor is going to get bundled in 1.35.

The beta go refers to in a version rewritten in PHP to drop the dependency on another service, parsoid, written in JavaScript/node.

kevmo• 6 years ago

I wish every service did this on a weekly basis.

johnnyfaehell• 6 years ago

You can delete your cookies on a weekly basis and it'll be like they did.

dylan604• 6 years ago

Or just block cookies and never have this issue

Xylakant• 6 years ago

Judging from the error description, deleting cookies would likely not help in this case. This sounds as if sessions are mixed up on the server side - deleting your cookie will only remove your session token, not remove the session server-side. You’d need to actually send a logout request to the server.

t0astbread• 6 years ago

I wish every service that has something like user sessions provided the ability to revoke other sessions of your account. Then, if the site isn't too hard to automate, you could write a program for that.

prionassembly• 6 years ago

Do they mean employees? My Wikipedia login lasts for 30 days, I think. I basically only login to edit articles that require so. Otherwise I just edit as anonymous.

dredmorbius• 6 years ago

No. Everyone.

M2Ys4U2• 6 years ago

No, users. I just had to log in, despite recently being active. I guess this is why.

mmaunder• 6 years ago

I'm reminded that MediaWiki runs on PHP, which isn't much loved here on HN, but does power some of the busiest sites in the world, and PHP 7 and 8 have done a great job of moving the language and performance forward.

ibejoeb• 6 years ago

It's been quite a while since I've looked at mediawiki. Has any of the codebase been transition to modern PHP?

edoceo2• 6 years ago

Not really, PHP is very backwards compatibility and I don't see lots of "modern" PHP (eg Traits, etc) in the code. I last looked at their release from a few months ago.

I do lots of PHP migration work (4 to 5, 5 to 7) and for much of it it just works.

Not much code change is necessary to the the upgraded features of the engine.

nurettin• 6 years ago

It needs to scale reads using in-memory caches. After that, all it needs to do is to be able to handle 10-20 posts a second and you're golden.

judge2020• 6 years ago

Assuming concurrency is done correctly (which FPM handles, although not transparently/as transparently as some would like) any language can be super-performant - PHP's leg up is long-term compatibility (a php 5->7 upgrade is better/easier than say a python 2->3 upgrade, at least in my experience) and syntax being easy to pick up and become proficient in.

8ytecoder• 6 years ago

I have seen this behaviour on Facebook, ages ago. Reported it and yet nothing came out of it.

jayar95• 6 years ago

That doesnt surprise me. They will let highly problematic security issues persist for weeks

frakkingcylons• 6 years ago

Have any examples?

CGamesPlay• 6 years ago

The "view profile as another user" feature was infamous for being... really thorough. At one point you would receive chat messages directed to the person you were viewing the profile as (I believe this was before the integration of chat and messages).

andrewla• 6 years ago

I remember early in the history of E*Trade, I went to my account, and it showed someone else's name and account information. Didn't even bother reporting it (I was just a kid); just logged off and on again and withdrew everything from my account and never looked back.

dionian• 6 years ago

What is the (he/him) after the guy's name?

dEnigma• 6 years ago

His preferred pronouns, I would assume.

Raphmedia5• 6 years ago

Pronouns.

That way, if someone is named "Billy" or "Alex", you won't be spending years mistakenly assuming they were a man while they could have been a woman.

This also has the beneficial side effect of covering trans individuals who use a birth name while having another gender identity.

Even more useful for usernames such as yours, "dionian". I do not know if it is your first name, last name or made-up username. So I have no clue what your gender is and I'd default to "they". If I knew, I could call you by the right pronouns.

It's just a slow shift away from assuming everyone on the internet is a man.

fennecfoxen5• 6 years ago

It also serves as a flag that allows people to identify you as having bought into the general set of practices which strive to change our culture to support trans and nonbinary gender ideology.

hinkley2• 6 years ago

I hesitate to call it virtue signalling because there are other forms of social signalling that aren't about virtue. Safe spaces require effort from the members who already feel safe.

Putting (he/him) in does three things. It invites you to provide your own. It also stops someone from starting a side-argument about you assuming 'he/him' in your response instead of you/your or they/their. Which means that we can talk about what we want to talk about instead of gender politics, if you don't feel like it. Or we can if you need to.

kepler1• 6 years ago

It's appropriate that the commenter below uses the phrase "bought in", because that conveys the general sense of mob/unthinking latching onto this movement. One which I'm quite disappointed to see companies just caving to, through little insertions of ridiculous practices like these.

Having everyone declare their gender is a little bit ridiculous, just to serve the desires of a ~1% group who are trying to gain more recognition. I am frankly surprised how people are willing to distort their behavior when they refuse to do so for other groups who are far more downtrampled in their rights in greater percentages. I suppose somehow transsexual people just became popular for some reason.

Frankly, it's a symbol and problem of the modern liberal/democratic mind (at least at the party-level) that these problems rise to the level of national and corporate attention -- and apparently we solved all our other material needs and have time to spend on this in comparison.

simcop2387• 6 years ago

This person has decided to provide the pronouns he would like to be used when referred to

Amorymeltzer• 6 years ago

Preferred gender pronoun:

Preferred gender pronouns or personal gender pronouns (often abbreviated as PGP) refer to the set of third-person pronouns that an individual prefers that others use in order to identify that person's gender (or lack thereof). In English, when declaring one's preferred pronouns, a person will often state the subject and object pronouns along with the possessive adjectives—for example, "she, her, hers", "he, him, his", or "they, them, theirs"—although sometimes, only the subject and object pronouns are stated ("he, him", "she, her", "they, them").

https://en.wikipedia.org/wiki/Preferred_gender_pronoun

narenkeshav• 6 years ago

I've been seeing this trend quite often these days.

function_seven• 6 years ago

When people list their preferred pronouns, why list them all?

Here we have "he/him". Sometimes I see "he/him/his" or "she/her/hers", "they/them/theirs", etc.

I doubt people are mixing and matching among the different types. In other words, I've never seen someone prefer "he/them/hers".

Seems like we can standardize this to just one type (e.g. the subject pronoun). A simple "he", "she", or "they" will do the trick.

Yes, I bikeshedded it. Sue me.

jamiequint• 6 years ago

I've seen a lot of he/they or she/they lately

geogriffin• 6 years ago

I think it's just more clear that they are listing pronouns, especially if someone isn't familiar with the practice or in verbal speech. Also note that some people use two sets of pronouns, with equal weight, and may list them such as "she/they".

DanBC• 6 years ago

Chris is a name used for men and women. He's letting you know he identifies as male.

DoreenMichele4• 6 years ago

I have begun putting she/her in some of my profiles (twitter, reddit) because I still get mistaken for a man at times. To be fair, my middle name -- Michele -- can be a male name in some places. It's the Italian version of Michael, IIRC.

It is my personal policy to not correct people who misgender me in most cases, especially if that is the only thing I would be saying (I will sometimes clarify if it is part of a larger comment, but I try to be gentle about it). I would rather put that info somewhere and let them have the chance to learn of it without me having to correct someone.

Different people have different reasons for noting their pronouns. Some do it because it is trendy. Some do it because they have genuinely been misidentified in online spaces. Some do it to show themselves as allies to certain groups.

Without him saying why he did it, no one here can genuinely tell you why he chose to do so.

throwaway_pdp094• 6 years ago

With my name I get presumed to be a woman - who cares? I don't.

usmannk• 6 years ago

Bugtracker: https://phabricator.wikimedia.org/T264370

foolfoolz2• 6 years ago

i worked at a company that spent 6 months and 2 devs to solve a huge infrastructure change without logging anyone out of their session

dahart• 6 years ago

Isn't that a tad different than discovering an active security hole?

2ion• 6 years ago

Logging people out seems like a hella low price to pay to pay for a potential security issue. Of course, hats off if you hold your engineering organization to such a high standard for normal changes.

kazinator2• 6 years ago

Not logging anyone out while upgrading the kernel of a time-sharing operating system: nice feat.

Not logging anyone out while updating a suite of REST-ful applications: ho hum.

foolfoolz2• 6 years ago

this was stupid. we were not a big enough org to spend 2 people like this. i never understood why forcing people to log in again was so controversial

aeyes• 6 years ago

Depends on the type of application, 99.9% of Wikimedia users are using the site without being logged in.

If the site doesn't work without being logged in you could frustrate users and they might just use a different product instead of searching for their login after being logged in for a year or longer.

kazinator2• 6 years ago

That doesn't seem difficult; just don't change the format of the session tokens.

Eg a web application: you can substantially rewrite a web application, without invalidating logged in sessions.

The point here is that the logged in sessions were suspected of being unauthorized. The unauthorized sessions had to be turfed, and the clearest way of being sure that all unauthorized sessions are turfed is to delete all the sessions.

Of course, some those with unauthorized access will also try to log in to resume that unauthorized access, but presumably there is some trap laid for that.

Maybe for the accounts suspected of having been breached, there will be a mandatory password recovery procedure or whatever. Or they will monitor for suspicious logins from different IP addresses.

pvg• 6 years ago

You can have state sitting in (serverside) session storage that is incompatible with the new version but you don't want users to lose. So now you have to migrate it, which can end up being actual work depending on the change.

tomc1985• 6 years ago

Assuming session ID's are tied to individual users and looked up via a "SELECT * from sessions where id='?'" type query, how does this even happen?

torben-friis• 6 years ago

Badly designed caches for example. Though I have no idea what happened in this particular case.

FilterSweep• 6 years ago

This (bad cache) used to happen with misconfigured shopify sites and had the same impact as described

duderific• 6 years ago

Not scoping a variable properly (causing a variable to leak out to global scope) could also cause this.

bawolff13• 6 years ago

Not usually in php, where each request has no shared state with any other (outside of apc, memcached)

TheRealPomax• 6 years ago

Remember https://www.theregister.com/2015/12/30/steam_security_blip_e... ?

trynewideas4• 6 years ago

None acknowledged publicly, but it's an open-source project with pretty open administration, so you can follow along: https://phabricator.wikimedia.org/T264370

Kiro• 6 years ago

This reminds of a thing that happened on MSN back in the day when I suddenly started to receive random users' chat messages in one of my chats. It only lasted a few minutes and it was all different kind of languages. Never heard anything about it but in retrospect it felt very serious. Imagine the same thing happening on Facebook Messenger or Whatsapp.

andrewmcwatters• 6 years ago

I’ve always wondered if this class of cache issue resulted primarily from collisions of some sort. I’ve only seen it a handful of times over many years. Others here have mentioned it with services using Varnish, for instance.

zapt022• 6 years ago

Varnish cache works by creating a hash of the request consisting of method, url and for a site where you log in, that would also include the cookie you use for identification. Your proposed collision idea is very improbable but possible.

There is actually a good article in the Varnish docs around hash collisions: https://varnish-cache.org/docs/4.1/phk/varnish_does_not_hash...

PS. Even though the article says "Varnish does not hash", I suggest everyone to keep reading until the end, when it turns out that they, in fact, do.

harryf• 6 years ago

I know the full analysis isn't online but I have a problem with this part...

This was done out of an abundance of caution, after we received one (1) user report of being logged in as someone else.

This _seems_ like a knee-jerk reaction to one data point.

There could be other causes for a user to report that, like a change to the cache key used for serving a users profile giving the _appearance_ that you're logged in as someone else, even though you're not really.

Forcing everyone to re-login could potentially make the system worse, in that you're now overloading parts of the system that has to handle those logins, plus causing all kinds of cache expiry...

I guess there's more to the story and someone who knows the system deeply knew this was the right choice but just reading the reports it seems knee-jerkish.

jeremyjh• 6 years ago

It seems adequately explained in the next sentence.

Said report coincided with the deployment of a new MediaWiki release which caused other problems around User session objects;

flemhans• 6 years ago

I remember back in the days, I disconnected my dial-up modem and re-connected with another public IP. Upon refreshing Hotmail, I was presented with another user's mailbox.

I was never able to reproduce the glitch.

polygot• 6 years ago

I had a similar issue happen to myself (not on wikimedia or anything related to wikipedia.) I clicked on "login" by accident without filling in my credentials and I was logged in as either an admin user or a user called "Adam".

m-p-3• 6 years ago

So that's why my account was logged out from the mobile app. No big deal, as long as the security of the system remains intact.

jacobwilliamroy• 6 years ago

My buddy told me that the U.S. marines in afghanistan once misplaced one of their fancy encrypted radios (or it might have been one of the keys to their fancy encrypted radio) so all radio communications were suspended until the next scheduled rotation.

hexbinencoded• 6 years ago

Battery ran out? Left one behind? Throw them all away and visit the quartermasters' Idiocracy/MIC version of Costco to pull another 100 pack off the shelf at a cost to the taxpayers of mere millions. Oh and F-35B's and Blackhawk turbines? just chuck those like Kleenex. My mom worked in civil service for the Navy and $20k parts in the '70's went "missing" and "lost" all of the time. Then there's a USAF Major I know who became unpopular for suggesting proper maintenance rather than ruining/replacing turbines might be a better approach (likely some sort of maintenance/HQ personnel-vendor kickbacks, MIC FTW).

One of the few stories VICE didn't botch entirely: https://youtu.be/_CvWJVtEkUE

tomcatfish• 6 years ago

Wow, this is the first time I've seen a company pick "better safe than sorry" when it had global effects (though this isn't a large hit, it's everywhere).

I appreciate it a lot, thanks WikiMedia

Logging Everyone Out

Related Stories