dhodell

Ironically, several folks who worked on Plan 9 later worked (or continue to work) at Google, although none of them worked on Fuchsia.

<ramble> To me the major overlap between them is their designs are clearly informed by the contemporaneous shape of network architectures. Fuchsia is a take on what an OS design would be as a set of named microservices that can be routed. Plan 9 noticed network topologies of compute labs and clusters weren't too different, and both graphs could be represented in filesystems. The major visible difference to me is that the visibility of routing is much more apparent in Plan 9 than it is in Fuchsia. It's still a little difficult to understand how and where capabilities propagate through the system.

Implementation-wise, FIDL is a much different take than 9P2K. Though much simpler, 9P2K forces every API to exist via a filesystem interface (many of the higher level protocols also involve quite a lot of string passing) and struggles with throughput of streaming operations. Individual FIDL APIs might have similar problems, but the message encoding itself is relatively more efficient. </ramble>

These things work in tandem: the base system is pinned, but can also be easily updated with an OTA. Packages existing outside that set are resolved on-demand, and are thus updated when components in a package are run after a new version is published to the package repository.

Sorry, I am doing a poor job communicating. I agree with your assessment of how the bazaar model works.

When I said "in this regard", the intent was to signal that "bazaar" explicitly doesn't mean an anarchic situation where all change is accepted. It's maybe more anarchic than the cathedral model, but it's controlled anarchy.

The reality is that most projects fall somewhere between the two models. The people most heavily invested in the projects tend to drive change, and this is sometimes at the behest of the community. That doesn't automatically make the project a cathedral.

Similarly, while Google controls the direction and funds the development of Fuchsia, our development process (which we publish on fuchsia.dev in the Governance section) is very much bazaar-like. This is difficult to see because we haven't been open about our processes, and we haven't engaged externally with developers. To that extent, we've been very much "cathedral". This expanded model is us inviting folks to join us in the bazaar.

Nobody will ever remember the rejected error handling or genetics proposals; but people will definitely take notice if the eventually accepted one come attached with Google names.

I guess, but my point is that many of the rejected ones are attached with Google names. Also it's unclear to me that the reason for not remembering is due to who proposes and implements the things -- it seems more likely that this is due to a huge volume of proposals over a long span of time. WRT generics, it's also very clear that community feedback is being incorporated.

So this is why I think the issue is really rejection. Nobody's really upset about how error reporting or generics are being handled because it's clear that community feedback is being honored, and it's clear that the core team is holding themselves to the same standard the community is being held to. Those two properties aren't as clear in the packaging situation.

That's why I think Fuchsia having a documented community process and also having been working within that process for some time is important. We're already holding ourselves to the standard we're asking of the community.

I believe it takes extra concerted effort.

This is absolutely true, and it's why I'm here trying to make sure that folks know there are people involved in the project who care greatly about its open nature.

Many people have pointed out that the existence of open source doesn't necessarily make the project accessible for contribution, and that's I think the more difficult problem to solve in the short-to-medium-term.

I agree with you. My point here was exactly that is not how a bazaar model functions. Hence the use of "in this regard."

they will also have to release their changes

Strictly speaking, they only have to release their changes to people they distribute their product to. It's convenient to keep it entirely open, but not a requirement.

I agree that folks should contribute code to places they feel comfortable in any case. As I mentioned in GP, if you separate out pay, there are other reasons to work on OSS, and these motivations can coexist.

I'm playing Devil's Advocate here because I sympathize with your point, but isn't there a way you can squint and look at this weird that it's true of pretty much everything else? IBM makes tons of money off of Linux, for example. iXsystems profits off of FreeBSD. Facebook profits off of their wildly popular C++ libraries.

I think we really need to divorce "profit" from "open source". OSS is great because we can contribute our own ideas to it, because we can learn from it, because we can form communities around it. It's also useful because it's a vector for entrepreneurship. Edit: why should these be mutually exclusive properties?

It's fair to point out that it can be a vector for exploiting unpaid labor. But that's an accessibility issue that exists already. For example, it's easy to tell folks interested in the field to "do open source work" if they want to get a job in the field. If you're a low-income single parent, this doesn't make CS more accessible, it's a sick joke. Even without profit, open source isn't this bastion of opportunity that we sometimes like to think it is.

You make an excellent point that it's hard to learn large new systems. But it's not impossible. Indeed, it's possible to make super meaningful contributions to the system in "limited time", starting with net zero knowledge: https://blog.quarkslab.com/playing-around-with-the-fuchsia-o.... To that end, I have to disagree that we're the kind of project that outsiders can't meaningfully contribute to.

I've addressed the point of community consensus elsewhere. All I can say here is that I personally owe my career to open source, and I'm therefore committed to helping others do the same. We opened the project to communicate with people. My personal goal is to do this and help as many people as possible succeed professionally through this vector.

But this is not always a fair label to apply. If I release a device running Android that prevents you from using your own Android, that's a problem with me and the device I released, not with Android. This seems to be conflating hardware and software -- which is understandable in the context of operating systems -- but we should be clear about the cause of the problem, which has nothing to do with the openness of the code.

Neat, thanks for that info. I really appreciated Pieter and his work; it's nice to hear his community and project are still well.

Without intending to either minimize or represent OP's opinion, my thought is the cases people get frustrated about are really unfortunate. In these cases, community opinions have been solicited -- sometimes committees are even formed -- and folks put in significant effort.

The oft-cited case with Go (and I'm paraphrasing heavily, so this account is likely unfair to everyone involved) was when Peter Bourgon formed a committee to design a Go packaging system, there were a bunch of meetings held that included core team members, and Russ apparently surprisingly came out with the Go modules proposal. The core team decided to adopt Russ' work. The team's perspective is it solved their problems simply and cleanly, and it came with an implementation. The community perspective was that the core team was now a sort of cabal.

I think the issue here isn't that the community's project was rejected, it's that there is a perception that folks were let to waste their time. It seems that sometimes people want this idealized model where an open project has a community that is on equal footing with the project owners, but I rarely see this to be the case in practice. Ultimately some individual or group holds a "voting share" that outweighs the community.

We have a documented process for system changes in Fuchsia that we have already been following internally. I've seen various proposals rejected; I've had proposals of my own rejected. It's always disappointing to have work rejected, because no matter what, it takes effort to come up, submit, and socialize work proposals. It's easy to feel slighted, especially when you're contributing for free while the folks making the decisions... well, it's their job.

I think I can say that it is nobody's goal on Fuchsia to waste folks' time. But I think it would be naive to think this kind situation couldn't or won't occur in the future. I just hope our transparency about our process and our availability to communicate with the community through lists will help mitigate negative feelings when a proposal representing a non-trivial effort is rejected.

I understand the concern. I spent the vast majority of my career not at Google while also involved in open source communities, and I can empathize with this. I think one place we have a leg up here is that we do have documented processes for performing this type of work. It's certainly possible that the outcome of a proposal is rejection, but the hope would be the process does that quickly. We're all human and we don't want to waste anyone's time.

One of the cases that people cite for Go rejecting the community outright is the modules work. My perspective is that could've been handled better, but I didn't have a stake in that. But then I look at the error handling proposals, all rejected, including the ones originating from Googlers. And the iteration on generics, which has been reworked several times due to dissatisfaction expressed by community members of all experience levels. I think it's fair to say that even though there are deciders of what happens, they hold themselves to the same standard.

I hope you'll consider at least taking a look as a hobbyist. It's a large system, so both lots of opportunity for contribution and lots to learn. I think many of us are very eager to work with people outside the team, especially as many strive for more social interaction in general, and I know many of us are individually looking forward to welcoming external contributions. Certainly we can never find out if the process works if no one tries.

edit: redundancies

There is not a RPi port, and that's non-trivial for reasons I vaguely recall to be related to display driver weirdness. The system does run on arm64le.

Maybe our section on packages will be of interest to you: https://fuchsia.dev/fuchsia-src/concepts/packages/package

It's kind of unfortunate because existing developers have the privilege of experience with the systems and where they're going that external contributors simply don't have, and it will take a non-trivial amount time for interested parties to develop that knowledge.

At the same time, it is an active project with active development that are informed by goals and processes not all of which are open. And really, while the development has been "in the open", it hasn't engaged the public until now. To that end, it's not possible to engage in a "bazaar" approach off the bat, whether or not that's a goal of the project.

Having been active in Go development and seeing some of the issues there, I understand what you mean. I don't think we state anywhere "this is clearly a cathedral model of development", but I think we're pretty clear on it:

* We have a section of documentation on project governance https://fuchsia.dev/fuchsia-src/contribute/governance

* We have a section of documentation detailing different kinds of contributors, acknowledging that there are kinds of contributors with special powers, and also reserving the right to revoke contribution privileges in some cases: https://fuchsia.dev/fuchsia-src/contribute/community/contrib...

To the extent that you can look at these as a set of policies, follow all the policies, submit a change, and have that change rejected, I think that is unfortunate. I think this is much less likely if you first engage with the stakeholders, and having opened up mailing lists, we've made it simpler to do that.

However, the "bazaar" is also a bit of a myth in this regard. I'm not really aware of any open source projects where I can go submit a PR without talking to anybody and have the expectation that it'll be merged without discussion. I can fork the repo, but that's also already the case with Fuchsia.

the fact they are only now taking potential contributions

We were honest about this too, and our documentation used to explicitly say that we did not accept external contributions.

Well, sure. Because people who want to develop products on some platform also want some kind of warranty, which is explicitly not provided.

Google establishes with the public a warranty- and guarantee-free use license. If someone feels they've been harmed by the software, they're prevented by the open source license from suing Google, but they're (more importantly) also giving up their right to sue the individual contributors.

At the same time, vendors want to be able to hold suppliers accountable for support and warranty on products that are supplied. Permitting sublicensing allows Google to sell the software to OEMs and vendors with support agreements that allow those vendors to hold Google responsible for that support (rather than the individual contributors).

There's a lot of complaint about the sublicensing going on, but I think most vendors aren't going to build systems where they can't get support, and I don't think contributors actually want to be liable under tort law for the code they provided for free.

Then again, maybe the objection is to giving code to Google that Google can profit from. I guess that's fair, but you don't _have_ to contribute code and you're also not prevented from sublicensing and trying to turn a profit yourself.

(edit: typo)

Is there? Apache 2.0 includes clauses for patent indemnification and allowing sub-licensing derived work.

Sure, but those projects can also trivially set up logging with their own bots and in their own clients, with better control of what is logged, when, why, and how those logs are shared. This service appears to have logged fewer than 100 channels. This wasn't an aggregate logger and anyway aggregate logging probably isn't permissible anyway due to GDPR.

Many IRC communities have shrunk. Different reasons. Things like Discord and mumble, combined with more collaboration and multiplayer competition in games took gamers away from IRC and crippled networks like Gamesurge.

Social media, especially things like Twitter and the various short video sharing services took away the "general chat" demographic -- partially because folks who were closer (real-life friends and family) like to feel closer, and folks who are not tend to be younger, and the younger demographic trends towards newer technologies.

Tech stuff has held up on freenode for a while, but as you point out this isn't always the case. I would argue that freenode is probably safe from too much decline for a while yet -- those of us who are "aging people who once used [IRC] in its prime" (I'm 36, guy, "aging" is a bit harsh :)) also tend to be people with a lot of experience in $technical_topic. If you want to ask questions about $technical_topic, and the people with experience are on IRC, that's where you go.

You've pointed out a lot of Javascript-related stuff is not on IRC. Javascript is a relatively new technology (compared to IRC), Node is relatively newer, and Elm newer yet. The boom in web technologies, and the ability to run native applications written in JS has caused that community to surge. It's also relatively easier to get into Javascript than it is to get into (say) C. So you attract more people, and you're going to attract younger people, and you're going to have a really diverse audience for a wide range of topics. All of a sudden a single channel on a chat network doesn't make much sense anymore, and running an IRC network yourself doesn't seem like fun, so you decide to use Discord or Slack. I think this makes a lot of sense for lots of communities.

For other (especially smaller, or very niche) technical communities, IRC is still a good solution. It's not fair to say that "its lunch has been eaten" -- it'll remain relevant until text-based messaging is no longer the most accessible way to communicate. But there are other, different solutions for similar problems that do a better job of catering to specific audiences. And that's just fine.

Unfortunately, it's kind of a PITA to get unique counts from server stats, which are mostly aggregates. So I can't answer that question. But Freenode's a rather active network.

I've used IRC for over 20 years and have never heard of this service. On your scale, it's "your second cousin's friend's sister's dog's goldfish had a VPS that went down."

This is #2 on HN because people are reading "Freenode IRC ... is shutting down." That'd be closer to the gmail end of your spectrum. We are not shutting down freenode.

Not sure what you mean. FIDL as a language and protocol is conceptually inherently cross platform. There are already bindings for multiple language platforms that can be generated from a FIDL specification, and theoretically one could implement a FIDL service anywhere. That said, FIDL services in the system provide a sort of ABI -- the F in FIDL stands for Fuchsia, after all -- and I'm not aware of any actual efforts to implement these on platforms that aren't Fuchsia.

This policy hasn't changed in over a year. I'm on paternity leave now, but my job is Go on Fuchsia, and I work with people doing Rust on Fuchsia. None of us are concerned for our jobs based on this document (which we've collaborated on).

This policy, like most technical decisions, may be amended when things change. We want people to have a consistent and stable platform to develop on, and if a language doesn't officially support our platform, it kind of doesn't make sense to support that language. And there's no commitment to support these languages for production services and end user development until there's a story for the stability of that toolchain on Fuchsia.

This shouldn't be surprising. Make a new system, bootstrap your programming environments. Why bother offering support for environments you've not yet bootstrapped?

As a thought experiment, consider the thousands of languages (including the tens or hundreds of popular ones) not listed on that page, and whether they're supported.

(Edit: I accidentally a word.)

Thanks for clarifying, I agree with all of this, and they're great points!

Yes, the paper is almost entirely focused on learning because it is a paper about applying somewhat recent research in social psychology into the area of debugging. It is not a paper about debugging. I can see how you might be disappointed if you expected a paper on debugging, but that is not what this is, and it says so at the top, in bold letters, above the abstract.

Anyway, I disagree with the rest of the points you make here, but I don't really feel like arguing about things that my paper is not (and was not) intended to be. I'm sorry you couldn't get much out of it, and thank you again for your feedback!

Thanks for this.

This seems to be basically what I'm saying through most of the article. Bugs occur because of a gap in a mental model. To fix the gap, we need more understanding. I'm not sure I would say that's necessarily a design or architecture flaw in the software.

Unfortunately, proving things correct in most popular languages is a Very Hard Problem. I do address that to some extent in the article, though. In particular, the sorts of languages that end up being easy to prove correct tend to be difficult for people to learn. Therefore, people are likely to either learn something easier, or they are likely to write code that is "correct" or "safe" but still has logic errors. I think that there are other many valid reasons to write software, and not all of them require correctness to the extent Dijkstra would like.

But I think that's a pipe dream anyway. For one, most software is not written in formally verifiable languages, or formally verifiable dialects of popular languages. Porting all software would be impossible. Second, who defines "correct"? What about bug-compatible software, where the definition of correct is imitating something else that is known to be incorrect? What about ambiguity in specifications like JSON, or undefined behavior in C (some of which can only be detected at runtime, because we don't actually have infinite tape in our Turing machines)?

What happens when there's a bug in the runtime of your formally verifiable language? What happens when there's a hardware bug? I'm not saying these are necessarily common, but regardless of your software environment, you're running on something else that can have the same bugs. People make mistakes, even when proving things. Math is great, but people make mistakes, and our tools are still bounded by the laws of physics.

No matter which way I look at this thing, I keep coming back to teaching and understanding as the only generalizable and approachable ways to address the problems bugs. (Whether that approach is taken by proving correctness of some implementation, or post-hoc with testing and tools.)

Thank you for the feedback. I chose (for better or worse) to cite the most recent article as opposed to maybe the least controversial one. Other papers I've read have suggested similar ranges; nobody presents a stddev so it's hard to compare. And how you classify "debugging-related tasks" is kind of subjective, as you correctly point out. So linking other papers might or might not be useful anyway.

In "A framework and methodology for studying the causes of software errors in programming systems" (Ko & Meyers 2005), they cite a NIST publication as saying 70-80% of time is spent debugging. I parroted this off in a talk once, and then I looked at the same publication ("The Economic Impact of Inadequate Infrastructure for Software Testing" RTI 2002) and the only 80% I found was time spent in testing software in the early days of software engineering. In the 1990s, it finds coding and unit testing to comprise about 30% of the time. This is as far as I can tell, at least similar to the (1) and (2) in the CVP survey. But again, indirect comparisons, who knows?

In tables 6-4 and 7-6, the RTI study finds different numbers for time spent on bugs, and different frequencies of where bugs end up being discovered. Whatever the rate, it is clear that post-production discovery bugs are the hardest to fix, and I think this is because you only ship code when you have high confidence in its correctness.

I think this is a hard measurement to make. It's hard to find participants, hard to guarantee they have the same idea of what "debugging means", hard to quantify. So thank you for calling me out on this.

But I think other studies have evidence that this isn't too far off from a correct measurement. For example, Gugerty and Olson in 1986 did some novice vs expert study at debugging LOGO programs. (I don't like the Dreyfus model for classifying debugging, but whatever.) They found that novices took about 18 minutes to fix bugs, testing an average of 3.6 hypotheses per program, with the first hypothesis being correct only 21% of the time, and re-executing code every ~3 minutes. Experts took 7 minutes, testing an average of 1.6 hypotheses per program, with the first hypothesis being correct 56% of the time, re-executing code every 2 minutes. So guesses are only right maybe half the time if you're really good.

Basically any novice vs expert study compares how much time people spend on the problem (they're trying to reduce debugging time), but it's hard to extrapolate this into percentage of development time. For example, some people spend 100% of their time debugging because they're on QA or sustaining engineering teams.

To be quite honest, there _fundamentally is not_ enough research into the practice and pedagogy of debugging. I have about 60 papers in my "Debugging Papers" folder that I've read over the past year, and basically everything up until the Dweck-inspired research in 2008 isn't great. There are only 9 papers in all of computer science that cite Dweck that I've found as of maybe 6 months ago (I'm not a researcher and I don't look that often). From memory, two of them were recommending further research based on her work, maybe five of them were from people in CS departments helping Dweck create games and programs to perform her research, and then only two were actually applying her work to debugging.

I would agree with you if you said this needs more research, and that the claim isn't strongly supported by the provided evidence.

I greatly appreciate your feedback.

Thanks for your feedback. This article was largely intended to explore why some folks just don't seem good at debugging, and don't seem to get better at it. This is where self-theories comes in. It does a rather good job of explaining that and has a large body of evidence supporting its validity. And importantly, that we can actually do something about that. I mean, in bold, at the top, it says "Understanding the psychology of learning strategies leads to effective problem-solving skills." So I think you should have been prepared for an article that wasn't going to talk about debugging strategy in depth.

I think you might have misunderstood what I meant with "develop a general theory." I say this because you've changed it to "develop a theory," which is explicitly not what I actually wrote. It literally means something completely different in the context of science. A general theory is just a concept. It's the formation of ideas. It's the process of information gathering. Where you go from "this is a bug" to "this is a bug in the logging module" to "this is a bug in the logging module when I'm producing more log messages than I can write" to "this is a concurrency bug" to "aha the actual bug is somewhere else, let me look there", etc.

Your first tangible goal should be reproducing the bug, because that is how a hypothesis is tested. But you don't magically start with a hypothesis unless you have very good intuition, which means this isn't a new bug for you, and so you didn't really need to employ any complicated strategies. My article is not about how to handle obvious, run-of-the-mill bugs.

And so between the implicit step 0 (there is a bug) and step 1 (I can reproduce it), you end up assuming quite a lot.

I mention in the article that reproducing a bug is really crucial, because that is how we know we've found where the gap in our mental model is. You've gone on to describe some ways to do that. I explicitly did not take this approach, because such a list would be exhaustingly long, and my article is already long for an ACM Queue article.

If you've reproduced the bug, you must have already figured out what was wrong. How did you do that? Were you staying up all night staring at the problem instead of taking deliberate breaks? Were you collaborating with colleagues or stubbornly tackling the problem alone? Were you scientific in your approach? These are all things that stem from the root of the problem, and are maladaptive strategies that folks engage in. Strategies that prevent them from getting from step 0 to step 1.

So it seems odd to me that you would call my "approach" flawed. To me, it seems you're not actually starting from the point of "I do not understand the problem." If you go to github, pick a random issue in a random project in a random language, and go to fix it, what is the likelihood that the first actual step you take will be reproducing the bug? This is where people actually start from, and this is where we are every time we encounter a new bug.

I'm not sure I agree with the order of your steps. If you are new to software (new job, entirely new codebase), and one of your first tasks is a bugfix, how are you possibly going to have step 4 ordered before step 1? You need to have some kind of mental model. You're going to need to talk with colleagues, interact with people. Gather some understanding of the system architecture. This is a learning process and an information sharing process. _This is the point of my article._ Thank you for taking the time to share your experience with us!

I'm sorry you didn't like my article. It sounds like we agree more than you seem to think we do, even if I'm a bit critical of your steps. I think we have different audiences in mind, perhaps.

Which sounds like a variation on various Socratic learning techniques.

Exactly this.

Thanks for this comment. I did worry that this wouldn't come across very well, but my attempts to make it explicit all ended up poorly worded or read (to me) patronizingly. Still, to me it seems that you actually agree with what I've written.

Forming a hypothesis it seems is the least understood / most confused part of approaching something scientifically. It does not mean that you need to test each thing individually every time you touch something. "Changing individual variables" as applied to this doesn't mean adding one log line in one place and moving it one step every time. Get the information you need! Be practical! I recognize state space explosion (the combinatoric complexity you refer to) in the article as a primary reason that debugging is difficult -- this complexity makes it difficult to form absolutely correct mental models about software.

I'd also mention that if you're not systematically gathering data, you're not being very scientific. I didn't dig too deep into that because I wanted to avoid being patronizing. If folks aren't familiar with the scientific method, there are probably better references than me.

I think what you're calling "divide and conquer," I am including (implicitly) as a strategy in the information gathering portion of the scientific method. I wish that I had more room to put more of these sorts of strategies in, but my article is already past the recommended length of Queue articles.

The divide and conquer strategy is mostly generally effective, but it's not a universal tool. Debugging a race condition in some concurrent systems using this strategy is one example of where it fails if you're using it to get to a hypothesis. It might get you to "there is a race condition in my code," but this is not a hypothesis. It will not generally get you to, "there is a race condition between points A and B of my program when condition C is true and object O is not yet initialized." You can hopefully see how I can go from here to bring combinatoric complexity back into this strategy, turning O(log N) into superlinear O(log N!) (since factorial dominates exponents). And there's really nothing to be done about it: as I mention in the article, software describes the change of state in a system over time, and combinatorics is a fundamental part of state space complexity. For many modern systems, and increasingly, combinatoric complexity is a fundamental part of the equation.

Finally, I felt that including strategies that were not globally generalizable and applicable to _all_ bugs would be in error. The paper is largely about cognitive / social psychology and pedagogy, less about process. I definitely agree that divide and conquer is a great strategy to use for gathering information in the debugging process. Use the approaches that have worked well for you in the past! Share those with people! Thank you for sharing this approach with people!

--dho