beliu

In case folks are interested, I had a delightful interview about a year back with Max Brunsfeld, one of the original Atom devs who is now working on Zed: https://about.sourcegraph.com/podcast/max-brunsfeld. (YouTube link: https://www.youtube.com/watch?v=0Jf8AqaZ7IU)

Very excited about where this new collaborative editor goes!

LOL! He did talk a lot about his programming backstory (played with BASIC as a kid, no formal training, chemistry major, discovering Linux) and how this informed his worldview. Bottom-line, he is very results-driven and I think he has a good point about how a lot of tech interviews screen too much for theory and not practice. Turns out you don't need to know how to invert a binary tree to build the most successful package manager of all time :)

This is super common in my experience. Senior/experienced devs spend a bunch of time ramping up the inexperienced or new members of the team and don't have much time remaining to work on their own initiatives.

Patrick McKenzie has a good take on this along the lines of "teach someone to fish" (s/fish/code search/): https://mobile.twitter.com/patio11/status/108611151367121715...

We've tried to make it easier to turn organic code explorations into interactive docs that can be read many times over with a notebook-like feature. Here's a notebook that bobheadxi wrote up after he noticed he was explaining Sourcegraph's authz model for the billionth time: https://sourcegraph.com/notebooks/Tm90ZWJvb2s6NTc0

Thanks! 100% agree with the sentiment that devs should get in front of customers. The best teams I've worked on, the devs have an "end-to-end" sense of ownership, and the organization fosters this sense of responsibility and agency. And great work should absolutely be recognized. It should be a win-win-win—better software for end users, more revenue for the company, and financial + creative satisfaction for developers. We are working with all our might to push the world in that direction.

World's best code search: https://sourcegraph.com/search Analyze code as data: https://sourcegraph.com/insights Interactive docs you'll actually use: https://sourcegraph.com/notebooks Take the pain out of large-scale refactors: https://sourcegraph.com/batch-changes IDE superpowers in code reviews: https://docs.sourcegraph.com/integration/browser_extension

This is really important to call out. There have been a lot of really bad practices in the past that try to reduce "developer productivity" to something that approximates a widget factory, and the results have been awful for both developers and companies. This feeds right into the last section of the post, "If we don't talk about developer productivity, someone else will"—we need more developer voices advocating for the creative spark of building software.

Sourcegraph actually does do proper semantic linking, integrating with compilers (and type inferencers for dynamic languages): https://docs.sourcegraph.com/code_intelligence/explanations/.... This is one area of significant differentiation between Sourcegraph and GitHub code navigation (which is based on GitHub's semantic library, and which ironically isn't actually semantic in the sense of understanding types as compilers do).

Admittedly, we need to do a better job of articulating this difference, but if anyone would like help getting spun up with precise code intelligence on Sourcegraph, please DM me: https://twitter.com/beyang.

I’m helping with the doctree effort at Sourcegraph. Apologies, the site isn’t actually up yet. This project is still very early stages and we wrote up the README to serve as a sort of launch spec that we could update in response to feedback we receive. We made the repo public so we could build in public but didn’t expect it to receive this much attention this early!

So sorry that the site isn’t up yet. We’ll update the README soon to reflect that. If folks are interested in trying out a super early version, there’s the Docker run command and if you’d like to help us realize this vision, please join our Discord! https://discord.gg/vqsBW8m5Y8

Sourcegraph CTO here. Does anyone know someone at GitHub or Microsoft we could get in touch with to take over maintenance of git.io? There are hundreds of thousands of references in open source, blog posts, and academic papers that will 404 if git.io is taken down. Sourcegraph would be happy to take over maintenance to preserve these links.

Here is a list of open source projects that should update their git.io URLs, ranked in order of popularity: https://sourcegraph.com/search?q=context:global+https://git....

Actually that’s not quite right. We offer auto-indexing now for some languages with more on the way. If neither auto-indexing nor manual LSIF generation are an option, we fall back to our search-based code navigation, which, similar to GitHub’s implementation, trades off accuracy for zero configuration (GitHub’s approach is parser-based while we use a combo of parsing, search, and heuristics that eliminates the need for an index entirely in some cases). This is all documented here for those curious to learn more: https://docs.sourcegraph.com/code_intelligence.

This reminds me of Shaple: https://swag.github.io/shaple

Thank you for this super clear and concise write-up. Used it to write up these instructions for our users and customers to patch the vulnerability across their codebase and sharing here in case it's of use/interest to others: https://twitter.com/beyang/status/1469171471784329219.

Appreciate the feedback here (Sourcegraph CTO). I agree that's annoying. Kicking off a conversation on our end to figure out how to fix that. In the meantime, the workaround would be to create a separate GH login, which can be used to add any public repo to our index. Or what's the URL of the repo(s) you'd like to index and we'll get those added to our index.

Hey Edwin, if you're open to providing feedback, I'd love to understand which types of searches worked well for you in Google Code Search but not in Sourcegraph. We've invested a lot of thought into our query syntax, supporting literal matches, regex, and the Comby pattern matching syntax with a rich set of keywords and filters—but we know the syntax isn't always intuitive for every user. We're always trying to improve the experience for all our users (I'm the CTO at Sourcegraph), so if you have any recollections you're open to sharing, would love to hear them!

Sourcegraph is open-core, with a dual licensing approach. You can run the open-source version here: https://github.com/sourcegraph/sourcegraph#sourcegraph-oss, and we have an enterprise offering for companies that want to adopt for their teams. Similar to GitLab, both our enterprise and OSS code is publicly available.

We built Sourcegraph taking inspiration from Google Code Search (https://about.sourcegraph.com/blog/ex-googler-guide-dev-tool...) to bring the power of code search—and precise code intelligence that just works—to every dev. Try it out here: https://sourcegraph.com. A super common thing we see is people leaving Google, missing code search, and then bringing Sourcegraph into their new org. We'd love to hear your feedback!

Sourcegraph CTO here, I'm very sorry this happened with you. If you shoot me your email (I'm reachable at beyang@sourcegraph.com), I'll ensure you don't receive any future sales/marketing emails from us.

This is something we should be clearer and more upfront about. We collect emails for sales purposes from self-hosted Sourcegraph instances, but I agree that entering your email as part of the installation process doesn't mean you want us to email you directly. Going to fix this!

Sourcegraph CTO here, thanks for posting this feedback.

Yes, we do collect information from self-hosted instances of Sourcegraph. (Note to other readers: the blog post is talking about sourcegraph.com whereas what we are discussing here is running a standalone instance of Sourcegraph.) Here is what that info is:

(1) High-level "ping" data (https://docs.sourcegraph.com/admin/pings#critical-telemetry) that includes the email address you put in during installation, the version of Sourcegraph that's running, an aggregate count of users, total codebase size.

(2) Additional telemetry (https://docs.sourcegraph.com/admin/pings#other-telemetry) that includes aggregate usage, latencies, and product actions (e.g., which features are in use, progress through onboarding). This can be disabled in config: https://docs.sourcegraph.com/admin/config/site_config#disabl...

No information about individual user identities, behavior, repositories, or code is sent outside the Sourcegraph instance, unless you explicitly enable a feature (like code monitoring alerts) that does so.

We follow the open-core model and our enterprise-licensed code is also public. You can use Sourcegraph to explore the source code of Sourcegraph and see how telemetry is implemented: https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-....

We collect this data in order to sell our software to companies that use it. We have no plans to charge for Sourcegraph for open-source development, nor will we sell individual user data (which we don't collect).

We've made the decision to make (1) mandatory, because we haven't figured out a better way to ensure we have reasonable awareness of which companies are running Sourcegraph. We do try to keep this data as high-level/aggregate/non-invasive as possible, but I also understand that some might not want to send any data. For folks that fall into that camp, there are a number of other code search engines that are great: Livegrep, Zoekt, Hound, and OpenGrok are ones I'd recommend. If you think the feature set of Sourcegraph is cool and want to use it strictly for personal or open-source development while disabling all telemetry, feel free to reach out (beyang@sourcegraph.com) and I'd be happy to do what I can to make this happen.

Of course, if you don't want to run your own instance of Sourcegraph and are okay with using a cloud service, you can also add your repositories directly to sourcegraph.com, where they can be discovered and used by anyone from a single search box :)

If folks have any questions or feedback on the above, I'd love to hear it!

Hey, I'm the CTO/co-founder of Sourcegraph. Thanks for the feedback here. I've opened up a pull request to make it clearer how to disable non-critical telemetry: https://github.com/sourcegraph/sourcegraph/pull/24171. Please let us know if you have any feedback on that!

Will respond to the OP's concerns, which I agree are important, in a direct reply.

Sourcegraph | Full-time Eng, Product, Design | REMOTE | https://sourcegraph.com

We're building the Google for Code and making great code search (the kind that Google, Facebook, and Microsoft have internally) accessible to every company and every open-source project.

* SRE

* Security Engineer

* Sales Engineer

* Software Engineer (frontend, backend, full-stack)

* PM of Growth

* And more, full job listing here: https://github.com/sourcegraph/careers

Tech stack: Go, TypeScript, React, Docker, Kubernetes

You can try before you apply: https://sourcegraph.com/search

Author of the post here. I would say that technically Kythe is the open-source version of part of Google's internal code search (specifically, the component that provides precise code navigation), but it doesn't include the search index or the UI. So it's not on its own an end-user product.

That having been said, I love Kythe, and we've actually considered using it as a semantic backend for Sourcegraph (and still might in the future). For the time, we're using indexers that emit LSIF (https://lsif.dev). This allows us to build on top of the substantial body of work provided by the many open-source language servers (https://microsoft.github.io/language-server-protocol). But Kythe has a far richer schema that can capture all sorts of useful relationships in code. It's awesome and I wish more people were building indexers for it.

Author of the post here. We're actively working on this at Sourcegraph. Up until recently, we've focused on working well on large private codebases, but looking forward, we have two big efforts that will make this a reality on sourcegraph.com:

1. Vastly expanding the size of our global search index to cover every public repository on github.com, gitlab.com, and bitbucket.org. 2. Enabling LSIF indexing (https://lsif.dev) in every major language for compiler-accurate code navigation (go-to-def, find-refs) across repository/dependency boundaries.

The latter is already working on a subset of languages for private Sourcegraph instances, and we want to scale it to the entire open-source world. We think a single search box that covers all the visible code in the world and allows you to seamlessly walk the reference graph is super super powerful and someday will be a thing that most developers use every day.

If you're looking for a GCS replacement, Sourcegraph is a thing! We've taken a lot of inspiration from Google Code Search (both the public version and the internal version which was even better): https://sourcegraph.com

Sourcegraph co-founder here, thanks for the mention :)

Doing code search well is basically what Sourcegraph is all about. We support full text and regex search over many repositories. Sourcegraph.com indexes most repositories on GitHub.com and we also have a Docker container that you can spin up to index code on private code hosts.

Our search is fast and our query syntax is powerful: https://docs.sourcegraph.com/user/search/queries.

Most of our users are daily users and most of our customers deploy us across their entire engineering organization to make all code accessible to all developers (subject to permissions). We view search not as a single feature, but a large collection of related, but distinct use cases:

* Honing in on error messages

* Debugging and resolving production issues faster

* Comparing different packages / libraries

* Figuring out how to use a particular API

* Understanding what a function does

* Detecting patterns and anti-patterns

* Enforcing code quality best practices (saved searches)

* Large-scale code changes (easy find-and-replace across your entire codebase)

* Walking the reference graph in code (i.e., jump-to-def and find-refs, and tooltips as you browse the code)

We were inspired by a lot of the great code search tools that companies like Google and Facebook have built internally for themselves. We're also investing in promulgating new pattern matching languages (e.g., https://comby.dev/) that we think will yield huge productivity gains for programmers who find regex has its shortcomings.

We will always be free to use for individual developers. Our code is open-source. We make money off of enterprise. You can try us out on open source (https://sourcegraph.com/github.com/sourcegraph/sourcegraph) or spin up your own free instance (https://docs.sourcegraph.com/admin/install/docker). If you end up checking us out, would love to hear your feedback!

Sourcegraph CTO here.

For example, if you pay for GitLab's self-managed "Starter" package, and assuming you spin up your own private Sourcegraph instance [1] to go along with your self-managed GitLab instance, what Sourcegraph features do you get? Will you get code review intelligence, which Sourcegraph charges $29/user/month for?

The GitLab pricing tier has no bearing on Sourcegraph features—it is the Sourcegraph pricing tier that determines what Sourcegraph features are available. $29/user/month is the Sourcegraph Enterprise tier, which includes full code review intelligence (jump-to-def, find-refs, hover tooltips).

Note also that Sourcegraph customers are only charged for active users, so if most of your team doesn't end up using the features provided by Sourcegraph, you aren't charged for them.

On Sourcegraph's pricing page, it says the open-source edition is missing features like "Single repository definitions and references" and "Cross-repository definitions and references". What does that mean for private GitLab instances?

"Single repository definitions and references" means go-to-def and find-refs work if the code you're trying to jump to is in the same repository. "Cross repository" means you can jump through to code in upstream and downstream dependencies.

If you're running a private GitLab instance, you'll need to deploy a Sourcegraph instance (runnable as either a single Docker image or Kubernetes cluster) and pay for the appropriate license (free if you're under 10 active users and just want single-repository code intelligence, $29/user/month if you're over 10 active users or want cross-repo code intelligence).

Sorry about the confusion here—does that answer your questions?

It is very exciting to see GitHub working on code intelligence features for open source.

Many folks are asking how this compares to Sourcegraph, so just thought I'd mention if you like this, you will probably like the Sourcegraph browser extension (https://docs.sourcegraph.com/integration/browser_extension). It provides:

* jump-to-def

* find-refs

* documentation tooltips

* fast, quality code search

* an open plugin API (https://sourcegraph.com/extensions)

with:

* high (in some cases, compiler-level) precision

* across many languages

and supports many code hosts:

* GitHub

* GitHub Enterprise

* GitLab

* Bitbucket Server

* Gitolite

* AWS CodeCommit

* generic git servers

for:

* public code

* private code

* on-prem private code

and:

* the source code is open

* there's a public issue tracker: https://github.com/sourcegraph/sourcegraph

(I am the Sourcegraph CTO :) )

The internal representation depends on the tool and application. Here are a few examples:

* For deeper analysis, we often use the same internal data structures that the compiler itself uses. This includes data structures that capture AST and type information. In the case of some languages, the compiler doesn't capture enough information (e.g., dynamically typed languages with no static typing). In those cases, the data structures are often enhanced/similar-but-not-quite-the-same as what the interpreter/compiler uses.

* In "shallower" analysis, you can use a language-agnostic parser to generate a generic AST. There's various techniques for doing so in a language-independent fashion, depending on what tradeoffs you want to make with speed, accuracy, and generality. The tree-sitter library is one such example. You can also use tools like CTAGS + textual heuristics in the extremely shallow case.

* Then, there's the Kythe approach, which constructs a taxonomy of source code graph nodes and edges that aims to capture every possible relationship in every language. Each language requires an integration to convert from the source code in that language to the Kythe schema; that integration can use either the "shallow" or "deep" approaches described above, and the right answer depends again on your tradeoff of speed vs. accuracy.

As for combining dynamic information like profiling and runtime info, this is something we've been thinking about for awhile. We think there's plenty of low-hanging fruit that combines static information (types, AST, source code) with runtime info (stack traces, profiling info). We've a couple of efforts in motion toward this end. One is the Sourcegraph extension API (https://sourcegraph.com/extensions), which allows injection of third-party info, often from runtime tools, into wherever you view code, whether it's in your editor, on GitHub, or on Sourcegraph. Another is allowing a user to plugin stacktrace data to heuristically select which jump-to-def/find-references options are most likely to be useful given what your currently debugging.

I'd love to hear what you're thinking of—care to share your ideas in this space?

If folks are interested in this sort of thing, either because they want to use it or because they want to hack on it, I'd humbly suggest checking out Sourcegraph: https://sourcegraph.com, https://github.com/sourcegraph/sourcegraph.

We've spent years building semantic code intel and search that's available for open source and software teams that aren't inside Google, which has an amazing internal code explorer powered by Kythe (though, unsurprisingly, a lot of our users and customers are ex-Googlers!)

Much thanks to GitHub for open-sourcing this. Our own open-source code analysis stack uses tree-sitter (as this does) in addition to other language analyzers via LSP, and this seems like a great contribution to the open-source code analysis canon!

Sourcegraph (https://sourcegraph.com) | Software Engineer | San Francisco, CA or REMOTE Sourcegraph is on a mission to make it so everyone, in every community, in every country, and in every industry can create products using the best technology. The innovations of the future will all rely on software and we want to bring the future sooner by building the best tools that help developers answer questions about their code.

With Sourcegraph, you can instantly search across all of your company's private code using powerful regular expressions, and with our browser extension you can go-to-definition and find-references while you are reviewing code on GitHub and other code hosts. Most of the developers at Uber, Lyft, and Yelp use Sourcegraph every day.

We are a small distributed team of mostly engineers who love to code. If you are passionate about making the world better through software, come join us!

Sourcegraph is an equal opportunity workplace; we embrace diversity and welcome people from all backgrounds and communities.

Technologies that we use: Go, TypeScript, React, RxJS, GraphQL, Docker + Kubernetes

Apply here: https://github.com/sourcegraph/careers/

Read our master plan to learn more about what we're building, and why it matters: https://sourcegraph.com/plan

Our code and our product roadmap are open source!

https://github.com/sourcegraph/sourcegraph

https://docs.sourcegraph.com/dev/roadmap