Notes on the Go translation of Reposurgeon (2020)

https://gitlab.com/esr/reposurgeon/-/blob/master/GoNotes.adoc

by spekcular • 5 years ago

127 151 5 years ago

Raphael Poss did a really insightful talk on how to gracefully panic[0]; this gives try catch like functionality, though should be used sparingly IMO.

- [0] https://m.youtube.com/watch?v=c78U0MZ4b_c

whoisjohnkid4• 5 years ago

Regarding your point on iterators, I typically use a channel with context; this lets you exit that range loop early and will automatically stop the related go routine. (assuming it’s context aware)

e.g.

func myFunc(cx context.Context) {

  ctx, cancel = context.WithCancel(cx)

  defer cancel()

  for item := range iterator(ctx) {
      
  }
}

jerf• 5 years ago

The major problem I have with that is performance is atrocious [1] if the iterator isn't doing something very nontrivial. Since the vast majority of iterators amount to incrementing at most a handful of things and then returning something indexed by those things, you're paying the cost of channel communication per item but not getting that channel cost amortized over any significant costs in the iteration itself. If the thing using the iterator is also not doing anything significant (i.e., adding integers together), this has very bad performance implications.

As a rule of thumb, the amount of work transferred by any concurrency primitive should significantly exceed the cost of the concurrency primitive itself. I do have a couple of uses of this pattern where what is on the other side of the channel is something reading off a network and parsing lines of JSON into internal structs, in which case the overhead of the channel isn't necessarily too bad. (In one case, it even chunks the lines of JSON into a slice of several parsed structs, reducing channel overhead even more.) But it's a terrible solution in general; iterating over an array and doing any sort of very fast "thing" to each element that only costs a handful of assembler instructions, a very common use case, has terrible overhead.

It's a real pity, because the semantics of that solution are pretty close to the right answer. But it's a huge performance trap. Something as basic as iteration needs to not be a huge performance trap.

[1]: Relative to Go, anyhow. I haven't timed it directly but I wouldn't be surprised that a channel-based iterator like that would be comparable to Python's general iteration speed, or at least not off by a very large factor. It's just that "Python's normal level of performance" is "atrocious Go performance".

whoisjohnkid4• 5 years ago

Hmm, can you provide some examples of how a context based iterator is slow?

If you need raw performance and you are just doing some minimal operations on a slice then yeah you’d want to use a simple for loop instead.

I typically use this pattern in situations where you have a ton of data coming back where you want to avoid storing all of that data in memory.

badhombres2• 5 years ago

Great writeup. Some of the points are moot now that generics are being released, but some very valid concerns.

I would also add that Enums + Exhaustive Switch is a very very weak area in Go that would really benefit the language a ton. I've used those features in other languages and that's one of the things I miss the most, especially when dealing with a ton of web API's that have a defined set of values for properties.

Able to have the confidence that we're checking for every situation that could occur on an enum type across the codebase is one less thing I need to worry about.

thegeekpirate• 5 years ago

Able to have the confidence that we're checking for every situation that could occur on an enum type across the codebase is one less thing I need to worry about.

golangci-lint (https://github.com/golangci/golangci-lint) is an absolute must, and includes https://github.com/nishanths/exhaustive which will check this for you.

badhombres2• 5 years ago

thank you, I'm going to take a look at these!

henvic• 5 years ago

my 50 cents:

I'd rather just use go vet + staticcheck. Way simpler, no complex configuration, and no license concerns :)

and then you can create a file like the following and add whatever analyzer provides value for your specific case (including this exhaustive), easily. Example: https://github.com/FiloSottile/mkcert/blob/master/analysis.g...

preseinger19• 5 years ago

Configurable linters are usually best avoided, because it's too easy to remove a check rather than fix your code, in a crunch.

marcus_holmes2• 5 years ago

Seconded on Enums. It's annoying that they're not there.

throwaway89434523• 5 years ago

Strongly agree here. I'd rather have this than generics, personally. Having to implement my own sum types every time is laborious (moreso than most boilerplate that people complain about) and it's also hard for users to understand (what are all of the permutations?) and extend (I added a new permutation; where are all of the sites I need to update?).

tjalfi• 5 years ago

go-sumtype[0] has completeness checking for sealed interfaces.

[0] https://github.com/BurntSushi/go-sumtype

LysPJ• 5 years ago

Really interesting read. Especially the suggestion for how to support iterators.

It should be noted that the writeup is a year old, which (understandably) means that the comments on generics are a little out of date.

wpietri7• 5 years ago

What a delight to read, even as somebody who has barely used Go. I really appreciate the author's self-awareness as demonstrated in things like "Expected problems that weren’t".

The bits about exceptions and typing remind me of an open question I have about Go: to me it mainly looks like a language for well-understood problems. Static typing and the lack of ability to do broad, high-level exception catching seem to me to be well-matched for problems where you already understand the solution pretty well. E.g., a port like this.

But how is it for poorly understood problems? E.g., You're doing a startup where the technical risks are low and the major unknowns are about user needs and the correct experiences to deliver. Or you're doing exploratory work for an art piece.

In those contexts, I've felt much more effective starting with vague/implicit typing and some broad, high-level exception-catching blocks. That lets me avoid a bunch of questions about the right types and structures until a more solid domain vocabulary emerges. At that point you can refactor/rewrite toward clarity. But I don't see a good way to be willfully vague/casual in Go.

whoisjohnkid4• 5 years ago

All depends on what’s important to you in that startup context. If you need a good mix of high productivity and low costs on the server side I think go is a good fit.

If you don’t care so much about costs and scale on the server side then something like Ruby on Rails might give you that extra productivity boost you need to verify the concept.

wpietri7• 5 years ago

Yeah, given how computers keep getting cheaper, I only start caring about low server costs once I have a demonstrated cost problem that's going to be material to the business model or to the budget. Until then I'm very happy to burn CPU time to accelerate developers. Who keep getting more expensive!

And yes, the point of verifying the concept is key for me in this. Before product-market fit, I just don't have much confidence in any domain model. Once we have demonstrated that particular people are excited to pay for a particular thing, that changes. Then we can understand how those people think, and how we think about their behavior, such that we can do real domain-driven design. At that point I'm much more willing to lock down types.

laumars• 5 years ago

I’ve never once felt like strong typing has prevented me from solving unknown problems. I won’t deny that there have been occasions where I’ve had to redefine a type across a large code base and that has taken me half an hour to a hour. But I’d take that over type validation at run time.

For me, having strict typing helps with unknown problems because it gives me greater confidence that refactoring wouldn’t introduce subtle regression bugs.

wpietri7• 5 years ago

Thanks for the reply.

For me the cost isn't just redefining a type. It's the continuous work of reifying types that could be implicit for a while and possibly forever, because concepts don't last long enough to need it.

For me, having written a lot of code in Java and Scala but also in Ruby and Python, typing is a nice adjunct to unit tests and operational monitoring, but not my fundamental way of ensuring correctness. And correctness often isn't the highest value in practice, especially early on in a startup's life.

marcus_holmes2• 5 years ago

Honestly, you don't notice it once you get into the Go mindset.

The exception handling looks onerous, and is a pain at the start. But it doesn't take long to get used to it, and then (for me anyway) it becomes second-nature. Everything returns an error, and you have to handle that error (even if only passing it up again).

Static typing is more "fun" when you're exploring new concepts, but interfaces are the key. Defining the interface at the beginning ("what do I need this thing to do?") and then implementing it with a struct/whatever, feels like the right way to do this.

oefrha2• 5 years ago

But it doesn't take long to get used to it

I did get used to it, but all the

  if err != nil {
    return nil, err
  }

sometimes taking up half or more of the vertical space of a function is still an eyesore to me, even after writing a quite substantial amount of Go.

wpietri7• 5 years ago

Not noticing it is exactly my fear.

Passing an error manually up several levels when it may only be a theoretical concern is a ton of expressive duplication. If I'm in the mindset, that would feel like valuable work, even though it isn't actually making things better for users.

It reminds me of one team I dealt with years ago. They were Java experts used to doing enterprise stuff; their overlords had put them on a scrappy, startup-like thing that was intended to be open source. As was in fashion for Java at the time, they had written things in many layers, the goal of which was to provide scaling cut-points. However valuable in theory, the project ever needing to scale was uncertain. What really mattered was finding something that served user needs. But the layer-by-layer duplication increased the cost of change significantly, lowering the odds we'd find the right product.

autarch4• 5 years ago

I feel like strong typing _helps_ when exploring a new domain, as it forces you to really think about the data you're operating on. In a language like Perl or Python, it's so easy to just throw around a bunch of hashes/dicts and that can get messy really quickly.

I don't miss _exceptions_ in Go but after using Rust for a while I've come to really love Rust's `Option` and `Result` types. They're more ergonomic and expressive than Go's errors-as-values, and the use `Option` also eliminate `nil` entirely, which has been a regular source of runtime panics in every large Go codebase I've worked on.

tialaramex3• 5 years ago

Sum types are a thing I don't want to be without again. Not only the built in Option and Result, but once you're in the mindset you find they're the right fit for other problems too.

In Rust itself, writing stuff where I actually care whether my structure is 16 or 20 bytes because I need to fit hundreds of millions of them into RAM, I like that Sum types ensure Option<NonZeroUsize> is the same size as usize, by reasoning that 0 isn't a valid NonZeroUsize and so it can signal None. However on a language like Go I don't miss that - what I do miss is the fact that in Rust I can't mistakenly end up with None(actually_something) or both an error and the result that shouldn't be there if there was an error.

teddyh• 5 years ago

Python is strongly typed, not weakly typed. What you are talking about is static typing, which Python is not; Python is dynamically typed. Please do not confuse the two concepts.

kraig• 5 years ago

With generics around the corner I am really hoping we see some flavor of Option & Result coming to Go. But I won't hold my breath for ?.

wpietri7• 5 years ago

It only helps to the extent that being forced to really think about the data is net useful.

In conditions of high volatility, that's not always the case. If I create some throwaway prototype to test a hypothesis based on talking to a few users and then test it on a dozen more users, I don't want to think a ton about the domain. The sheaf of possibilities is often quite large, and narrowing down the possibilities requires better understanding of the users and the things we're creating for them. That understanding is only available in the future, and we only get to that future by making something.

I really do love designing clear, expressive type systems. But the beginning of the project is when we know the very least about what will happen, so it's the worst time to invest in those type systems. It can be fine anyhow if the domain is stable and well-understood. Which I gather is Go's sweet spot, and I'm fine with that.

nicoburns• 5 years ago

The article mentions that Go, OCaml, or a compiled lisp were considered for this project. I wonder why Rust wasn't on that list. It seems to cover every concern that is raised here except for keyword arguments (which are high on my list of desired Rust features too).

I guess maybe they were worried about the lack of GC, but my experience has been that's it's generally quite easy to port code from dynamic languages like Python/JS/PHP to Rust. So I suspect that may have gone into their "Expected problems that weren’t" section if they'd tried it.

mwcampbell2• 5 years ago

Here is what esr wrote about Rust in early 2017:

http://esr.ibiblio.org/?p=7294

http://esr.ibiblio.org/?p=7303

kubb4• 5 years ago

It sounds that it was hard to learn and use for him, that's why he preferred Go and Python. It's a frequent complaint and the step from Python to Go is indeed easier than from Python to Rust. He also had some philosophical concerns about the crate system.

brandmeyer2• 5 years ago

A lot of Rustaceans don’t seem to grasp why, when the question is “where do I get feature X?” the answer “oh, there are 23 crates for that” is objectively terrifying.

This is exactly the impression I have gotten from the crates system every time I've looked. It happens for things as foundational as mmap. Also terrifying: The plethora of highly-recommended crates that have not yet committed to a stable API (ie, are still semantically on version 0.X).

db48x9• 5 years ago

Async wasn’t in yet at the time. He did investigate Rust to a certain extent, but bounced hard. His chosen program didn’t really show off Rust’s strengths, because he started by trying to call select and write essentially the same program that he would have written in C. That’s a pretty painful way to go.

Incidentally, I helped with the port to Go, and I spent some time polishing the code, and finding and fixing performance problems. When we were helping GCC convert their SVN repository to Git (a repository with 287k commits, btw), we reduced the memory usage by 50% (from over 250GB to under 128GB), and the run time by quite a lot as well (down to just around 2 hours to read in the SVN repository and convert it to a basic Git repository).

Now that we’ve done that work, Reposurgeon spends 50–60% of its cpu time scanning the heap for garbage. There is often garbage to find, but just as often there is not. GC is useful, but for Reposurgeon it has become a bottleneck.

My preliminary work on a Rust port shows that it is around 4× faster than the Go version. I personally think that Rust is the future, but I haven’t been able to put as much effort into the port as I would like.

stouset2• 5 years ago

Appreciate your thoughtful reply! One question. Regarding this bit:

He did investigate Rust to a certain extent, but bounced hard. His chosen program didn’t really show off Rust’s strengths, because he started by trying to call select and write essentially the same program that he would have written in C. That’s a pretty painful way to go.

Is this stating that he was writing the Rust version similarly to how he would have written it in C because that was the most natural way to do it in Rust or simply because it was the first path he went down? In other words, was your point that Rust is a fundamentally poor fit for this problem, or was it that his lack of experience with the language took him down the wrong path?

Thanks!

steveklabnik• 5 years ago

Two threads from back in the day about this:

https://news.ycombinator.com/item?id=13385530

https://news.ycombinator.com/item?id=13398068

eikenberry• 5 years ago

Go, OCaml and Lisp (depending on the Lisp) all have one other thing in common.. a fast compile/feedback loop. Rust doesn't have a good story here. This would be my guess as to why (at least in part). Particularly coming from a dynamic languages which have super fast compile times if they compile at all.

mwcampbell2• 5 years ago

Personally, I would have gone with Kotlin for a translation like this. It's a closer match to Python than Go, and I think his rule swarm [1] approach to semi-automated translation would have been effective. But then, he might not have liked the fact that, like Python 3 and unlike Go, the Java platform treats strings (particularly filenames) as a sequence of Unicode code points rather than bytes.

[1] http://esr.ibiblio.org/?p=8153

colin_mccabe• 5 years ago

Go treats strings as a sequence of unicode code points.

fnoof• 5 years ago

I believe they’re treated as byte sequences: https://go.dev/ref/spec#String_types

dbt00• 5 years ago

It's a closer match to Python than Go

I like Kotlin, but I really think of Go as the most probable outcome of someone saying "I'd like a staticly typed, concurrent/parallel Python", and then mumbled "but I hate exceptions". Go is very close to python in many respects.

kubb4• 5 years ago

What I really missed was generic map-function-over-slice

Me too, and mapping a map, a channel, etc. With generics, things will be easier, we can have iterators, even lazy iterators, but there will still be no type inference in lambdas for arguments and return values and no easy currying, so things will still be awkward, but better.

Keyword arguments

You can kind of do that by passing a struct as an argument, and represent the optionalness by having a pointer in the struct. That's still awkward, you can't easily make a pointer to an integer or string literal, you need a separate helper function for each primitive type. Overall it could be nicer.

Annoying limitations on const

I think the Go constants are good, the fact that you can perform calculations on them in compile time also. But I wish there was some way to express immutability (and non-nilability) in the language. Programs are simpler with less moving parts.

14KLOC -> 21KLOC

That's to be expected, Go could be nicer if it was more concise, obviously that's a hard thing to balance, because you can end up with too obscure code, I just wish that there were some small improvements from time to time.

Absence of sum/discriminated-union types

Yeah, I refer to it as Algebraic Data Types, where you can for example have your tree type to be a Node with children or a Leaf with value. And a function can accept a Tree that can be one of these. IIRC, the Go people claim that you can do kind of something like this with interfaces, but it's not very nice to use interfaces in this way and has some drawbacks.

Catchable exceptions require silly contortions

I don't like exceptions, they disrupt the flow of the program - any function call could "return early", so you need to be always careful and account for that possibility. In C++ this was solved with RAAI, and it was a source of so many bugs that some companies just disallow exceptions internally.

Aesthetic doubt

Yeah, it's there, some things in Go just come out ugly.

Absence of iterators

Yes. Hopefully generics will enable implementing them.

autarch4• 5 years ago

You can kind of do that by passing a struct as an argument, and represent the optionalness by having a pointer in the struct. That's still awkward, you can't easily make a pointer to an integer or string literal, you need a separate helper function for each primitive type. Overall it could be nicer.

I think using a builder is a better option in Go if you have more than 2-3 arguments.

Yeah, I refer to it as Algebraic Data Types, where you can for example have your tree type to be a Node with children or a Leaf with value. And a function can accept a Tree that can be one of these. IIRC, the Go people claim that you can do kind of something like this with interfaces, but it's not very nice to use interfaces in this way and has some drawbacks.

Except you can't really do this with interfaces. If you need to get back to the concrete implementation of the interface, there's no way to know what all possible implementations might be. This is especially true since any type can implement an interface, not just the types you create initially. So someone else could implement `Tree` and you would not know to account for that in your function that accepts the `Tree` interface.

I don't think there's any substitute for proper sum types in Go.

kubb4• 5 years ago

I think using a builder is a better option in Go if you have more than 2-3 arguments.

But then you need to write all the setters, and something that was supposed to be simple, a function, becomes a whole contraption with a bunch of auxiliary code. Personally I would avoid that, and I definitely wouldn't make it a rule of thumb to use builders in every function with more than 2-3 arguments (maybe you meant something else).

kubb4• 5 years ago

Except you can't really do this with interfaces

Apparently they decided that it would be too confusing to have variant types alongside interfaces for some reason, and they claim that interfaces handle a lot of the use cases of variants:

https://go.dev/doc/faq#variant_types

preseinger19• 5 years ago

Builders aren't a good pattern in Go, because it's difficult to express continuations, and they leave types in incomplete states. It's almost always better to use config structs.

JulianMorrison• 5 years ago

FWIW, Go _does_ have keyword struct members, so it's possible to do

    type A struct {
      B int
      C int
    }

    myfunc(A{B: 1, C: 2})

throwaway89434523• 5 years ago

I do this a lot. You can even do `A{B: 1, C: 2}.myfunc()` in many cases.

azth2• 5 years ago

It's not surprising that the code base grew when reimplemented in golang. It would have probably been even shorter had it been rewritten in Python itself.

Just the other day, I was able to condense over 15 lines of golang code into 3 lines (could also have been 2 lines) in a Python-like syntax, both reducing code length, and substantially increasing readability as it would make the underlying logic clearly stand out instead of having several loop and map constructs.

throwaway89434523• 5 years ago

I think you have to be very careful before claiming readability gains from fewer lines of code. I've done a lot of Python programming and for simple things like mapping a single function over a list, a list comprehension or generator expression can indeed be a tiny bit more readable than a for loop, but combining that with some filtering or flattening and colleagues (and my future self) get frustrated. I've seen the same thing with overly fancy iterator chaining in Rust.

There's some temptation to think that terse==readable, but in practice this rarely extends beyond the simplest cases.

formerly_proven• 5 years ago

I didn't yet get around to writing Go in any significant capacity (it's not in an interesting spot for me) but I've had to read some and to me it seems like about half of all Go code does approximately nothing. It seems really weird to me that Go doesn't have a Result/Optional type (or sum types in general) and instead prefers multiple return values instead, and consequentially does not have error-handling operators but requires if err != nil repeated approximately 9000 times per file.

azth2• 5 years ago

I didn't claim that terse is always more readable. It was in the example I gave, and practically every golang program could be written in a more terse, more readable, and less error prone way in Python/Java/C#/etc.

nemo16185• 5 years ago

There's some temptation to think that terse==readable, but in practice this rarely extends beyond the simplest cases

A quick glance at any array language (APL, J, K, Q, etc.) confirms this. ;)

nauticacom12• 5 years ago

One thing that the people saying "just use a struct for keyword arguments" are missing is that structs should signal intent, i.e. "this is a concrete concept in the system." A Repository, Commit, Message, Person, etc. struct are all concepts in the domain, whereas "the arguments for this particular function" is not. I think Go people are allergic to writing code that does anything other than functionally work.

scndrycntct• 5 years ago

I think Go people are allergic to writing code that does anything other than functionally work.

I think that's what Go was designed for, as a language. To be readable, usable. It's uncaring for your personal programming philosophies. I think that's why it's been successful.

danudey4• 5 years ago

Specifically, Go was designed to be effectively and safely writeable and readable by mediocre programmers.

nauticacom12• 5 years ago

I'd agree that it was one of their design goals, but I wouldn't go as far as to say that it makes the language "readable and usable." That depends on what your values are.

Go's philosophy (which clearly flows from its creators being C-enthusiasts) is that the only thing that matters for reading, writing, and understanding a program is what it concretely does, i.e. what structures are created, where values are stored, how computations are performed etc. If that's also your philosophy, then of course it's going to jive with you.

But plenty of people also have different philosophies. Maybe you think the main thing that's important in crafting programs is developing a rich domain vocabulary that expresses concepts and how they interact. Maybe you think that what's important is formal proof of both logical and concrete correctness. In those cases, Go's rigorous opposition to abstraction (coming from its philosophy that what's important is concrete operations) will probably irritate and slow you down.

I couldn't say exactly why it got popular. I'd guess that some significant segment of programmers also share its philosophy, but I have no evidence to back that up. Certainly any reasonably uncontroversial language with a large suite of libraries backed by Google is bound to have some level of popularity.

SquishyPanda23• 5 years ago

To be readable, usable.

Go doesn't particularly value readability or usability. For example, the short variable name convention makes it harder to read code you're unfamiliar with, and there are a number of noticeable usability shortcomings, some of which are mentioned in the article.

I think Go's design goals were really to (1) reduce compilation time, which explains why Go has human programmers do work that compilers do in other languages, (2) be statically typed and compiled, so you can use it conveniently for microservices, and (3) have syntax somewhat similar to Python.

I think of Go as the successor to Java. Go is to Python as Java was to C++. That, plus the integration with many libraries is why it's taken off in some niches.

LandR2• 5 years ago

I don't find go readable in the slightest, the syntactic bureaucracy is just too high.

There is a forest full of code hiding a trees worth of business logic, always.

Go is one of the least expressive languages I've ever used.

CGamesPlay• 5 years ago

Well, "Command to Create a Repository" could easily be a concept which contains all of the options necessary to create a Repository, which is a different concept.

Now, there is a problem that all of the struct fields have to have sane zero values for this to work, but I think that is a problem which would naturally arise with keyword arguments as well.

nauticacom12• 5 years ago

Sure, if you're pushing things through a command bus or something. Otherwise you're just stretching it. If you wouldn't use it in a sentence to describe a use-case, I'd say that it's probably not a concept.

Would it? Only if you allow default values, which is a separate discussion. And even then only if your only mechanism for default values doesn't let the definitions specify what, exactly, the default value is.

jasonhansel• 5 years ago

IMHO, if the arguments to a function are just a bunch of heterogeneous values that cannot be sensibly grouped together, then that's usually a sign that the function needs to be refactored (e.g. because it's trying to do a bunch of unrelated things, or because it couples tasks together in an unnatural way).

preseinger19• 5 years ago

structs should signal intent

Isn't a collection of parameters to a function an intent?

nauticacom12• 5 years ago

Not really? I suppose what I meant by that was that, if you make a struct, it should represent a concept that makes sense outside of the context of passing it to one function in particular; you're signalling that this collection of data represents a concept in your program

throwaway89434523• 5 years ago

Not everyone is a Kingdom of Nouns purist. Even most OOP proponents that I've spoken with reject Kingdom of Nouns because it's pretty indefensible (in the worst case it leads to banana-gorilla-jungle problems[0] and in the best case it imposes arbitrary and unnatural restrictions on program design). Note also that "modern Java" and "modern C#" and "modern C++" and "modern Python" virtually all reject these kinds of designs and end up looking a lot more like Go with respect to their use of structs/classes.

[0] https://www.johndcook.com/blog/2011/07/19/you-wanted-banana/

nauticacom12• 5 years ago

Even if you're not doing OOP you're modeling your domain in one way or another. Whether those models are explicit and expressive or not, and how you define those qualities, is another question. In neither case are the arguments to a particular function typically a concept in your domain

danudey4• 5 years ago

Reposurgeon was a pretty enticing option for us, as we're currently in the migration of SVN to Git, and the string of tools we have is kind of complicated to explain to newcomers.

Currently the workflow is svn-all-fast-export (SVN to Git) -> git filter-repo (trim content from the repository) -> Git LFS (store large files out-of-band), where each stage has to be manually tested and written, and then hopefully put into a shell script or Bash history or something.

Unfortunately, I have yet to get Reposurgeon working for us at all. The documentation was inaccurate, in that it referred to things which only existed in the Go version, and the Go version went OOM (on a server with 384 GB of RAM) on every repository I've tried it on. This includes just reading from an existing, pruned svndump file, so it's not resource contention either.

Basically, the tool is great conceptually, but it absolutely does not work for our use case. We had to stick with our existing, messy, multi-component solution, which has worked surprisingly well over the last two years, and since I'm the one doing most of the processing, having kind of a messy system is relatively acceptable for the time being.

db48x9• 5 years ago

We’d be interested in hearing more about your situation, so that we can further improve Reposurgeon. If you can profile it as it reads your svn dump (using the `profile start` and `profile save` commands), it could really help us out.

How many commits does your repository have? The largest one we’ve successfully converted from SVN to Git ourselves was 287k commits.

danudey4• 5 years ago

Sorry for the late reply, but:

1. I'll try doing some profiling and see what happens

2. The latest repository I tried to import has 671,190 revisions

nemo16185• 5 years ago

   What I really missed was generic map-function-over-slice, which could be
   handled by adding a much narrower feature.
   
   If one graded possible Go point extensions by a figure of merit in which the
   numerator is "how much Python expressiveness this keeps" and the
   denominator is "how simple and self-contained the Go feature would be",
   I think this one would be top of list.
   
   So: map as a functional builtin takes two arguments, one x = []T and a
   second f = func(T)T. The expression map(x, f) yields a new slice in
   which for each element of x, f(x) is appended.

I think this is an excellent way of evaluating new features, and it represents a real missed opportunity for Go to explore new PL territory. Instead, we're getting full-blown user-defined generics, which increases the "denominator" far, far more than it increases the "numerator."

throwaway89434523• 5 years ago

I also came from Python (15 years of experience) to Go and I think newcomers to Go index too hard on terseness. The for-loop equivalent for a map over a list is more characters, but it's really straightforward and easily recognizable in the code. In the general case, I'm glad that Go doesn't try to explore new PL territory, optimizing instead for things that are known to improve developer productivity. None of this is to say that Go has no room for improvement--only that "terseness" is not high on the list of improvements I'd like to see (for example, I'd rather have sum types with exhaustive pattern matching).

nemo16185• 5 years ago

For me, it's not a question of terseness -- it's more about communicating intent, and not polluting the scope with incidental variables. Everyone knows what map/filter/reduce do. When reading new code, seeing "map" is better than seeing a for loop: you don't have to think about the underlying iteration at all, you can skip directly to the essence of the transformation. As a side effect of this, when you do see a for loop, you can safely assume that something about the iteration is non-trivial: maybe the loop exits early, for example. I wrote a bit more about this here: https://twitter.com/lukechampine/status/1463715093733122054

btw, by "new PL territory" I mean "reifying a small set of container operations, without supporting user-defined generics," which to my knowledge is not a position taken by any mainstream language.

pcwalton• 5 years ago

ESR's proposal here is basically just another way of spelling Python list comprehensions or Perl 5 map (which is likewise a builtin, with special parsing rules). It's not "new PL territory" by any means.

throwaway89434523• 5 years ago

I think the "new PL territory" was referring to a type-safe generic map builtin. Obviously Python and Perl don't have a concept of type safety. That said, I still wouldn't call that novel in the sense that a builtin version of a well-understood generic function doesn't seem like a feat of ingenuity.

preseinger19• 5 years ago

a real missed opportunity for Go to explore new PL territory

But Go was not intended to explore new PL territory. Is that not OK?

nemo16185• 5 years ago

I would argue that it is: it's explicitly a language designed "in the service of software engineering," which is something we're still figuring out how to do. It has implemented novel features that serve this end, and left out features that don't. PL design doesn't have to just be about dependent types and borrow checkers, after all.

Notes on the Go translation of Reposurgeon (2020)

Related Stories