Einstein Analytics and Go

https://stackoverflow.blog/2019/10/07/how-salesforce-converted-einstein-analytics-to-go/

by bsg75 • 7 years ago

55 83 7 years ago

moksly4• 7 years ago

GO is always listed as popular, but if I search my entire country for GO jobs there isn’t a single one. Well, there is one single listing, but that’s for a C/C++ job at Google, and it’s listed under “Nice to know” along with Rust and a few others.

Are we behind the curve?

weberc26• 7 years ago

What is your country, out of curiosity?

moksly4• 7 years ago

Denmark.

hderms• 7 years ago

Go is certainly somewhat popular in the US, but imo the language isn't well suited for all tasks.

marcrosoft• 7 years ago

It is well suited for most tasks though.

cwbrandsma• 7 years ago

I would argue that no language is suited for all tasks. Some can handle more tasks than others tho.

wil421• 7 years ago

Just did the same thing for my US city. There were a total of 5 job postings from scummy head hunting companies.

Try to search Golang.

peterwwillis• 7 years ago

It's a trendy language. It will increase in popularity (e.g. developers want to learn it because it's new) until it eventually wanes. Remember when everybody wanted to write Ruby?

This page shows job postings per programming language from searches on Indeed.com: https://www.codeplatoon.org/the-best-paying-and-most-in-dema... Go isn't on the list. The TIOBE index has Go at #17: https://www.tiobe.com/tiobe-index/

moksly4• 7 years ago

I remember Ruby, and it’s a similar story really. There is one Ruby shop in my region.

asdfman123• 7 years ago

People really need to break down salary analysis by region.

If a skill commands a higher salary, does that mean I can actually get paid for it more in Houston, or does it mean that it's only used in tech hubs which pay way more to keep up with cost of living?

fjp• 7 years ago

Remember when everybody wanted to write Ruby?

There is an absolutely mind-boggling amount of Ruby jobs in the US.

davzie• 7 years ago

In the UK Go jobs are so prevalent it's insane. But then again we have a lot of fintech, AI and blockchain startups that are knocking about so that could be a contributing factor.

TrackerFF• 7 years ago

Norwegian here. Searched for "Golang", which yielded a whooping 4 results. "Go" gave me 8, but of those only two were the main language, the rest lumped it in the nice to know category.

Python on the other hand yielded almost 200.

robbrit4• 7 years ago

Having Go experience on my resume, I do get a lot of recruiters reaching out to me for Go jobs. It's quite popular for infrastructure work.

This is in the US, not sure if it's the same in other countries.

kjeetgill• 7 years ago

Most companies will hire targeting the language that the vast majority of the company is using: often Java or JavaScript depending on the role.

They will also have often have projects in "support languages" for internal tools and services. Often in python atleast, and more and more in Go. Those are often unlisted.

Edit: Just noticed you said in your country. I suspect this is true broadly but what do I know?

rb808• 7 years ago

Yeah I'd love to work in Go, but there are so few jobs it either isn't worth it or makes it risky to specialize in it.

jhoechtl• 7 years ago

What is your job search engine?

trimbo• 7 years ago

Java was not considered?

blub4• 7 years ago

One one hand one would be tempted to dismiss Java as uncool, but on the other hand we're talking about Salesforce, so this is a very legitimate question.

The truth is probably rather ordinary: even dull corporations want to look cool.

elisharobinson7• 7 years ago

what is this strange obsession with type , it improves performance yeah sure . but to go around claiming that it is a panacea for all things poorly planned is pure unadulterated stupidity. i can concede that some immutable data structures are needed , but types not for me .

Python doesnt handle threads well this can be true. also if you are straining python threading system you ought to take a look at your own abstractions and design. chances are you wont go very far before you hit the same bottle neck in a different language if you assumed python gave up at X you could achive 100X with go but you will almost certainly hit the same issue when your data grows 100X.( i am aware of computational complexity is a factor. but the author claims that part is handled in C , from what is described it looks like they replaced flask with go)

wyldfire• 7 years ago

what is this strange obsession with type , it improves performance yeah sure

Static typing allows for a lot of static decisions which in general brings better performance for a language. CPython pushes so many decisions to runtime that it is a very significant performance impact.

But the article cites more significant challenges related to type. Not performance, but design:

"First, Python uses loose typing, which was great for a small team rapidly developing new ideas and putting them into production –but less great for an enterprise-scale application that some customers were paying millions of dollars for," he writes.

Cue the reminder that in fact Python is strongly (but dynamically) typed. But LeStum's point stands: dynamic type hurts developers trying to read/write unfamiliar python code. The mypy static typing should help out a lot but I don't think it's very popular yet.

For performance, IMO in this order you should consider (1) PyPy, (2) multiprocessing, (3) cython and/or c-extensions. (and I suppose implicitly (0) analyze your algorithm, exploit numpy where possible). If you exhaust those, Go seems like a great alternative.

elisharobinson7• 7 years ago

i agree . i might clarify my original post, im half asleep. but i agree with the correction you pointed out in my original post.

fiedzia2• 7 years ago

For performance, IMO in this order you should consider (1) PyPy, (2) multiprocessing, (3) cython and/or c-extensions

They are all very limited in what they can do (neither can replace others) and bring tons of complications. Modern languages can give you all of that without big issues.

nightski2• 7 years ago

A good type system is like a set of unit tests that are written for free and checked automatically every time the software is built. The better type systems feel natural and do not create extra work.

Actually, having worked with some great type systems in my life (especially in the ML family of languages) I feel they can assist with design and thinking through a problem by explicitly stating your intentions without being tedious. Obviously type systems like C++/Java do not fit this bill.

But lastly, they are even more useful in my experience when dealing with other people's code. Types can be self documenting to an extent and provide certain guarantees (or lack of guarantees) about the code itself. Of course how much information the type system conveys varies wildly based on the power of the type system (Haskell vs. C++) and the desire of the original developer to communicate intent through types.

elisharobinson7• 7 years ago

the problem i have with premature type or type obsession is , you commit to a data model without even completely understanding the business model / rules there is this myth that you understand the requirements clearly and that it wont change quickly or by a lot . Python gives me the flexibility to whip up a service and check if fits if it doesnt fit throw it and try again, i wouldnt need to start from scratch , i can do well be removing dead or unwanted code and some basic cleanup to have a good start point. I cant comment on ML family cause i have not used it in prod env.

jumpingmice2• 7 years ago

Types improve productivity in large legacy projects. When I have to review a change in python I have to check absolutely everything from scratch. I have to look at the signatures of the called functions to see if the call sites are correct, where a typed and compiled language would simply fail to build. And I have to carefully check every block to see if attributes of none-typed variables are being accessed.

Python is a bit like a high-interest credit cards for coders. You can use it to make it look like you just whipped out a huge project from scratch. But you will spend the following decade paying down the tech debt.

elisharobinson7• 7 years ago

python has a awesome type system , even though it has support for dynamic typing. you cant use dict, int, list interchangeably given enough time you notice operations and methods which can only work for a given type. but when the types share similarity like they are iterate-able they work predictably.

although its not for everyone , but if your talking about large projects you could use metaclasses to enforce type checking during instantiation.

tech debt can sometimes be unavoidable. i often find the case to be that a project stops progressing due to fear of breaking something and not knowing how to fix it. In python this fear is reduced almost by an order of magnitude compared to other programming languages.

also in typed languages used in large projects. the project itself behaves like a standard or a protocol , people memorize the values of bytes each request contains , people start maintaining informal codes to access the project or it becomes some hob goblin of word documents , pdf and jira .

and god forbid you need to make a change in the datamodel to a more optimized model, you would most likely have to throw away most of your code which you wont do because solving all the build issues and waiting for it to compile from start doesn't go on the clock as time spent coding something useful.

cle• 7 years ago

but to go around claiming that it is a panacea for all things poorly planned is pure unadulterated stupidity.

The article never claimed this:

"First, Python uses loose typing, which was great for a small team rapidly developing new ideas and putting them into production –but less great for an enterprise-scale application that some customers were paying millions of dollars for," he writes.

There are tradeoffs with both static and dynamic typing. Know the tradeoffs, know your business, then make the right decision for your use cases.

elisharobinson7• 7 years ago

i assumed that the implication was type checking the project was able to help them scale the team. which makes absolutely no sense whatsoever.

derivagral2• 7 years ago

I found an interesting subtlety to the message here. They had a Python wrapper around a C library, and effectively never updated the library. I wonder if this was due to engineering decisions, hiring/expertise of their staff (or future hires), or something else (risk? etc). I imagine it is easier these days to find Go devs than C devs who want to work at a company like this?

Also, for a sense of scale, this rewrite took about 4 years to complete from first concepts.

sho• 7 years ago

For 9 out of 10 articles like this - "We rewrote our old code in tired old language FOO with hot new language BAR and it's SO MUCH BETTER!" - I suspect they could have simply rewritten with the same language, engineered it better and applied all the lessons learned, and achieved basically the same result, probably faster. But that doesn't scratch the "new hotness" itch, does it?

That said, considering the requirements this thing sounds like it had to meet, perhaps golang was the right choice after all. Unlike certain other projects I know about where they HAD to write everything in golang/microservices/k8s/etc because it HAD TO SCALE, took 18 months instead of the 3 months it would have taken with rails but credit where credit is due - those 2 or 3 requests a minute (peak) are handled very, very quickly.

Barrin922• 7 years ago

That said, considering the requirements this thing sounds like it had to meet, perhaps golang was the right choice after all.

Yes, in this particular case I don't think 'the new hotness syndrome' applies. The performance limits of python given that it's dynamic and the multithreading problems are fundamental to the language.

sidlls• 7 years ago

That may be true in many cases, but this article really resonated with me. I spent more years than I want to admit working on a large machine learning and analytics platform that was originally built with Python. It was, frankly, a nightmare. The dependency problems, the performance issues, the problems with typing and use of data structures were all huge headaches to manage. I have a blog in the engineering organization I work in where I'm basically publishing a series of articles that reflect a similar path the Salesforce article describes, for basically the same reasons. (My group is using Java, not Go, though.)

fiedzia2• 7 years ago

I suspect they could have simply rewritten with the same language, engineered it better and applied all the lessons learned

No, you can't do that. There are many benefits that come from underlying technology choices that you can't backport to old stack, and if you try, you'd create a monster nobody would like to touch. There might be fewer developers willing to work with Elm than something else, there are none willing to work with "we rewrote javascript standard libraries so that it kind of resembles Elm, but poorly".

bsg75• 7 years ago

It is also easier for Python devs to pick up Go and become productive than with C. I suspect SF had more people that could move quickly from Python to Go than those who could add features to the C lib.

This is from personal experience where Go is used when performance is needed, and the end result is a single program, vs a C lib and a Python wrapper, gaining some level of simplicity along with the needed performance.

We don't get quite the performance of C, but we also don't need to include memory management. Sort of a happy medium between Python and C.

robbrit4• 7 years ago

it is easier these days to find Go devs than C devs

Not really actually, there are still loads of C devs (or C++ at least) and a lot more jobs in C and C++ than for Go.

Having made engineering decisions between C++ and Go, my key reasons for picking Go over C++ are:

* Simplicity when multi-threading

* It's much easier for someone who knows neither language to become productive in a professional environment in Go than in C/C++

* Fast compile times - no more typing "make" and then going off to get a coffee

* Lots of modern niceties that are more fragmented in the C world: third-party vendoring, unit testing, style guides, etc.

blub4• 7 years ago

Most C++ devs won't want to touch C and being the performance person in a Python project sounds like a sideshow job.

Mixing C and C++ like you do makes little sense, they're very different.

scythe2• 7 years ago

So I worked on this project. As with many stories this one has been simplified to be less boring and emphasize key points. I generally agree with Guillaume’s characterization—he’s a brilliant engineer and it was great to work with him.

The most important advantage of Go was that it is statically typed and there was a broad agreement that we should use a typed language to prevent errors. The other advantage was that Go is faster than Python, allowing us to move logic out of the C codebase and unify the high-level operations in a single codebase. The other other advantage of Go was that it is not C++, for which we were all grateful. As with most real projects, the rewrite occurred concurrently with adding features and tightening coding standards — `go fmt` was a big motivator and the Go codebase had much more comprehensive test coverage than the Python one.

Also, for a sense of scale, this rewrite took about 4 years to complete from first concepts.

That’s not quite accurate. In fact the Python production code was largely ported by mid-2015, and the Go library was feature-complete by 2016, albeit still linking to C. However, there was a lot of non-production prototype code, implementing more complex optimizations and written in a few additional languages, which took another two years to port, and also to be made production-ready. Perhaps what’s important is that we were able to keep adding new improvements while still “porting” this code. (Real life is a poor laboratory.)

effectively never updated the library.

It is, or was, still maintained. Most of the very low level file operations had been optimized to death already. Higher-level functionality was slowly moved out of the C library. I left Salesforce in 2017 so I’m not sure what happened next. (Incidentally I left not only Salesforce but tech entirely - I’m now at grad school for medical physics.)

I imagine it is easier these days to find Go devs than C devs

Maybe? None of us knew Go when the porting began. I think if you know at least 2 programming languages and one of those is C-like, Go will be a piece of cake. I was hired with no Go experience and ramped up quickly.

derivagral2• 7 years ago

Thanks for the detailed reply! Getting a real perspective instead of an article is always great.

jbergens• 7 years ago

With the current upwards trend for python I wonder how many companies will come to the same conclusion the next 5 years?

Maybe ending up rewriting large parts of their system. If not to Golang then to some other fast and statically typed language like c# or java.

thanatropism• 7 years ago

I'm reminded of the trend of starting systems in Lisp (I believe this included reddit) and then rewriting in Python.

Maybe the Blub paradox has been hitting its limits. The point of using leftfield expressive languages was that developer time is orders of magnitude more expensive than machine chugging time. But maybe web-scale-big-data-etc. needs enough (latency) juice that the scales tip and the comparative advantage is clearly on the side of adopting "industrial-strength" enterprisey technologies again.

robbrit4• 7 years ago

They touched on this in the article, but I'll go into more detail here since it's been fashionable for years to bash on "Blub" languages like Java or Go. I myself was guilty of this for a long time until I started using these languages in settings where they shine, and developed an appreciation for them.

The argument is nothing to do with machine chugging time, and is entirely towards developer time. The problem with expressive languages like Lisp, Ruby, Python, etc. is that the language ends up varying from person to person - the more expressive the language, the more variance there is. This is a feature when you're a small team because the abstractions you build let you move quickly, but it is a bug when you're a large team maintaining a piece of software over years, where developers have come and gone. The ramp-up time to learn and understand the various abstractions that people have built over the years ends up accumulating and cancelling out the gains that those abstractions gave earlier on.

Blub languages on the other hand tend to be more uniform, so it's easier for someone who isn't very familiar with the code to dive in and understand what is going on.

blub4• 7 years ago

Java is no longer uniform, especially in the latest versions they're adding more and more features. And there's also Kotlin and project Lombok for people that want even more excitement.

And yet Java somehow manages to be both a boring language and still have too many things to lean. This achievement is probably not appreciated enough by those that criticize it.

crimsonalucard• 7 years ago

Analytics needs unstructured data in order to function well as a modular service. Golangs lack of coproducts makes such data much harder to deal with.

As much as I hate js. Nodejs is actually better for one of these servers. Then again it's mostly the database that determines the schemas.

erokar2• 7 years ago

Of all new programming languages that have received a certain amount of hype, Go is the only one I never felt inclined to play with after reading about it. It just seems too dumbed-down and backwards. It's like someone wished CS stopped evolving after 1970 and designed a language to pretend it did. The simplest of abstractions are removed from the language, like operations on collections that abstracts away for loops (e.g. filter, map, reduce, etc.).

I don't doubt that it has good CPU performance and concurrency, the developer experience just seems so frustrating.

eitland3• 7 years ago

I belive it when other people say they like it, but I agree with you.

(That said I really didn't like Java either, and it later became my favorite language for a decade or so.)

winter_blue2• 7 years ago

Why did Java later become your favorite language? And why only for a decade?

(I started liking Java a lot more after Java 8 came out.)

ben_jones• 7 years ago

It's good at moving bytes around. Turns out most networked applications are just moving bytes around. I've been using it for four years now and honestly would ask for an extra 10%-20% to be working in most other languages.

tschellenbach• 7 years ago

I like that it's more productive to work with than C++ and about 20-40 times faster than Python (more if you take the concurrency improvements into account). That's a perfect sweet spot for many companies.

weberc26• 7 years ago

I like that the tools just work. I don't have to learn a DSL just to write my own build system for my project (unlike CMake, gradle, opam, etc); it's just `go build` or `go test`. And it works for almost everyone's project, so anyone can contribute to anyone else's project without learning a crappy one-off build system just to compile the code or run the tests. Similarly, I don't have to learn a special comment syntacs for docs nor write CI jobs to publish code or documentation packages--docs are just comments above functions and code and docs are automatically published when you push to version control. Also, `gofmt` is ubiquitous, so everyone has the same style.

Go was built for software development, and it addresses real, practical problems.

K0SM0S2• 7 years ago

About "dumbed down" and "backwards", which can also be expressed positively to mean the same (e.g. "essential" or "synthetic"), it's because of Go's problem space: to tackle complexity at scale, i.e. solve relatively "simple" problems (or with more atomicity in architecture) but maximize both maintainability and concurrent performance.

Key words being to design and maintain concurrence as compared to C++ or Java. Go is everything but "backwards" in that regard, it's fantastically efficient and actually fun to code. It builds upon SCP, a formal language designed in 1978 for concurrency, and it's Pike 3rd time (the charm of v3!) in that regard. They aced it, seriously, and thus scaling is part of the core paradigm (if you want to build for that).

The trade-off is in elementary complexity, core and syntax sugar ("absence" of generics etc). But in actual practice, people are effectively hard-pressed to find examples where Go applies (we're talking server / CLI / infra software, backend and middleware mostly) and e.g. the absence of generics is an issue. It turns out, in actual practice, you can very pragmatically approach the problem from the Go programming paradigm standpoint and solve it quite elegantly (if a little verbosely sometimes, but it's nowhere near C++ or Java either once you factor in everything else).

The "absence" of a feature in a nonetheless 'complete' programming language is only a lack if you want to think this way and use said abstraction; the makers of Go argue that in a number of cases, it's not just possible but better to keep it simpler at the unit level, because you massively improve readability, and simplicity makes performance and security also simpler.

Keep in mind, when they made Go, they never thought it would become so big, nor was it the intent. They made Go for Google to solve Google-y problems of scaling. It just turns out, as Docker illustrated, that many other companies and problems could benefit from Go. Hence, it has its place. It'll probably never be "THE" programming language like Python is, but it doesn't try either. I find there's elegance in that restraint, there's a certain nobility to stay true to the UNIX philosophy of solving your problem space, no more no less, and doing it as efficienctly as possible.

As for "frustrating", one of the language's explicit goals is to "make programming fun again" — back to sitting in a Google office to solve tech-giant scaling problems without going insane thanks to C++. Maybe they failed for you : ) But for many people, beginners notably, the language is really fun — if you forget everything else and strive for "idiomatic" Go, in syntax and patterns and architecture. The tooling is young but great — things like integrated testing, benchmarking, documenting, fast compiler.

Sorry for a long piece. This low-profile language is one jewel if you don't expect it to be Python or Js. I hear the same enthusiasm from the community (let's call it: love) as I used to hear from Ruby lovers for instance — the "match" is real between the Go philosophy and some people.

erokar2• 7 years ago

Thanks for a thorough answer. You do make a convincing argument for Go's simplicity.

stunt• 7 years ago

I also see all the benefits but GO is very regressive and feels like a backward step.

But when you consider how many developers have a hard time to learn GIT after so many years, you realize why companies need something as simple as GO.

Another reason is rise of new specialized engineers that programming isn’t their main focus and hence they are looking for easy solutions and easy languages to pick up. (DevOps, Data Scientists, SRE etc)

scythe2• 7 years ago

Previous discussions here:

https://news.ycombinator.com/item?id=21203651

I worked on this project; my comment is here:

https://news.ycombinator.com/item?id=21205504

weberc26• 7 years ago

After these ports, our team has built up some expertise with Go and its compiler quirks. But you can still get burned. For example, you can very easily write data that you want to place on the cheaper stack instead onto the much more expensive heap. You won’t even know this is happening by reading over your code. That’s why, as with any new language that you require high performance from, you need to monitor processes closely and create benchmarks around CPU and memory use. And then share what you learn with the community so that this knowledge becomes less tribal.

While Go doesn't have formal semantics that let you specify whether a value is allocated on the heap or stack, it seems like it would be easy enough to create your own. The compiler has flags that cause it to output the sites where it heap-allocates--if you add comments above a function or an allocation site such as `// noalloc`, then you could write a "linter" that compares the allocation sites against those comments and errors if one of those noalloc sites allocates.

In lieu of allocation semantics, this seems like a better approach than writing performance tests for each of these sites.

twotwotwo2• 7 years ago

I see why you'd do perf tests as well (you might fail to annotate everything that could make your function run slowly), but a static analyzer that looks at -gcflags -m output does sound like a super cool idea--direct way to check if your intuitions about the code and its compilation are right.

weberc26• 7 years ago

Yeah, I envision fewer performance tests at a higher level of granularity. Not only might you miss annotations here and there, but allocations aren't the only thing (although they are probably the most significant) that would affect your performance and my proposal won't cover you there.

hinkley• 7 years ago

There are at least two ways to fail. You can get something wrong from the start, or you can get it right and fail to keep it that way.

Perf sensitive code can often be that way. Some innocent coworker comes in to add a feature and they don't get why two idioms are not equivalent from the compiler's perspective, and so they change things.

Meanwhile, you're busy doing something else. The code is still correct from a testing standpoint. It still satisfies your code standards. But it no longer meets your response time expectations. It's a lot of work to maintain the sort of toolchain that lets you reliably spot this sort of thing at build time or even in pre-prod environments, and it's difficult to identify places where those tools are failing to detect problems until there's been a problem.

weberc26• 7 years ago

I agree with all of this. My proposal addresses exactly this problem for the allocation related bottlenecks. CI will fail if someone introduces an allocation in a noalloc—annotated block.

SomaticPirate• 7 years ago

I’m curious to what tribal knowledge they are referring to? What are these incantations to make my Go code faster? Does this mean just stripping debug symbols or are there flags I should know about?

weberc26• 7 years ago

They were talking about writing code that doesn't allocate in the hot path. Mostly this is pretty easy to eye-ball, but you can make sure it doesn't allocate by passing `-gcflags -m` to the `go build` command. This is almost always the initial bottleneck in naive Go code.

twotwotwo2• 7 years ago

The post dates the rewrite effort as happening in 2016-7. In the past couple years escape analysis has gotten a lot of effort (https://github.com/golang/go/issues/23109) and there's also now mid-stack inlining (https://github.com/golang/go/issues/19348) which obviously can be useful in itself but also help escape analysis and other optimizations work.

Does anyone know if there's any big picture work on GC throughput in progress? The idea of the ROC (fast freeing of memory never shared between threads on goroutine exit) was canned since it regressed on some workloads and I'm having trouble finding signs Austin's idea with hashing memory landed either. A moving GC or a read barrier seem mostly out because they can hurt other things (C interop, performance of code that doesn't allocate much). Though they seem to have addressed pauses quite well, and lots of code out there isn't GC-heavy, it still seems like a key area where it might be possible to give a "free" throughput win to existing code.

SnarkAsh• 7 years ago

Why is this on the Stack Overflow blog?

When I replied to their survey that I wanted "tech articles written by other developers", I was imagining a platform for Stack Overflow authors to contribute longer-form work -- an idea that's been floated by staff for most of the life of the site! I wasn't expecting random cross-promotional content.

thumbsdownnow• 7 years ago

Simplicity is king.

lugg• 7 years ago

First, Python uses loose typing, which was great for a small team rapidly developing new ideas and putting them into production—but less great for an enterprise-scale application that some customers were paying millions of dollars for.

Code review is supposed to catch people trying to mix return types, mess with or overload parameters in unexpected ways.

Static typing removes this need but adds a million more.

If your people aren't catching this stuff in review, they're probably not catching other stuff, that is, static typing isn't going to save you like you think it is.

Einstein Analytics and Go

Related Stories