Why Swift for TensorFlow?

https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md

by nnd • 7 years ago

212 140 7 years ago

I can imagine swift really taking off in this space. It’s going to be a battle between Julia and Swift for who does the best automatic differentiation.

FridgeSeal5• 7 years ago

I’m happy for Swift, but I really, really want Julia to win out here.

There’s some pretty impressive ML frameworks in Julia and the language can do some really cool things, so I’m hoping that gives it the edge.

Plus, I found tensorflow exceedingly painful to use, so hopefully something else prevails.

pjmlp10• 7 years ago

Google would need to make Swift a first class citzen on Windows, currently Julia is winning.

skohan14• 7 years ago

That's one of the outcomes I am hoping for in this. I would love for Swift to be a first-class language.

I believe the Swift for TensorFlow team is currently hiring for this.

make32• 7 years ago

how can you not mention Python when it's currently what is used 99% of the time, the other 1% being R

elpakal• 7 years ago

I also wonder how much of this coincidentally lines up with Chris Lattner landing at Google. As Chris will admit, and as was left out of this analysis, Swift has also been given the humble goal of achieving world domination. All joking aside I'm very thrilled about this and have enjoyed tremendously watching the Swift language mature since its launch due in large part to the open source community and the Swift team's admirable commitment. Onward!

tempdeadbeef• 7 years ago

http://nondot.org/sabre/Resume.html

"Swift for TensorFlow rethinks machine learning development ... I imagined, advocated for, coded the initial prototype and many of the subsystems after that; recruited, hired and trained an exceptional engineering team; we drove it to an open source launch and are continuing to build out and iterate on infrastructure."

DannyBee2• 7 years ago

What do you mean?

This was not in process before Chris came, it was a project he suggested and started pushing on?

What is there to be coincidental?

insulanus• 7 years ago

I think they mean the exact opposite of "coincidental", in the sense of "hmm... is this a coincidence" (no, it isn't)

gok• 7 years ago

It's not a coincidence at all

sjwright• 7 years ago

I have no idea what TensorFlow is (other than the basics) but I enjoyed reading that entire document because it did such a wonderful job of explaining a complex and potentially contentious decision. It’s fascinating to see Swift feature so strongly in a pragmatic analysis that doesn’t explicitly favour Apple platform interop.

skwb3• 7 years ago

I am a bit ignorant on the topic, but is swift available for Windows/Ubuntu? Most of the deep learning scientists I know and work with use either of the two setups. I know there technically exists CUDA GPU support for Apple, but I have frankly never even attempted to mess with it.

rudedogg3• 7 years ago

Ubuntu is supported (see https://swift.org/download/), but Windows is pretty early I think.

See https://github.com/apple/swift/blob/master/docs/Windows.md and https://forums.swift.org/t/windows-nightlies/19174 for more info.

m0zg4• 7 years ago

Is there a "standard" way of running Swift on Ubuntu LTS nowadays? A while back I looked into it, and ran into some hokey and unsatisfying solutions. I used Swift on iOS, and I like it a lot, but if they care about adoption, someone needs to reduce friction of getting up and running to approximately zero. A snap package a-la Go or per-user script based installation a-la Rust would be quite OK, as long as it's just one, easy to discover command.

rudedogg3• 7 years ago

It looks pretty straightforward, see the Linux section of https://swift.org/download/#using-downloads

m0zg4• 7 years ago

Yeah, going through two screenfuls of text every time I want to upgrade is not "straightforward".

igotsideas4• 7 years ago

Well written explanation! I really enjoy Swift but it's not as accessible as some of the other languages mentioned. I have a 2011 Macbook Pro and wanted to use the latest and greatest new Swift features. Unfortunately, my machine is too old to upgrade to Mojave which means I can't download the latest version of xcode, which means no new version of Swift. I'm not mad at Apple in the least bit. I just wish I could use Swift 4 on my machine.

gthippo2• 7 years ago

You might be interested in checking out Swift on Google Colab (e.g., https://colab.research.google.com/github/tensorflow/swift-tu...)

coldtea• 7 years ago

Isn't Swift open source and available to build anyway, regardless of Xcode and OS X version? Even on Linux etc?

delinka• 7 years ago

Have you considered trying to build Swift from source? It's a bit time-consuming the first time, but subsequent updates less so - and you'd have Swift 4 at your disposal.

stephencanon2• 7 years ago

FWIW, you can install Xcode 10.1 on High Sierra, and use the Swift.org 5.0 toolchain (https://swift.org/download/#snapshots) or build swift from source. You can't ship App Store apps this way, but it works great for experiments.

igotsideas4• 7 years ago

My machine is too old to upgrade to High Sierra.

yoz-y• 7 years ago

If you don't mind stranding from the 100% stable roads, you can install Mojave on your macbook using this patcher: http://dosdude1.com/mojave/

Personally I am running Mojave on a late 2009 macbook pro and it still works amazingly well. Transition from Mojave and especially the new XCode are also way faster than previous iterations. There are caveats though, as the processor in my computer is too old, I had to hack homebrew to compile everything from source.

(Also, using the patcher does not hinder my ability to push updates to the App Store or use iMessage, if that is a concern)

igotsideas4• 7 years ago

I'm gonna try this, thank you!

gthippo2• 7 years ago

Swift is also now supported in Colab (Google-hosted Jupyter notebooks) and there's a nice tutorial on some of the features of Swift for Tensorflow at https://colab.research.google.com/github/tensorflow/swift-tu...

aaaaaaaaaaab• 7 years ago

Why not Rust?

Edit: I wonder if Swift could be replaced with Rust for iOS development?

criddell• 7 years ago

From the article:

We believe that Rust supports all the ingredients necessary to implement the techniques in this paper: it has a strong static side, and its traits system supports zero-cost abstractions which can be provably eliminated by the compiler. It has a great pointer aliasing model, a suitable mid-level IR, a vibrant and engaging community, and a great open language evolution process.

A concern with using Rust is that a strong goal of this project is to appeal to the entire TensorFlow community, which is currently pervasively Python based. We love Rust, but it has a steep learning curve that may exclude data scientists and other non-expert programmers who frequently use TensorFlow. The ownership model is really great, but mostly irrelevant to the problems faced by today’s machine learning code implemented in Python.

coder543• 7 years ago

As I pointed out in two lengthy comments on day one[1][2], that reasoning is nonsense. If Chris wants to use the language he created in this new endeavor for machine learning simply because he made it, that's totally fine and completely his prerogative, but he should just say so, rather than trying (and failing) to convince people that other languages aren't better suited for this task.

From my point of view, a weak justification is worse than no justification in cases like this.

Rust is much better suited to this task than Swift from a technical point of view. The far superior platform support for Windows and Linux is ample reasoning to say Rust is better suited for this task, since very few data scientists will be training models on macOS. However, that's only one of several areas where Swift has shortcomings for a project like this. Swift is great for iOS and macOS development, of course, since it was designed for that. I don't think Swift is a bad language by any means, and with enough effort, it can be reshaped to be good for Tensorflow... the GitHub document just provides zero useful justification for the work required to make it good for Tensorflow.

EDIT: to some of the replies talking about Rust's learning curve, that mostly applies when you start trying to design efficient, interlinked data structures involving ownership. For most applications of machine learning, this simply wouldn't be a problem. The library would provide the data structures, you just have to use them. Rust can provide simple interfaces to complicated things.[3] The compiler's error messages are usually incredibly helpful.

The learning curve of Rust should not be relevant here, compared to Swift, which is also full of idiosyncrasies. Swift and Rust both have a large learning curve for someone coming from Python. This is because they're statically typed languages that are just different from a scripting language. For an application like this, I would say those learning curves are roughly equal at the language level, but as I pointed out in my comments, Swift has an enormous learning curve of requiring many data scientists to either install and learn Linux, or throw out their current computer, buy a Mac, and learn macOS.

My point here is not that Rust is the most suitable language for Tensorflow (although it could be), but rather I'm making the point that Rust is more suitable than Swift for a project like this, and therefore this document is just annoying. It would be better for them to delete this document and just say "we're using Swift because our team has a lot of experience with it and because the creator of Swift is leading this project, so we would lack enthusiasm and momentum if we were using something else, even if it were more suitable."

Julia would be really interesting to see explored further, since it would appeal much better to many existing data scientists who would be transitioning from Python. The times that I've played with Julia, I was amazed at how slow the JIT is for even tiny scripts. LLVM is powerful stuff, but it is painfully slow at everything. It would be nice if Julia offered an alternative backend for rapid development.

[1] https://github.com/tensorflow/swift/issues/3#issuecomment-38...

[2] https://github.com/tensorflow/swift/issues/3#issuecomment-38...

[3] http://kiss3d.org/

Game_Ender• 7 years ago

Chris Latner is the driving technical force behind the project and he wrote Swift. So they were able to fix any issues with Swift so the trade study was “unfair” in that regards.

pjmlp10• 7 years ago

I wonder if Swift could be replaced with Rust for iOS development?

If you like the pain of using a non supported language without all the XCode, UIBuilder, CoreData, Instruments, Metal Shaders debugging,... goodies then yes.

hooloovoo_zoo• 7 years ago

https://xkcd.com/927/

adamnemecek2• 7 years ago

This can be used as an argument against just about anything.

shuoli84• 7 years ago

noop, some solution will dominate. Which will make 14 => 1 or 2.

skwb3• 7 years ago

Forgive me for my ignorance, but does swift have any good plotting and interactive "notebook" ability? Specifically the ability to plot images such as matplotlib.

I ask this because the number 1 reason my deep learning research group chose python was because of the extensive and interactive scientific plotting ability that's built into python jupyter notebooks. While our volume of analysis isn't on the scale of say a google/fb (primarily biomedical image analysis), the ability to easily visually debug the results is much more important for developing robust models.

dynamicwebpaige• 7 years ago

Yes! Swift is supported in Google Colab, and as a Jupyter kernel: https://github.com/google/swift-jupyter.

skwb3• 7 years ago

What is the plotting experience like though? As I previously mentioned, plotting is one of the main reasons our group uses python.

Another reason now that I think about it, is the number of scientific libraries that I can just "pip install" without much thought (such as scipy/opencv).

eschaton• 7 years ago

Interactive plotting and “notebook” capability isn’t a property of a language so it’s fallacious to ask if Swift has it. (Or Python, or Julia, or Wolfram, etc.)

3jckd• 7 years ago

It's interesting how it's going to play out. On one hand side, Swift is a pleasant language to work with (despite its infancy). But on the other, having a Tensorflow API doesn't suddenly give it a bunch of libraries for statistics, comp. vision, modeling, visualisation, etc. that Python/R/Julia coughMATLABcough have.

Nowadays, it's difficult enough to convince people to drop e.g. MATLAB for R or Python for Julia (let's assume that there's some merit to it), despite them having excellent counterparts for almost everything. Swift's success in this domain depends solely on the adoption by developers/researchers/engineers. Unless they're just going to mostly use it internally (as Google is known to).

Which brings me to the last point - why on Earth would they pick Swift (apart from Christ Lattner being involved) when Julia was on the table? It ticks all their boxes and has more mature ecosystem for all things "data". Provided rationale is hardly convincing.

marmaduke• 7 years ago

Julia doesn't tick the "compile to .o/.h" box. As far as I can tell, the use case for AOT Julia is avoiding package compilation overhead, not delivery of standalone code objects.

_edit_ seems JuliaC does support this sort of thing:

https://juliacomputing.com/blog/2016/02/09/static-julia.html

StefanKarpinski• 7 years ago

Julia does already support this kind of thing. Moreover with a minuscule fraction of the money that’s being poured into making Swift usable for data science and machine learning, truly top notch support for generating standalone binaries from Julia could readily be developed. Which is kind of frustrating but what can you do?

celrod2• 7 years ago

If you'd like to track the latest developments:

https://github.com/JuliaLang/PackageCompiler.jl

aschampion• 7 years ago

Yeah, I don't buy the justification versus Julia because of community size either, given most of Swift's community has little to do with data science. The document even says as much, contradicting that rationale, later on.

As someone who uses TF heavily, I would be much more excited about this project if they'd chosen Julia. Swift's tooling isn't great, and I already have a foot in one language with an immature data science ecosystem (Rust).

tomjakubowski• 7 years ago

How is Rust-Julia interop?

microcolonel2• 7 years ago

I feel like GraalVM has a chance to solve some of this at least. I wonder if anyone will make an Octave GraalVM frontend, they already have one for R.

c-cube• 7 years ago

Isn't Graal an Oracle thing? I don't understand why anyone would want to touch that even with a 10-foot pole.

ychen3064• 7 years ago

I will say this in the risk of talking out of my ass as I have no experience in either language :). Having a statically typed language greatly simplifies the tooling because static analysis is much easier; graph program extraction involves one such analysis. When you have to deploy the trained model in production one would hope not to use Python or Julia.

I'd like to add that, with my limited experience in prototyping some of my ML models, having a static checker to check that your tensors have the right shape is much better than having to run your code.

dnautics8• 7 years ago

what's the problem with deploying julia in production in inference? Some occasional piece of data that looks wrong in an unanticipated way causes a runtime type fault? People deploy high uptime websites with django - how do they do it? Well you use kubernetes (or, gasp, systemd) and have restart and load balancing logic. Even if you were typecheck-compiled, you can't guarantee some other developer logic or system error, or an errant bit flip from a cosmic ray, won't take your setup down. Static checker doesn't really matter. If you're at the point where you're ready to deploy, you're probably good for at least 95-99% of the data you'll ingest. The rest of the gap can be closed using rolling update.

cageface3• 7 years ago

Swift is a nice language but its reliance on reference counting means you have to work a lot harder to avoid retain cycles than you do in a garbage collected language.

That might have been the right choice for Apple’s uses of Swift where GC pauses affect the user experience but for most other use cases it’s too much of a cognitive burden IMO.

saagarjha2• 7 years ago

Personally, I find that this only really comes up rarely. Most of the time strong references are fine.

plg• 7 years ago

why on Earth would they pick Swift

Because of iOS?

phillipcarter5• 7 years ago

iOS isn't really relevant for this. You would certainly deploy a model into an app, but that is likely to be using ONNX: https://medium.com/@alexiscreuzot/building-a-neural-style-tr...

agibsonccc• 7 years ago

I'd argue in general that outside of python, there's not really much of a focus from google itself. They are largely leaving other language bindings to other people (see what's currently going on with tensorflow 2.0).

Their focus is more on the c bindings and allowing other people to build what they want on top of that.

Other language bindings aren't generally going to be used for anything more than inference. First class actual data science work isn't going to happen in other languages anytime soon (at least outside of julia and R which are at least trying to compete in this niche).

newen• 7 years ago

Hah...it's funny how they pretend to give objective rationales for choosing Swift when it's pretty clear the decision was made long before.

skohan14• 7 years ago

Tensorflow API doesn't suddenly give it a bunch of libraries for statistics, comp. vision, modeling, visualisation, etc. that Python/R/Julia coughMATLABcough have.

Actually in this case it does. Swift for Tensorflow includes python interop out of the box: https://www.tensorflow.org/swift/api_docs/Global-Variables#/...

The supported use-case would be to do your ML work in Swift, and then call numby etc. from Python.

victor106• 7 years ago

You might agree/disagree with their decision but this is one of the most honest and comprehensive evaluation for using a language I've read.

kodablah• 7 years ago

Strange to see a requirement for choosing one language over another is supposed ease of adoption and then they choose the one not easily adopted across every platform. That easy to write syntax takes precedence over general easy to write/run is unfortunate.

byt1434• 7 years ago

Julia would have been a much better and more cost effective choice in my opinion.

It's a superior platform to on which develop this sort of thing, and further along at that. Also easier to use.

siproprio• 7 years ago

Julia doesn't have a debugger... They specifically claimed this is a very important thing.

In my experience, Julia has also inscrutable scoping rules, a slow REPL, and it's only fast if you don't count the "startup time" of having to precompile everything.

byt1434• 7 years ago

Re: Debugger, fair enough but it's in the works

Re: Scoping rules, these are being evaled.

Re; startup time, already better in 1.1, and will soon be marginalized from two ends: Better static compilation and better tiered compilation.

mamcx2• 7 years ago

I vaguely know tensor flow as the most(?) popular lib of his kind, but I wonder how is the history of swift on non-apple platforms and its impact of the actual users.

Is TensorFlow "huge" in linux, windows, android? Because I also evaluate swift for my use case (https://www.reddit.com/r/swift/comments/8zb9y1/state_of_swif...) and decide instead on use rust mainly because the lack of solid support on non-apple platforms. However, after use rust for months now I still consider swift a better contender between performance/ergonomics than rust (rust is damm hard sometimes, and suddenly you could hit a trouble with above-average complications. I don't see how put this burden in a library to be used for more "regular" folks could work)

skohan14• 7 years ago

The story of Swift on Linux is now quite good.

Windows is less far along, but recently a contributor got nightly builds started on Azure, and it appears there is serious work on this front.

In any case, it's already possible to run Swift for Tesorflow on Windows using WSL and Docker.

mamcx2• 7 years ago

Ok, that is for swift...

But is not tensor flow popular on windows? Because then build on top of swift will mean:

- Put swift on a fast track to be decent on windows, linux, android(?)

- Ignore the windows users and let them battle a bad dependency?

5stospace• 7 years ago

I still don't understand why they would choose Swift over C#?

They complain about C#/Java having "highly dynamic constructs" but correct me if I'm wrong but isn't swift also a GC/OOP like Java and C#?

I don't think Swift has any inherent objective advantages over c#.

I think it would have been a better decision to go with C# over Swift as Microsoft has a clear roadmap with the language and it is already supported on linux/mac/windows.

barbecue_sauce• 7 years ago

While I do think the choice of Swift is kind of weird, you are wrong about Swift being garbage collected (it uses Automatic Reference Counting). It also compiles to actual machine code (rather than an intermediate representation for use in a VM).

pjmlp10• 7 years ago

Reference counting is a garbage collection algorithm as per CS literature, you are mixing it up with tracing garbage collection algorithms.

Swift makes use of SIL and LLVM bitcode before the final binary is produced.

Likewise C# can be AOT compiled to actual machine code via NGEN, .NET Native, CoreRT and Mono/Xamarin.

FridgeSeal5• 7 years ago

I would rather claw my eyes out than write any ML stuff in C#.

It's a fine enterprise language, but good lord writing data science and machine learning stuff in it would be an right pain. It's also not super high performance, and when you're doing a lot of maths heavy operations, high performance is absolutely crucial. I had great difficulty establishing whether SIMD/vectorisation was even supported, and then even more difficulty getting it to work.

Julia would have been a far, far superior choice than Swift.

pjmlp10• 7 years ago

SIMD is supported already for quite some time on RyuJIT. Quite easy to find out when searching the MSDN .NET Blog.

Its performance is good enough for doing medical digital imagining as presented by Siemens at FOSDEM 2019.

It is a matter to properly use the features that the language gives us.

bonesss• 7 years ago

For data science and anything with a demanding domain model I find F# streets ahead of C#.

For a project like this, though, the type F# providers are a bit of a game changer that opens a lot of roads to create a 'best of both worlds' experience. For example, offloading heavy maths to other runtimes while providing a mature stack for everything outside of ML. The F# Type Provider for R (http://bluemountaincapital.github.io/FSharpRProvider/), is an example of this hybrid approach.

I believe Julia looks to be the better choice over Swift, tho.

tanilama2• 7 years ago

Fail to see the point of this project.

Swift for Tensorflow might work if the scope is to create a client side model definition loader natively for various TF models.

Nobody use Swift seriously for server side training, there is no point in doing so except to add swift to the list of language that claim to do deep learning but in reality nobody will consider them.

moocowtruck• 7 years ago

why? because i made the language, thats why...no real good reason

FridgeSeal5• 7 years ago

And here's 3 pages of very vague, half-justifications as to why we didn't choose anything else to head off any complaints.

solidsnack9000• 7 years ago

Setting aside for a moment the appropriateness of Swift for TensorFlow, this is a very impressive example of using an embedded DSL to work with a component that is a full programming system in its own right.

On the one hand, we do want full access to the programming model exposed by the component -- its control structures, abstractions, everything else. One the other hand, these are mostly duplicated by our host programming language: it's going to have variable bindings, operators, iteration, conditionals and everything else. Doing an embedding like this is a way to expose most of the component's facilities without introducing a ton of "new syntax" in the form of combinators or having programs where a lot of the critical code is escaped in strings.

This same problem shows up in programming interfaces to RDBMSes. LINQ is a good example of the same embedding technique.

tylerwhipple• 7 years ago

I am surprised Dart is not mentioned at all (maybe implied under the OOP languages?). While Flutter and Tensorflow are very different usecases, I am surprised there is nothing in the document on why Dart specifically would be a good choice. I believe if they used Dart for Tensorflow as well, the community would be able to get behind the idea that will not be an abandoned language.

masha_sb• 7 years ago

I'll wait for the PyTorch version.

xiaodai• 7 years ago

Haven't we seen this before?

skohan14• 7 years ago

It's been in development for about a year, and I have seen several posts about it here. It's still not quite feature-complete or ready for real use.

sometimesijust• 7 years ago

https://github.com/malmaud/TensorFlow.jl/blob/master/docs/sr...

ramoz• 7 years ago

Is there an official release roadmap?

skohan14• 7 years ago

Last I heard the goal for "initial adoption" is set for Spring 2019.

midgetjones• 7 years ago

I wonder what this means for iOS apps themselves

skohan14• 7 years ago

At the moment not much. Swift for TensorFlow is a fork of the language, with language-level support for some features which are useful for data science, for instance automatic differentiation and dynamically-callable objects.

Some of those features are making their way into the main branch, but at the moment you could not import the TensorFlow library into an iOS project and use it. Swift for TensorFlow needs to be built using a separate toolchain.

mikkelam• 7 years ago

As as python machine learning practitioner and previous iOS engineer I have for while come to miss using swift and type safety for that matter. I really like the language and wish great success for the TF team with swift.

Side note, does anyone know the effort required to get various python based libraries running on swift? i.e. numpy, scipy, pandas and so on?

physicsyogi• 7 years ago

The Swift for Tensorflow team has added some python interop to Swift. So you’ll be able to, for instance, do an “import numpy” in your Swift code.

lovasoa• 7 years ago

Reading the document really gives a feeling the author is not being honest on why they chose Swift.

The lack of windows support is addressed in just two lines. Julia being an already established language in the domain of data science does not seem to be especially important to them.

I think the most honest part of the document is:

because we were more familiar with its [Swift's] internal implementation details

Why Swift for TensorFlow?

Related Stories