Tilt – Local Kubernetes development with no stress

https://github.com/windmilleng/tilt

by ducaale • 8 years ago

217 72 8 years ago

github.com

flurdy• 8 years ago

Uuuuh not using YAML. Controversial. (in the docker/k8s ecosphere, where YAML seems ubiquitous)

I don't love YAML though I am not sure Starlark looks any more readable.

YACL => Yet another config language...?

sb8244• 8 years ago

This would be a stopper for me simply because all k8s config my team does is in yaml

q3k3• 8 years ago

YAML, especially how it's used in k8s, is terrible. Writing it by hand is mind-numbingly toilful and extremely error prone, and templating YAML with Go (like Helm does) is a travesty.

The most sensible approach I've seen is to use jsonnet+kubecfg. This lets me use a somewhat sane, composable, turing complete language that spits out a tree of k8s objects. I can use as much or little repeated definitions (or abstractions) as I like. Then kubecfg performs all the kubectl invocations to actually make the cluster state match my calculated definitions.

fermuch• 8 years ago

I'd also point out pulumi. It is a service, but it solves a lot of trouble when dealing with k8s configuration, and you have a real language to to create your definitions...

dilyevsky• 8 years ago

Have you seen https://github.com/stripe/skycfg ? It’s kind of early development stages tho

sergiusignacius2• 8 years ago

I don't see how Starlark is better... Does it validate what you write somehow - in a way that couldn't be done by validating a YAML?

gingerlime• 8 years ago

I'd love to see a writeup about your process or workflow.

I keep seeing helm, and the overall idea of a "package manager" sounds nice, but having looked at some charts, and tried to understand the value proposition, I get a very uneasy feeling somehow. Perhaps it's lack of familiarity? but your comment seems to suggest something is inherently wrong, so I'd love to understand more...

linkmotif9• 8 years ago

My biggest problem with local k8s dev is building containers. I’m on macOS. Does this solution address that?

yebyen3• 8 years ago

Is the problem with building containers that you don't have a Dockerfile and you don't want to spend time making one?

Draft does that pretty well with Draftpacks, in my experience if there is a Draftpack for what you're using then the Dockerfile that draft spits out will get you pretty close to a workable container image with basically no effort, or sometimes even all the way there.

I'm trying to understand what you mean that building containers is a problem. Is it the procedural act of actually building and rebuilding the container that is the problem? (Draft handles that too...)

Edit: According to the Tilt tutorial, yes, tilt will build your containers. Eg: https://docs.tilt.build/first_config.html

linkmotif9• 8 years ago

I’m all aboard Docker, but I don’t use Linux natively, so I’d need to run a VM inside macOS, or have another box to build containers. Even if I did, I use GCR in production, so I’d need to template my container images in helm to use GCR only in production versus whatever I use locally for development. And what would that even be? I haven’t explored minikube to know what it does for local dev container registry.

Anyway, thanks for the link. Missed that.

EDIT: Looks like tilt is using GCR in those examples. I’m not trying to traffic container images off of google cloud for local development. That requires (fast) internet just for local development, and outgoing traffic really adds up quickly in terms of cost.

PSS: never heard of Draft. Checking it out now. Not using azure though, maybe that’s not relevant.

johnmarcus2• 8 years ago

Try working with Elixer and Phoenix and you will start to appreciate how builds can be a nightmare.

rcarmo2• 8 years ago

I have no problems building containers on macOS. Can you elaborate?

linkmotif9• 8 years ago

Building containers requires Linux, so I need to run a VM or build on another box/the cloud.

figpope• 8 years ago

Is your issue with building containers and getting them into the minikube VM?

If so, you can use "eval $(minikube docker-env)" to make your shell's docker commands talk to the docker daemon in the VM.

linkmotif9• 8 years ago

Running a VM is a total drag. I only have 16GB RAM. Then even if I run a VM, the docker registry is not the same as my production docker registry (GCR). I guess I can account for this with a variable in my helm templates, but then it’s just more complexity.

Would you recommend minikube even with 16GB RAM? How much do you allot to the VM?

maxdo• 8 years ago

is there anything working with helm in the similar way ?

GarMan2• 8 years ago

There are at least two options that work with helm, google's skaffold (https://github.com/GoogleContainerTools/skaffold) and microsoft's draft (https://github.com/Azure/draft). The former supports helm along with kustomize and just plain old kubectl yaml files, draft seems to be tied to helm.

Personally I have used skaffold with success, and draft paniced on me so I stopped playing with it.

kkapelon2• 8 years ago

there is also garden.io

dexcs• 8 years ago

Tell me you don't work for them...

metzby12• 8 years ago

Tilt can use helm! (Disclaimer: Tilt CEO here)

You can just call k8s_yaml(local('helm template <args>')). That tells Tilt to parse the yaml generated by running (locally) helm template. Feel free to reach out if you'd like to pair to try it.

rlamana• 8 years ago

At Okteto we had similar issues so we built our own tool and open-sourced it as CND (https://github.com/okteto/cnd). Our approach is to be decoupled from how your pod is deployed so it works with Helm, CNAB, kubectl, etc…

rcarmo2• 8 years ago

I love the name, especially considering that the GitHub username is 'windmilleng'.

(if you don't get it, look up Dom Quixote)

FrankPetrilli• 8 years ago

Hey, awesome!

I met the Windmill Eng team a few weeks ago at Kubecon in Seattle, awesome to see tilt here. It's definitely a young product, but it's a good start to exactly the experience I want.

By experience I mean that when I do local dev and edit code live while reloading or restarting the service, I see changes reflected immediately in 2-3 commands that take seconds.

In moving software towards Kubernetes, I come to rely on the service abstractions available therein. So it's sensical to develop in the same kind of environment. Tilt gives me the ability to get the save => build => package => push => reload loop into Kubernetes and without the manual hassle thereof.

I've used it in minikube, on-prem, and GKE. Running a dev box in a GCE instance and running tilt against GKE is a super nice environment.

(Fair notice: I'm a Googler who focuses on GKE, so bias and all, my opinions are not Google's, etc.)

metzby12• 8 years ago

Thanks Frank! It was great to chat at Kubecon. (I'm Dan, the Windmill CEO, goofy big-haired guy)

karakanb2• 8 years ago

I think the docs kind of misses what problem it solves and how it solves it. I do all the local stuff with docker-compose while using the same images for k8s deployments, how does this help me? Does this aim to replace docker-compose with k8s, or is this just for testing k8s resource changes in realtime, or is it testing your local k8s resource definitions against a remote cluster? Obviously this is useful for some people, I would like to understand what problems this project adresses and how it adresses them.

Also, doesn't not reusing the k8s definitions and re-defining them in a tiltfile cause yet-another-config-to-manage, which can quite easily digress from the production resource definitions, which is one of the problems using docker-compose for local development and k8s for deployment? I would really like to read the problem and proposed solutions in a detailed documentation page. I might have missed it if there is already one, please correct me if so.

metzby12• 8 years ago

Hey, I'm CEO of Tilt. Thanks for the feedback.

The problem this solves is that distsys devs don't get enough feedback. You say you use docker-compose. Compared to docker-compose, Tilt has two advantages: 1) It updates the services as you edit them 2) Its UI makes it easy to see where errors are occurring, without you having to play 20 questions on the command-line or worry about things scrolling off-screen.

The Tiltfile doesn't redefine your k8s definitions; it reuses your existing k8s definitions. The Tiltfile just tells Tilt how to get them. (Many teams don't just use yaml files, but generate them, so Tilt has to be able to do that, too.)

There is a landing page at https://tilt.build which has more of the "why Tilt". Also, we're adding native support for docker-compose next week. I'd love to understand your case more, because at a high level you sound like exactly the kind of dev we want to help.

hodgesrm2• 8 years ago

Hi! Thanks for checking in on HN. I'm very interested in k8s productivity but had some trouble understanding what Tilt is actually doing--is it a replacement for minikube, hosted kubernetes, etc.?

Is there an architecture diagram that shows what runs where?

jacques_chester• 8 years ago

There is a landing page at https://tilt.build which has more of the "why Tilt".

As a tangential by-the-way, I notice you're based in NYC. Of interest, Googlers working on Knative Build (hi Jason!) and Pivots working on Cloud Native Buildpacks (including me) are based in NYC also.

GarMan2• 8 years ago

(Many teams don't just use yaml files, but generate them, so Tilt has to be able to do that, too.)

Tell me more about this? I have my own helm like product and I had issues getting that to work with skaffold.

johnmarcus2• 8 years ago

It looks like your basically recompiling/rebuilding the container and just updating what’s running (in what I assume is a k8s cluster on OS X). This all seems like a terrible idea. No way devs want to wait for a container rebuild to see updated code changes, that’s highly unproductive. They usually don’t run k8s locally either, because it’s an abstraction later they hardly want/need to concern themselves with while writing code. And as a say admin, truthfully I don’t really want them to try because it would double my own already heavy workload. This seems to solve a problem in a way that only works in the most ideal enterprise (slow moving) work environment. I currently solve the problem with carefully built containers which permit volume mounting and and code reload natively using typical dev flags for the run command. It requires more thought upfront than is ideal, but does typically solve the problem. Cheers.

karakanb2• 8 years ago

AFAIK, the whole idea behind the cloud-native approaches and tools in the ecosystem, such as Docker, Kubernetes, Helm, Terraform, focus on reproducible builds that are platform-independent, at least that is my understanding of the ideal. I imagine having an application code, my Terraform definitions, k8s resources and helm charts, and basically I should be ready to deploy my application to any of the big cloud providers. This doesn't have to be the case for each and every application, but I believe we should all be moving towards a decoupled infrastructure where the application code itself is independent from where it is running in terms of infra.

So, considering the point above, using Tilt for updating production cluster state and resources with hot reloading seems like a no-no, as this may easily cause configuration drift, which would make the whole cluster state independent from the k8s resource definitions. We should be moving towards reproducible builds and infra, so this usecase seems to fall short in my opinion.

Replacing docker-compose seems like an exciting point, though the value addition is still not clear for me to abandon a tool I have been using for months for developing the application code, not the infrastructure stuff. I can see all the logs from all the containers on a single terminal, every developer in my team can run the whole application with a single shared docker-compose definition, and everyone in my team is familiar with it already. As far as simplicity goes, I don't see how Tilt simplifies my workflow since it requires a configuration script instead of docker-compose.yml file, and it is yet another tool to learn in that regard. Ideally, I would like to remove all the dependency from docker-compose as keeping it in sync with the k8s resource definitions requires effort, though Tilt also requires a similar configuration AFAICT from the docs. The best workflow for local k8s-based development seems like using minikube with the mounted directories or something similar, which means every change in the k8s resources made locally would also be present on the production setup, hence keeping production setup completely reproducible on every cluster, including local stuff like minikube.

Please don't take my words in a bad way, I am really looking forward to replace my docker-compose based setup with something based solely on k8s definitions, therefore I am playing devil's advocate for Tilt in order to clarify its benefits so that I can make an informed decision about employing it or not. Thanks for jumping in on HN to answer questions and accept feedback, I really appreciate that.

antics• 8 years ago

It can be a pretty big pain to get local versions of your all your k8s services running on, say, minikube. Tilt makes this really easy because you're developing your app against the real services in your cluster.

kkapelon2• 8 years ago

How does this compare with garden.io, skaffold, draft, microk8S etc???

metzby12• 8 years ago

Hey, Tilt CEO here, so I'm certainly biased, but also semi-informed.

We think you deserve three properties: 1) it's easy to start your whole app 2) it's fast to update your app as you edit 3) common problems can't be missed.

docker-compose gives you 1. (some people hack in mount points and file watching to get 2, but it's hacky)

skaffold and garden.io give you 1 and some of 2. (Tilt has fast_build, which can update pods in-place, without rebuilding the whole image, which I don't believe either skaffold or garden.io do)

Our big value-add is the UI. Which may sound weird for a Terminal UI, but if you look at the demo video on https://tilt.build , I think you'll get a sense for how Tilt is working to keep the problem that's blocking you in your face, so you don't waste time playing 20 questions with kubectl.

eysi• 8 years ago

Hi all, Garden CTO here. First of all, great job on Tilt! A lot of interesting stuff happening in terms of DevEx in the multi-service realm.

I just wanted to chime in on the points above. Regarding 2), Garden does indeed support updating pods without re-building and re-deploying via our hot-reload feature (https://docs.garden.io/using-garden/hot-reload) which essentially copies source files into the running container on file save (works best for dynamic languages).

Regarding 3), Garden has a terminal UI that shows the status of individual services and updates as changes are made to the codebase. It will for example print error messages for failed container builds and failed deployments. If configured so, Garden can also run tests on code changes and will print the error output if tests fail. Our next release will also contain the first version of a dashboard which displays service statuses and dependency graphs and updates in real time. However, our terminal UI is not interactive like Tilt’s—which looks really nice!

jaytaylor• 8 years ago

In the readme there's a link to kubernetes slack. Anyone know how can I get an invite to join the workspace?

metzby12• 8 years ago

http://slack.k8s.io/

leandot• 8 years ago

It actually uses your existing Dockerfile and existing k8s yaml definitions contrary to what some people are saying in the comments,the Tiltfile seems to be just a handful of glue logic. The logs ui is pretty nice too, I've definitely looked for an easier way to check logs from multiple running pods.

tirumaraiselvan• 8 years ago

Seems like this is the main product of a full-time startup. In that case, I am curious how are you thinking about the future and business model considering it is 1) open source and 2) many competitors.

phentz• 8 years ago

How does it compare with Telepresence + skaffold ?

I like the console UI idea.

I dislike the server components (the same way I dislike draft and tiller).

Tilt – Local Kubernetes development with no stress

Related Stories