Pyinfra automates infrastructure super fast at scale

by gjvc • 4 years ago

147 37 4 years ago

turtlebits• 4 years ago

Does this have any advantages over Ansible? It seems to have some config management, but not as robust or well documented. Ansible also does parallel remote execution.

dal• 4 years ago

I guess one would be that it is python and not Ansible.

karlicoss• 4 years ago

Yes.

- you use an actual programming language (which you're likely to already know) instead of desperately figuring out how to simulate a for loop in yaml or to replace a part of a string, or whatnot

- you have all Python static analysis tools at your disposal. mypy/pylint etc can make your deploys less error prone

- you can easily implement a custom DSL in python, so deploys ends up more declarative and with less copy pasting

- it's easy to implement custom arbitrary operations (since you're working in python) -- and again you can reuse variables, etc instead of having to use some weird templating or pass them in command line

usrme2• 4 years ago

At what point are you just re-inventing what Ansible already does if you're going down the DSL path? I sometimes feel like I'm in the minority, but I truly do like Ansible and if you slap Mitogen[1] in front of it makes the whole process even more enjoyable!

I'm honestly trying to understand whether this hating on Ansible is more of a bandwagon thing (similar to how people shit on the Python dependency management ecosystem) and stems from having bad experiences that have completely turned them off of it.

Honest questions in response to your points:

1. What are you deploying where you would need static analysis?

2. If you did end up writing your own DSL on top of Python at what point would you concede that it's probably better to use someone's else creation as opposed to forever iterating on your own thing for the sake of using your own thing?

3. Why would you want "custom arbitrary operations" and how custom are they that Ansible and Jinja2 can't do that for you in usage patterns that are far more common than your own creation would be?

---

[1] https://mitogen.networkgenomics.com/ansible_detailed.html

blondin• 4 years ago

so pulumi[1]?

[1] https://github.com/pulumi/pulumi

catstack• 4 years ago

Does any one have any links to some good writes ups on this being used in a bigger environment?

Anyone want to post their experience using it here?

mdaniel• 4 years ago

There was more commentary about it when it was posted in 2020: https://news.ycombinator.com/item?id=23487178 (although I didn't go through them all looking for endorsements)

Fizzadar5• 4 years ago

I wrote pyinfra so am biased but we used it at my previous company on a fleet of roughly 1.5k physical boxes (including deploys spanning all of them at once) extensively, so it scales to that level well. We had probably 30 groups within that each with their own deployments as well and all tied together using a single Jenkins box.

I would love to test beyond that but haven’t had the opportunity sadly! Performance and scale is a headline feature so will always look to push this further.

gautamdivgi• 4 years ago

What’s the difference between pyinfra and fabric? Both seem to be similar.

BiteCode_dev• 4 years ago

idempotence

mdeeks2• 4 years ago

How is the performance with larger deploy scripts? Ansible really struggles with hundreds of hosts and for simple things like uploading a dozen files to each host. The benchmarks are promising but the test really only does four things. https://docs.pyinfra.com/en/2.x/performance.html

Fizzadar5• 4 years ago

Way back when I started pyinfra my only goal was performance I recall turning a 40 minute ansible run on 500ish hosts into 5 minutes. Big caveat: anecdotal and pretty old information there.

It’s been too long since I did perf testing but I still believe pyinfra Will significantly outperform ansible for the same tasks, you’ll have to try it ;) (also if perf is not good please raise an issue perf is absolutely a feature I wish to maintain)

reifyx• 4 years ago

I haven't used this for large deployments but I use it for my personal server and it works perfectly. Almost everything is built in and I can easily write my own custom operations when I need them. Documentation is good and the operations are well designed.

Only downside is I couldn't make it work with my SSH agent, but that might be a problem with Paramiko and not Pyinfra.

Fizzadar5• 4 years ago

Only downside is I couldn't make it work with my SSH agent, but that might be a problem with Paramiko and not Pyinfra.

Interested to hear more about this, might be something that can be added to paramiko (or on top of).

dang• 4 years ago

Show HN: pyinfra v2 - https://news.ycombinator.com/item?id=30999030 - April 2022 (2 comments)

Pyinfra v2.0 Released - https://news.ycombinator.com/item?id=30973976 - April 2022 (3 comments)

Show HN: Pyinfra v1.4 - https://news.ycombinator.com/item?id=26983266 - April 2021 (3 comments)

Pyinfra – automate infrastructure super fast at scale - https://news.ycombinator.com/item?id=23487178 - June 2020 (64 comments)

Pyinfra v0.2 - https://news.ycombinator.com/item?id=12956784 - Nov 2016 (2 comments)

Fizzadar5• 4 years ago

Hey all, I (& many amazing contributors) built pyinfra! Crazy to see it on the homepage I’ll do my best to answer any questions!

dolni• 4 years ago

Does pyinfra play nicely with devices that provide an SSH interface, but aren't traditional Unix boxes?

I use Ansible currently to manage the config on a Ubiquiti EdgeRouter. Using actual Python would be even better.

Fizzadar5• 4 years ago

Yes! I know of people that have connected to systems without any traditional shell/unix stuff. You can change the shell (or have none) used if needed using the global `_shell_executable` global arg (https://docs.pyinfra.com/en/2.x/arguments.html#shell-control...).

Most operations rely on various Linux/similar tools but the `server.shell` operation plus shell flag above should get you connecting and executing commands. Please do reach out if this doesn’t work for your setup!

jononor• 4 years ago

Interested in the same usecase, but for Mikrotik routers.

ngcc_hk• 4 years ago

Read a few pages I gave up. May I ask - what is its use case? - security - charge model - self hosting Sorry if miss a faq on the site ?

inhumantsar• 4 years ago

Late reply but I'm curious if the focus is idempotent execution (like ansible) or desired state (like puppet).

I'm guessing the former since it's agentless.

ebreton• 4 years ago

How would you use pyinfra to manage a fleet of edge devices ? The main difference with a park of servers being that the edge devices are mounted on vehicles and therefore switching on and off with their own rhythm each. We cutrently get an alert from each device when an update is needed, ssh into it, and docker-compose the latest services.

emrahayanoglu• 4 years ago

Nice question! I am also wondering how it helps for the IoT devices... Up to now, What we found out is that Pull way makes more sense instead of Push way for the fleet of more than 500Ks devices in the field. We use Ansible Pull, but would also considering of such a way through GitOps approach. Not sure, if it would be also provided via Pyinfra to get such stuff via Pull way.

v3ss0n• 4 years ago

That the tool i am looking for a while. I used fabric before but it's too old and limited, no longer maintained. Plumini is focused on cloud, not on dedicated or bare metal servers. Why I don't like Ansible is : YAML is a bitch to maintain and python is much more dynamic and flexible.

So this is the real thing. I am going to use it for my DevOps life.

Keep it up!!

zach_garwood• 4 years ago

I use pyinfra for my personal projects and I love it. Compared to Ansible it's a breath of fresh air. Ansible is very straightforward to start with, but quickly devolves whenever you step off the beaten path, and you end up with a tangled mess of unintelligible YAML after a while. Pyinfra is just Python. If you can read Python, you can understand what the deployment module is doing.

darthrupert• 4 years ago

It seems like Pyinfra is sort of declarative in the same style as Ansible. Is this a correct assumption?

rkwasny• 4 years ago

Pyinfra is great, simple and no DSL! just pure python

Pyinfra automates infrastructure super fast at scale

Related Stories