HN user
meet.hn/city/47.0708678,15.4382786/Graz
Socials: - github.com/majodev - mranftl.com
---
Uff, I had no idea that Postgres v15 introduced WAL_LOG and changed the defaults from FILE_COPY. For (parallel CI) test envs, it make so much sense to switch back to the FILE_COPY strategy ... and I previously actually relied on that behavior.
Raised an issue in my previous pet project for doing concurrent integration tests with real PostgreSQL DBs (https://github.com/allaboutapps/integresql) as well.
Oracle Cloud Infrastructure tries to fill exactly this sweet spot. Cheaper compute than the other hyperscalers, while still offering similar security features (TPM, Shielded Instances, Measured Boot) and a bare-metal-first focus.
Disclaimer, just joined Oracle a few months ago. I'm using both Hetzner and OCI for my private stuff and my open-source services right now. I still personally think they've identified a clever market fit there.
Location: EU, Austria
Remote: Yes, EU timezones preferred
Willing to relocate: No
Technologies: Go, TypeScript, Kubernetes
Résumé/CV: https://www.linkedin.com/in/marioranftl
Email: hn@mranftl.com
Made the switch to self-hosted https://mailu.io (on k8s) 2 years ago when Gandi announced the deprecation of their free plan.
Happy after IP was off most blocklists, but setup was kinda rough - https://mxtoolbox.com is your friend.
Location: Austria, EU
Remote: Yes, EU timezones preferred
Willing to relocate: No
Technologies: Go, TypeScript, Bash, PostgreSQL, and all things cloud-native: Kubernetes (GKE, RKE2, K3s, EKS, kind), Ansible, Argo CD, Terraform, Helm, Docker, GitHub Actions, CKA Certified.
Résumé/CV: https://www.linkedin.com/in/marioranftl
Email: hn@mranftl.com
--
Former Platform Engineering Lead specializing in Kubernetes and cloud-native technologies. Over ten years of experience building client applications and designing maintainable, scalable API services on highly available infrastructure. Experienced working with both corporations and startups! I build (bare-metal) clusters for fun. Seeking my next challenge with an international team and/or in HPC or massive cluster environments. Let’s connect!
Location: EU, Austria
Remote: Yes, EU timezones preferred
Willing to relocate: No
Technologies: Cloud-Native, Kubernetes, Go, TypeScript, CKA Certified
Résumé/CV: https://www.linkedin.com/in/marioranftl
Email: hn@mranftl.com
Happy to hear that! When it comes to testing services that depend on PostgreSQL, this is still my preferred solution.
https://github.com/allaboutapps/integresql
disclaimer: author
google webfonts helper - https://gwfh.mranftl.com
According to the Cloudflare December monthly stats, I had roughly 57k unique users, 15m requests, 1,3TB traffic. Though, most requests are likely to be bots/integrators spamming the API...
Running on a bare metal k8s cluster (libvirt) on top of a single dedicated Hetzner server, ~70€/month. Not going to monetize it, but will maybe accept donations/sponsorings in the future...
Just moved off my 8 years old project google-webfonts-helper https://github.com/majodev/google-webfonts-helper from their free tier to my own private infra and replaced the current dyno with a 301 handler: https://github.com/kenmickles/heroku-redirect
AFAIK sadly Heroku does not provide some other _free_ permanent redirect option for their *.herokuapp.com sub-domains without actually running a dyno there.
Very happy user of https://github.com/watchexec/watchexec (rust), can recommend.
No and we are happily using it within our overcommitted cluster (combination of shared and dedicated nodepools).
We are a small team of 5 infrastructure engineers and previously managed 200+ libvirt VMs running on bare-metal HA hypervisors in a GlusterFS storage pool (software agency, different customer application services). We started to migrate to GKE in 2017 and finished within a year or so.
I know many associate k8s with a yaml mess, but this is actually our most favourite part of it. We are able to describe a whole customer project in this format and it's not something we have to maintain in-house (Ansible). As long as you don't try to be smart (templating/helm, operator dependance), it works out pretty well, prefer plain manifests and extend that with you own validation scripts.
Nevertheless, if you have no 24/7 operations, stay the hell away from bare-metal - go managed.
I really tried to like netbird, but their current hard dependency on auth0 (also when selfhosting) unfortuantly put me off. Your statements regarding NAT punching are on the other hand very interesting.
I'm currently trying out innernet, mostly for the interconnected server nodes (k8s, not for actual real users). Seems to work fine (double NAT + single public coordination server).
Unrelated to Wireguard: I really liked Nebulas certificate-based client setup and its reduntant lighthouses (public coordination servers). May be an alternative if you want something at least a little faster than wireguard-go based implementations.
I really love tailscale for my private network.
Nevertheless, I still hope they will revisit support for proper linux kernel wireguard sometime in the near future. This would allow to ditch separate meshing technologies for connecting server nodes (and routing into separate subnets via a tailscale subrouter node). Best of both worlds - ease of use and performance.
I really only started to have a look at other tools (like netmaker, netbird, innernet, wesher) because of these performance caveats (wireguard-go).
I can confirm that goaccess databases works great for incrementally regenerating these HTML reports. We currently pipe all logs from all our ingress-nginx LBs every 15 minutes (and grep by virtual host) to goaccess and thus update its reports periodically. Each month, all reports get archived so we can start with a fresh report again.
Semi live and super useful without having to use 3rd party services like Google Analytics. The HTML report is self-contained (single file) and thus can easily be shared (or just statically hosted).
Author here, your feedback is highly appreciated.
Providing better documentation for the RESTful JSON endpoints of this project is our next top priority.
Seems like it would be important to have a minimal example (e.g. curl + psql in a shell session) as well.
Yes, monolithic Go RESTful JSON API services - we currently depend on go-swagger, SQLBoiler (PostgreSQL) and echo.
Workflow: we use VSCode with devContainers (docker-compose local dev env), our tasks are simply defined in a Makefile. Our high-lvl dev workflow is documented here: https://github.com/allaboutapps/go-starter/wiki/FAQ#how-does...
Difficulties: Nothing special actually. We switched from Node.js to Go in the beginning of 2020 and it's been a pleasure for our new projects (software agency). I thought it was going to be hard to onboard our staff / new hires to Go and our stack, but this really turned out to be a non issue.
yep, currently it forces our engineers to use VS Code or a terminal-based setup (or GitHub Codespaces). Hopefully Goland / IntelliJ catches up in this year...
Can only second this, we use a VSCode devContainer-based setup in all of our projects and even migrate our legacy projects to it (software agency).
Here's our current base go template, you only need Docker+VSCode on your system to get started: https://github.com/allaboutapps/go-starter
Bonus points:
* As all IDE operations solely run within the local Docker container, all developers can expect that their IDE will work the same without manual configuration steps.
* We can easily support local development in all three major OSes (MacOS, Windows, Linux) and even support developing directly in your Browser through GitHub Codespaces.
* Developing directly inside a Docker container guarantees that you use the very same toolset, which our CI will use to build these images. There are no more excuses why your code builds differently locally versus in our CI.
Edit: format/typos
Highly appreciated hint (original author). Note that I haven't updated this project in ~4 years and it's still spinning on Heroku's free tier, haven't spend a penny for the API hosting costs.
Some recommendations regarding the CSS may be outdated by current "webfont hosting standards". Let's see what 2021 will bring for this project...
While PostgreSQL templates allow to scaffold new databases (one per test-spec) quite fast, it's even faster to have a warmed up pool of databases ready: https://github.com/allaboutapps/integresql#background
Disclaimer: author.
Great points! Any hints regarding preexisting linters for go code that errors if we execute outside (an already opened) db transaction?
Also, regarding writing tests against real PostgreSQL databases, isolated per test and without losing speed, I can recommend to take a look at https://github.com/allaboutapps/integresql - disclaimer, we are the authors.
I'm sorry, our documentation is not ready for primetime yet (an OpenAPI spec for this RESTful JSON API managing PostgreSQL test databases is still missing).
However, I've just updated the Background - feel free to file an issue if some aspect is totally left in the dark.
I'm generally not a fan of emulating database behavior through a mocking layer while testing/implementing. Even minor version changes of PostgreSQL plus it's extensions (e.g. PostGIS) may introduce slight differences, e.g. how indices are used, function deprecations, query planner, etc. . It might not even be an erroneous result, just performance regressions or slight sorting differences in the returned query result.
We try to approximate local/test and live as close as possible, therefore using the same database, with the same extensions in their exact same version is a hard requirement for us while implementing/testing locally.
However, I have no experience with H2 (and the modern Java ecosystem in general), so cannot talk about how close their emulation implementation resembles PostgreSQL behavior.
Yep, inspecting the actual database state after multiple test failures directly is very nice (we log the used test-database after before every test, so it's easy to connect afterwards).
Regarding cleanup: We run configure a max pool size in integresql after which "dirty"-flagged test databases are automatically removed (and then recreated on demand). We may also support auto-deletion (e.g. after a successful test) in the future, however currently it just does not seem necessary even when we have 5000+ test-db, it's just a disk-space concern.
We did several benchmarks back in the day and it definitely depends on the size of your migrations + fixtures that you need in every test. Performance increases were simply huge for us, despite the serial execution of our tests in our legacy backends.
A new database scaffolded from a predefined template becomes available in several milliseconds or few seconds VS. truncating, migrating up and seeding (fixtures) would always take several seconds. We initially experimented with recreating these test-databases directly via sql dumps, then binary dumps until we finally realized that PostgreSQL templates are the best option in these testing scenarios.
Sorry, I've got no benchmarks to share at this time.
Yep, disabling fsync, synchronous_commit and full_page_writes ONLY for this PostgreSQL integration testing strategy gives us an additional boost of ~35% in overall testing speed. https://github.com/allaboutapps/go-starter/blob/master/docke...
Regarding being language agnostic: YES, clients for languages other than go are very welcome. The integresql server actually provides a trivial RESTful JSON api and should be fairly easy to integrate with any language. This is our second iteration regarding this PostgreSQL test strategy, the last version came embedded with our legacy Node.js backend stack and wasn't able to handle tests in parallel, which became an additional hard requirement for us.
See our go-starter project for infos on how we are using integresql, e.g. our WithTestDatabase/WithTestServer utility function we use in all tests to "inject" the specific database: https://github.com/allaboutapps/go-starter/blob/master/inter...
We are using PostgreSQL templates to speed up our integration tests. This allow us to have full blown (and non mocked) databases which are isolated to each parallel running test.
We have recently open-sourced our core-server managing this (go): https://github.com/allaboutapps/integresql
Projects already utilizing Go and PostgreSQL may easily adapt this concept for their testing via https://github.com/allaboutapps/integresql-client-go
Creator here. Yeah, you can possibly switch to "Modern Browsers" by now. Haven't touched this project in a while...
We at allaboutapps are using verdaccio since ~1year, primary for centrally publishing our internal packages (e.g. our whole base backend stack) and testing packages before actually releasing them to the wild. Typically yarn interacts with it, never encounted any problem.
The proxy has also been a handy feature for us to mitigate the risk of broken builds due to now unpublished packages from the official npm (though this shouldn't be a problem anymore). Can really recommend it.
Attention!
Former downloads only provided fonts with the default Latin subset, this has been resolved (charsets are selectable). I'm sorry if this causes any inconvenience (please keep in mind, it's still a prototype)...