jake_morrison

I thought about doing this years ago, but without AI, the only way was to have people do the analysis. I thought this was bad karma, as I would end up paying people in Bangladesh to look at the stuff that Americans eat.

Embedded systems often have crappy compilers. And you sometimes have to pay crazy money to be abused, as well.

Years ago, we were building an embedded vehicle tracker for commercial vehicles. The hardware used an ARM7 CPU, GPS, and GPRS modem, running uClinux.

We ran into a tricky bug in the initial application startup process. The program that read from the GPS and sent location updates to the network was failing. When it did, the console stopped working, so we could not see what was happening. Writing to a log file gave the same results.

For regular programmers, if your machine won't boot up, you are having a bad day. For embedded developers, that's just a typical Tuesday, and your only debugging option may be staring at the code and thinking hard.

This board had no Ethernet and only two serial ports, one for the console and one hard-wired for the GPS. The ROM was almost full (it had a whopping 2 MB of flash, 1 MB for the Linux kernel, 750 KB for apps, and 250 KB for storage). The lack of MMU meant no shared libraries, so every binary was statically linked and huge. We couldn't install much else to help us.

A colleague came up with the idea of running gdb (the text mode debugger) over the cellular network. It took multiple tries due to packet loss and high latency, but suddenly, we got a stack backtrace. It turned out `printf()` was failing when it tried to print the latitude and longitude from the GPS, a floating point number.

A few hours of debugging and scouring five-year-old mailing list posts turned up a patch to GCC (never applied), which fixed a bug on the ARM7 that affected uclibc.

This made me think of how the folks who make the space probes debug their problems. If you can't be an astronaut, at least you can be a programmer, right? :-)

I am optimistic about Ubuntu Chilselled, which automates this process: https://canonical.com/blog/chiselled-ubuntu-ga

Here is a full example: https://github.com/cogini/phoenix_container_example/blob/mai...

Apple making cars was an interesting play when they had a bunch of cash overseas and good relationships with Chinese manufacturers. Tax law changes made bringing money back to the US less punishing, and manufacturing in China is becoming problematic.

You can do something similar with docker compose, driving the system from the outside. Create dockerized versions of dependencies like the database, build and run tests, and then run tests against the production app container. It's particularly useful for testing a set of microservices.

See https://github.com/cogini/phoenix_container_example for a full example. This blog post describes it in detail: https://www.cogini.com/blog/breaking-up-the-monolith-buildin...

Yeah, I actually used OpenEmbedded, the predecessor to Yocto. The project was in 2011.

A few years ago I did an embedded Linux project on an ARM7 CPU without an MMU, a vehicle tracker for trucks with GPS, cellular modem, ODB2 interface, and other I/O.

Lack of MMU caused some unexpected issues:

* Because there is no MMU, there are no shared libraries. All the binaries have to be statically linked. We had a total of 2MB of ROM, of which about 1M was the Linux kernel. So we ran out of space for the programs we needed for the application about 3/4 of the way through the project. The solution was to use Lua, which was a 200k binary, and it could run scripts to do some things.

* Lack of MMU means no memory protection between programs. One program can step on another program's memory. Just like the good old days of DOS. Ouch!

* We started with the ucLinux distribution, which is pretty lame. A lot of programs in the tree didn't build. We ended up rebuilding with Yocto, and making patches to things where necessary.

* We found other entertaining bugs: https://www.cogini.com/blog/debugging-your-space-probe/

It's a bad market, probably the worst since 2000. Some of this is overhiring and zero interest rate behavior coming to an end. Companies are shifting from prioritizing growth to profitability. There is also a lot of asshole behavior. They are laying people off to please Wall Street, not because they need to for financial reasons. And they are doing layoffs in extremely callous ways, e.g., right before Christmas.

My daughter's data engineering team in finance was reduced from five to two, and they are not giving raises that match inflation. She is almost singlehandedly responsible for servicing million-dollar accounts. It's not like there isn't money.

There is also a lot of hiring of remote people outside the US. The companies figured out how to do remote work during Covid, and they are using it to reduce costs.

Things will probably improve in the next year, but it will be rough for a while.

Sure, you can make deadlocks in any language, but it's uncommon in Erlang. Shared state is the exception, and message passing means that things that manage state, such as gen_servers, only process one message at a time from their inbox.

Contrast this with languages like Java where every object is a potential concurrency problem. Or the 10+ years of trying to make Python async (see Twisted).

In some companies, staff positions can only be effectively done by someone who has been there for 10 years. They know the specific tech stack that the company uses, know the application, know where the bodies are buried, and everyone in the company trusts them. It's basically impossible for someone to parachute in and do the job.

Erlang is based on virtual threads (confusingly called processes). The Erlang virtual machine schedules them on OS threads. Erlang processes communicate using message passing, preventing deadlocks. You can use millions of Erlang processes without problems, e.g., to handle millions of Elixir LiveView sessions.

Like ogres

I had a client who spent $1M+ per year on Google Ads, and Google would fly them to California each year to meet with their account manager.

I don't know what the equivalent GCP spend would be to get similar treatment. Presumably higher. So that's a data point, at least.

I am an American who has worked with startups in Taiwan for 30 years. I am happy to talk.

If you need a company in Taiwan, it's generally best to have your main company outside and have a branch in Taiwan. It has significant tax benefits, particularly for foreigners.

A branch can repatriate profits to the parent company, after paying Taiwan tax. If you have a standalone Taiwan company, then you have to pay dividends, then pay personal tax on it. If you are not resident in Taiwan, then you would pay 20% tax on those dividends. There is also a 5% tax on retained earnings.

Another benefit is that if your company pays someone overseas for services, then Taiwan considers that "Taiwan source income", and they owe 20% tax on it. Other countries like Hong Kong would not tax that money.

As foreigner, if you leave, then you would find it difficult to run your company remotely. So you might as well set up somewhere else.

I built a mobile app to read data from blood glucose meters.

It used a magic cable that talked to the meters using RS-232 and converted the data into audio, plugging into the audio jack on the phone.

So, basically, it was a modem. The cable just had a little bump on it that did everything you used to do with a huge device.

The fundamental problem is that domain name trademark enforcement is handled by the legal department when it should be handled by marketing, business development, and/or DevRel.

While companies can enforce their trademarks, it's generally better to allow people who are using domains in a reasonable way to continue. Taking the domain away and then doing nothing with it doesn't benefit anyone.

Even in parasitical scenarios, allowing squatters to get affiliate revenue by redirecting traffic to the company is better than fooling around with the legal process.

I think Scrum is a transitional phase. It is a solution to the problem of having a dev team shared between multiple business functions.

The dev team keeps getting yanked around and interrupted. So every two weeks you get all the business people in a room and you have them make a list of things that the dev team will work on this cycle. Then the dev team can execute in peace. The Scrum Master's job is to protect the dev team. The Product Owner's job is to get the business people to converge on the prioritized list of work.

In a healthy organization, Scrum is much less useful. Scrum combines three different cycles: defining work, executing work, and reviewing work. In a flow-based process like Kanban, you can split them up.

The product manager builds a prioritized backlog of work, and the dev team estimates tasks to help do business ROI calculations. It's a collaboration between product and development.

The dev team implements tasks from the backlog one by one. You can batch up work to make a release, deliver incrementally, or use tools like feature flags. You evaluate whether features are effective working based on metrics.

I use distroless images based on Debian or Ubuntu, e.g., https://github.com/cogini/phoenix_container_example

The result is images the same size as Alpine, or smaller, without the incompatibilities. I think Alpine is a dead end.

When I was a kid I ran a lawn service, and I got all my engines out of the trash.

When the mowers are cheap, it's rational. They get the lawnmower out of the shed the first time in spring and it won't start, it would cost a significant amount to get it fixed, and they have to waste your time and wrestle a dirty lawnmower in and out of your car. So they buy a cheap new one and throw the old one away.

Paying someone is still more expensive than doing it yourself.

I wish that everyone in the US had affordable, convenient, and private access to a doctor who could help them choose contraceptives and ensure no side effects.

Unfortunately, that's not the case, so getting them over the counter is an improvement.

Linux from Scratch is educational but fundamentally impractical to use.

I recommend using embedded Linux instead, e.g. with Buildroot. You get an understanding of the fundamentals of the hardware, kernel, and build toolchain. And you can create small hardware systems that do fun things.

I ran Gentoo for a few years, and enjoyed it. Doing the install from stage1 gives you a similar understanding of how the pieces fit together and enables you to fix low level system problems, e.g. with disk failures. The community is also smart, as it self-selects for people who have skills and good attitude. Unfortunately Gentoo is too slow to install and quirky to deploy on servers.

Many of the leading Linux users are never going to pay for RHEL.

That includes expert users who don't need enterprise support. If you make them use something else, then that will become the most battle-tested enterprise distribution.

It also includes developers. If they run on your platform, then it will get native support. If you make it easy for them to test on your platform, then it will get first-class support. If you make it difficult and they feel like second-class citizens, you will get no support.

Open source is about community, and RedHat has made it clear that the only community they care about is companies willing to pay for expensive per-server licenses.

With the rise of the cloud and containerized workloads, the business model of providing long-term support for on-prem servers is declining, and they are just milking the customers who remain.

Thanks for your service supporting the Ansible community. Your books are also great, and I highly recommend them to anyone learning Ansible.

Ansible is still the best for tactical automation tasks. The syntax is also approachable for people who don't use it often, e.g. developers.

Ansible has poor performance on larger tasks. Idempotency checks to see whether an action needs to be executed require manual work to set up and time to execute. This also makes it slow to work with cloud resources.

I switched to Terraform years ago for low-level resource provisioning and cloud automation. I still occasionally use it in conjunction with Packer to create AMIs.

The shift to containerized application deployment makes Ansible much less relevant in general.

Dockerfiles are bad at idempotency, but the solution is probably not Ansible.

The problem is that RHEL is too expensive for people who don't need support. IBM is getting greedy and screwing up its pipeline.

Our dedicated servers are about $100/month for pretty serious hardware (2 x 8-core CPUs, 64 GB RAM, 2 x 8TB HDD, 30TB transfer). Paying $349/year is a significant percentage of that. It is worse for smaller servers, and ludicrous for small virtual machines in the cloud.

I would not mind throwing them a bone to have e.g. security updates at some reasonable cost. Their official repositories don't have enough packages, so I end up having to use 3rd-party repos for normal things, e.g. Postgres. Having a more full-featured repo of up-to-date software would be useful to me.

I have been running CentOS 7, but when that is no longer supported, I will switch to something else like Ubuntu or Debian, whatever our hosting providers support. I am already using Debian or distroless for containers.

And then RHEL will be completely irrelevant to me. Good job, IBM.

This is such a baffling failure to understand how social networks function (see the 1% rule: https://en.wikipedia.org/wiki/1%25_rule).

They are alienating the most engaged users who provide the content and pissing off the moderators who provide free service to keep the communities running. Replacing passionate moderators working for free with disinterested paid staff will reduce quality and significantly increase costs.

Postgres forks an OS process for each connection, which is relatively heavy weight. Oracle has a similar architecture to Postgres, and they solved it with a "multi-threaded listener". MySQL, in contrast, uses threads, which makes connections lighter weight.

It's fundamentally a question of how the connection listener communicates with the rest of the database, e.g., using shared memory or some other IPC mechanism, work queues, etc. Having too many connections results in problems with concurrent access and lock contention independent of how heavyweight the actual listening process is.

I run a product development consulting agency that works with entrepreneurs and early-stage companies.

For the past 8 years, we have mainly used Elixir. It is good for building straightforward web apps and APIs and great for more tricky things like chat. We also use Ruby on Rails, mostly for existing projects.

A lot of the initial work of building MVPs does not need to be fancy, it's just CRUD. Traditional frameworks are great for that. We also use React where it makes sense. The JavaScript world has a lot of accidental complexity that slows you down and doesn't add much value. Tools like Phoenix LiveView are great for adding fancy UI without overhead.

For mobile, products need a first-class user experience, and getting access to native platform functionality can be critical for some apps. Because of this, we have historically done native development. React Native is the first framework that is really competitive, but it still has its weaknesses. React Native may allow you to share code between platforms and use web people for native work, but it can result in somewhat generic apps. If you follow the "full stack teams, not developers" approach, then speed and cost of development of native is fine. After the initial development phase, the bulk of development work ends up being on the web.

A customer needed help with the size of their DataDog bill. A big part of that was log messages. They had poor logging discipline, so they started by filtering out all logs that were INFO or lower priority. But then the developers couldn't see what was happening in production, and they started logging messages at WARNING and ERROR in production. That made the output of unit tests in CI light up like a Christmas tree, and the output was unusable. Good times.

The solution was to carefully audit all log messages, reducing the number of messages, making sure they were at the right level, and replacing log messages with trace attributes. We also implemented trace sampling to avoid paying for redundant success messages. That reduced the bill by 80%, saving tens of thousands of dollars a month.