HN user
Co-founder of Memfault, a firmware delivery and monitoring solution for hardware products.
My background is in firmware development and 10-year goal is to modernize embedded software development.
The most important part, no matter if it's email, Nextdoor, WhatsApp, is that we can use it to disperse information quickly, and then get off the platform and meet up in person. Our stoop coffees are broadcasted via WhatsApp, and then some people tell their immediate neighbors via text, and then people show up.
You say the 'watch parties' are too much, but these are actually the events that are the best. They are at 8p, people come over when their kids are in bed, we have 4-10 people show up, everyone's in their sweat pants, and we watch a TV episode. We discuss it for 10-15m afterwards, and we generally part ways.
It's such a low-effort and small event, and it allows people to get into other people's homes in a low-judgement way. It's been one of the more successful events at getting neighbors to become friends with each other.
Good insights here. I'm going to steel a couple of notes here and update the post if you don't mind!
Author of the post here - would love to hear about your trials and tribulations of building battery powered devices.
<3 Me and a few others created this course in 2014. Glad you found it useful! I got so much joy out of the content creating the content for the lectures and labs.
Hope you liked the course! I was the one who first made and taught the course. To my knowledge, it’s still a required course for all CS freshman and is still taught by students.
You speak of such truths :D. It's a seriously under-documented knowledge gap on the Internet. There are an infinite number of Cortex-M4 MCU's but only a few I would recommend hardware companies actually use, primarily due to power consumption of production work-loads (sleeping and deep sleep) and the software packages that are bundled, both of which are generally not assessed before choosing hardware.
It’s primarily about the number of hours. At Pebble, we’d be able to get a pretty good gauge on battery life with about 10-20 devices and about 24 hours worth of data. We recorded the battery’s percentage drop over the course of each hour, averaged all samples together, and then used that to estimate how long the battery charge would last.
This strategy is covered here: https://interrupt.memfault.com/blog/device-heartbeat-metrics...
Of course it’s better with thousands of devices, but we were always surprised how accurate it was (as long as we had a few Android users in the office - it reliably got worse battery life due to worse Bluetooth performance)
These two commands ultimately do different things. `stg publish` was in the business of not force pushing and only committing the 'delta' between what lived on a branch and the currently applied patches, so that you can `git push [no -f]` to a branch.
Definitely cumbersome to use, but it had fair reasons to be cumbersome.
Preface: I love Stacked Git and have convinced many co-workers to use it and they all love it. It's really great, especially when used with Phabricator!
---
A question for the maintainers...I was enjoying the command `stg publish` to allow myself to keep refreshing patches locally but every so often push to a branch that others non-stg users were pulling and pushing to. The command was removed in 1.0.
I'm curious why it was deprecated and removed and if there is a good replacement flow for working with non-stg users on branches I can't force push to.
Oh you sweet summer child
Should have clarified. On the embedded systems I work on, generally Cortex-M4, you set an MPU region on address 0x0 and it turns into a hardfault, so it's pretty easy to catch.
Author here. Firmware, for better and worse, is stuck in C/C++ land, so many of the topics here are essential to keep a complex, multi-MCU, multi-board setup in working order.
I'm actually really curious what you all do in C/C++ to prevent bad operations from ever being performed in the first place.
For example, should we just change our internal malloc / free to use double pointers instead?
bool malloc(size_t n, void **ptr) {
*ptr = <memory>
}
void free(void **ptr) {
... <free>
*ptr = NULL
}
Author here! I don't think we are in disagreement, just mostly in different interpretations. You are talking about a car, with likely 200-300 embedded systems backed into it.
If the light sensor or the braking module goes into a completely unexpected state where it can no longer be communicated with, it has corruption, etc, you need to do something about it.
The solution isn't to reboot the car, it's to reboot the individual module and quickly get it back into working order. Rebooting the braking or lighting module should take on the order of milliseconds (if done correctly), and that is much, much better than having a piece of software in a corrupted state for seconds/minutes/hours.
In a car, there is a mothership board that is orchestrating everything, pinging every module, and ensuring everything is in working order. Just like a kubernetes setup, if any one piece fails to ping or looks off, just reboot it and get it working ASAP.
Memfault (YC W19) | Senior Full Stack Engineer, Senior Backend Engineer | San Francisco CA, Boston MA, Berlin Germany | Full Time
At Memfault, we're helping companies that build their own hardware automatically catch, triage, and fix issues in the field.
Unlike mobile or web developers, hardware engineers today do not find out something is wrong with their device until their customers start calling (or worse, tweeting). Memfault gives companies the initiative by providing a real-time view of errors their devices are encountering in the field, and infrastructure to deploy targeted fixes.
We're mostly ex-Fitbit, Pebble, and Oculus engineers. We know the industry well, and we are excited to solve the problems that we have faced endlessly for the last 10 years.
Open Roles:
* Full-stack Software Engineer (React, TypeScript)
* Backend Engineer (Python, PostgreSQL, AWS)
* Marketing Manager
* Business Development Manager
* Customer Success Manager
* Sales Development Representative
Launch HN for more context: https://news.ycombinator.com/item?id=20801512
Angel Jobs Page: https://angel.co/company/memfault-inc
If you're a web developer who happens to have a passion for or wants to learn more about embedded systems and hardware, we'd love to hear from you.
Feel free to reach me directly at tyler(at)memfault(dot)com
I find it most similar to bubble.io, which includes databases but also allows linking to external data sources like Airtable.
Project/Article turned out very well. It was more of a proof of concept with the code open-sourced so people can have something to start with to track firmware binary sizes on a commit-by-commit basis.
https://interrupt.memfault.com/blog/code-size-deltas
Unsure if their minds were blown, but I at least know that mine would have been if I showed it to myself a couple of years ago. During that time, I had never touched databases, migration files, or devops in general. Now that's changed, but I still try put myself in those shoes.
Very cool! I was looking for something like this about a month ago when I was building an example web application for firmware developers. Firmware engineers aren't expected to know anything about databases and usually don't, so I just wanted some dumb JSON store in the cloud that didn't need any provisioning.
In the end, I went with Heroku and it's included PostgreSQL offering and stomached the complexity, but along the way I found https://jsonbox.io/, which I thought was neat and seems very similar.
Thanks for the link! That’s got a lot of goodies.
One of us from Memfault also wrote a post about tips with Binutils, and this approach is covered in it.
Excited to finally get this out after sitting on it for a year. Might seem hacky at first but it works similarly to plugin managers like oh-my-zsh or vim-plug where they require edits to the users dotfiles. However, the beautiful thing is we can take advantage of PyPi and Python to handle the heavy lifting of the plugins.
JSYK: the end result of the Buildpacks project is a Docker image. It's just that if someone wants to package up NGINX inside of a container alongside their app, they can just add the buildpack and an nginx.conf rather than going through the complexity of installing and configuring it themselves.
Conveniently timed post! I was just looking at buildpacks.io over the weekend. I was not aware there was a shim to conver Heroku buildpacks to the v3 spec. Neat.
stacker.app has impressed me. It's by the same team that built (and recently posted) Toga (https://news.ycombinator.com/item?id=22746663).
they think of it like removing some characters in a sentence on a word document
Too real. In the past, the way we usually went about saving space when we needed it was removing small animations, shortening log messages (eventually encoding/hashing them), compressing icons that were stored in the firmware, and ensuring large functions don't get inlined.
LTO was also an option, but wow, does it slow down build times and make the development cycle a pain.
OP here. I've worked at two embedded software shops, and every time we are working on a new product and cramming to get it shipped, we run out of code space. Each of those times, PR's stay open for 2-3 weeks waiting to be merged until engineers can work their magic and reduce the size.
I've even seen instances where deals take place. We'll clean up our module today and you can merge your 1k feature, but we'll need 1k in 2 weeks and you need to find it for us by then.
Software engineers usually laugh when I tell them about my struggles with code space and memory. I do enjoy it though.
I’m also intrigued by Qt for MCU’s. It looks very promising and the toolset to compliment the library is likely better than anything else for the embedded space.
I really wish we had open-sourced the UI framework we built for the Pebble firmware.
The one common paradigm of Makefiles I'm enjoying (and using myself) is the pattern for enabling more verbose logging.
ifeq ($(V),1)
Q :=
else
Q := @
endif
somerule:
$(Q)$(CC) -c $^ -o $@
$ make V=1
It's mind-boggling to me that many projects I've heard about and worked on treat (hardware) watchdogs issues as impossible to debug, for the exact reason you mentioned...they didn't set up a software watchdog.
Doing some basic groundwork and implementing the timer/software watchdog, writing out to logs when a watchdog is about to trigger, and not disabling the watchdog during development are all basic things teams can do to more easily catch these issues before shipping firmware.
JSYK: I do believe this was mentioned in the post as well, under the section 'Adding a “Software” Watchdog', but when a little bit further and implemented a per-task watchdog.
Completely agree. This is definitely one of the perks of the new Rust push...they have a package manager from day where as C is still struggling to have any reasonable solution after 20 years.
Precompiled headers are definitely a hack to a problem that shouldn't be necessary.
Feel free to email me at tyler at Memfault dot com.
We have a list of contractors we’ve enjoyed working with. I can try to help pair you up based on the project description.
We are taking it very seriously ;)
Happy to hear that an you thought to build this translation service even for early development! It's usually an after thought at the companies we've talked to, and an expensive one too.