diamondap

It is extremely mentally demanding to do both, even for someone with an overactive brain. I reached a point of severe burnout in 2025 and cut back to part-time work. I'll have to ramp up my hours again soon because, contrary to one of the other comments, I didn't get rich during 1990s/2000s boom.

As for how I did it, I would wake up very early and put in a few hours of writing, then I would put in a full day's work at the office. After my first twelve years or so in the industry, coding was almost second nature and came quite easily. Also, I do much of my writing and programming in my head while I'm walking, running or biking. (I bike to work and back every day.) By the time I sit down to write or code, I already know what I'm going to write, so it's just a matter of getting it through the keyboard.

But yeah, do both for a few years and it really taxes your brain. In both novel writing and programming, you're carrying an entire world in your brain: the plot and characters are one world, the architecture of the software you're working on is another. By 2025, I truly, literally felt like I needed to be shipped off to a madhouse in the countryside. A hundred years ago, people used to call that "the rest cure for a nervous breakdown."

Anyway, cutting back to part-time work has been a huge help. Sanity restored.

Ha! I started my career in a company where I had to pump out custom web apps as fast as humanly possible. The apps were successful, and then I had to maintain them for several years. That's where the real learning occurred. I learned the cost of every one of my shortcuts and poor design decisions the hard way.

Many years ago, some famous developer said, "Always write your code as if the person who's going to maintain it is a violent psychopath who knows where you live." As I fixed my poor design choices one by one over endless late nights, I sometimes felt the anger of a violent psychopath toward the former, ignorant me who had stupidly plagued current me with all these problems.

When you learn the hard way, you know exactly why good design decisions are considered good. In later jobs, one of my fundamental goals for every new project was "I never want me or anyone else to have to answer a 3 a.m. call about why this system isn't working."

I tried make my PDFs X-1a compliant with WeasyPrint, then ran them through Adobe's PDF/X validator and they kept failing. I was in a bit of a hurry and found a way to do it with ghostscript. I would like to remove ghostscript from the mix, so when I have some time, I may try again to do it all with WeasyPrint.

Kudos to you for doing that.

I've been publishing print and ebooks since 2015, and I can attest to the fact the Word to PDF X-1/a to epub/kindle pipeline is painful. Making minor edits after publication is also painful, as the author notes, and can be error prone if you fail to make identical changes to all formats.

The problem was bad enough that I built by own markdown to HTML to PDF/X-1a processor using Python, WeasyPrint, and ghostscript. This also allows me to use git for version control, and I can make formatting changes using vanilla CSS. My tools are currently too crude for the average non-tech writer to use, but they save me hours every time I use them.

For any of you hackers out there looking for an untapped market, try making a user-friendly tool that converts Word, PDF and/or similar formats to the print-ready PDF/X-1a, PDF/X-3 and PDF/X-4 formats. At the moment, all the existing tools are proprietary and expensive, and many are difficult to use. This won't be a big money maker, but it will certainly be welcome by many indie authors.

Humans are objectively quite cheap.

I disagree with that statement when it comes to software developers. They are actually quite expensive. The typically enter the workforce with 16 years of education (assuming they have a college degree), and may also have a family and a mortgage. They have relatively high salaries, plus health insurance, and they can't work when they're sleeping, sick or on vacation.

I once worked for a software consultancy where the owner said, "The worst thing about owning this kind of company is that all my capital walks out the door at six p.m."

AI won't do that. It'll work round the clock if you pay for it.

We do still need a human in the loop with AI. In part, that's to check and verify its work. In part, it's so the corporate overlords have someone to fire when things go wrong. From the looks of things right now, AI will never be "responsible" for its own work.

I think AI will substantially thin out the ranks of programmers over the next five years or so. I've been very impressed with Claude 4.5 and have been using it daily at work. It tends to produce very good, clean, well-documented code and tests.

It does still need an experienced human to review its work, and I do regularly find issues with its output that only a mid-level or senior developer would notice. For example, I saw it write several Python methods this week that, when called simultaneously, would lead to deadlock in an external SQL database. I happen to know these methods WILL be called simultaneously, so I was able to fix the issue.

In existing large code bases that talk to many external systems and have poorly documented, esoteric business rules, I think Claude and other AIs will need supervision from an experienced developer for at least the next few years. Part of the reason for that is that many organizations simply don't capture all requirements in a way that AI can understand. Some business rules are locked up in long email threads or water cooler conversations that AI can't access.

But, yeah, Claude is already acting like a team of junior/mid-level developers for me. Because developers are highly paid, offloading their work to a machine can be hugely profitable for employers. Perhaps, over the next few years, developers will become like sys admins, for whom the machines do most of the meaningful work and the sys admin's job is to provision, troubleshoot and babysit them.

I'm getting near the end of my career, so I'm not too concerned about losing work in the years to come. What does concern me is the loss of knowledge that will come with the move to AI-driven coding. Maybe in ten years we will still need humans to babysit AI's most complicated programming work, but how many humans will there be ten years from now with the kind of deep, extensive experience that senior devs have today? How many developers will have manually provisioned and configured a server, set up and tuned a SQL database, debugged sneaky race conditions, worked out the kinks that arise between the dozens of systems that a single application must interact with?

We already see that posts to Stack Overflow have plummeted since programmers can simply ask ChatGPT or Claude how to solve a complex SQL problem or write a tricky regular expression. The AIs used to feed on Stack Overflow for answers. What will they feed on in the future? What human will have worked out the tricky problems that AI hasn't been asked to solve?

I read a few years ago that the US Navy convinced Congress to fund the construction of an aircraft carrier that the Navy didn't even need. The Navy's argument was that it took our country about eighty years to learn how to build world-class carriers. If we went an entire generation without building a new carrier, much or all of that knowledge would be lost.

The Navy was far-sighted in that decision. Tech companies are not nearly so forward thinking. AI will save them money on development in the short run, but in the long run, what will they do when new, hard-to-solve problems arise? A huge part of software engineering lies in defining the problem to be solved. What happens when we have no one left capable of defining the problems, or of hammering out solutions that have not been tried before?

Non-paywalled version: https://archive.is/AkgMN

OK, maybe it isn't capitalism but US society that implies that promise. Many (insensitive) people respond to tales of economic woe by saying, "Get a job!" As if that's going to solve even your basic money woes. As others in this discussion have noted, many people employed full time still struggle to provide the basics. That part of the social contract just isn't holding up for a lot of people. And that's a recipe for social unrest.

There are more people struggling these days than the mainstream news reports. The lines at the food banks where I live are much longer than they were before COVID. One of the local food banks says they're distributing three times as many meals per month as they served before the pandemic. And this is an area of relatively high employment.

People working service jobs simply can't afford the basics, and that's a problem. Part of capitalism's implied promise is that if you work full time, you should be able to feed and house yourself. But for huge numbers of people, that doesn't seem to be true anymore.

My all-time favorite flippant description of a programming language: someone once described Java as "a domain-specific language for converting XML files to stack traces."

I love this part:

A workman could check the stone size himself by seeing if the stone would fit into his mouth.

They're probably laying down over a million stones per mile. Do all of them have to pass through a workman's mouth?

"And if yes, who cares?"

One of my managers used to say that the only reason people write software is because there isn't already an emacs or Excel macro to do what they need.

After 25 years of coding primarily in emacs, I switched mainly to VS Code for larger, more complicated projects last year. I did it mainly because it eases navigation through large code bases and because the visual debugger is so easy to use.

I still use emacs in the terminal for scripting and for smaller coding projects, for anything over SSH, and for blogging and fiction writing. It's a distraction-free editor that has very powerful features when you need them, but doesn't shove them in your face. It's also nice to be able to do everything with the keyboard and not have to keep reaching for the mouse.

A few years ago, I talked to three retired federal law enforcement agents who had each spent 20+ years fighting the cartels that were bringing drugs into the US from Mexico and South America. All three felt betrayed by the US government. They said, essentially, "We put our lives on the line many times to try to stop the flow of drugs. Then the government turned around and basically gave the pharmaceuticals license to sell this stuff professionally, through people's personal physicians."

These retired agents didn't see the drug war as a war on "bad guys," but as an effort to stop the destruction drugs wrought in people's lives. After all their work, their own government undermined their efforts.

My own doctor was taken in by Big Pharma's sales pitch and wound up going to prison for over-prescribing their pills. https://en.wikipedia.org/wiki/William_Hurwitz. Though to hear Big Pharma tell it, there was no such thing as over-prescribing.

If you want to get an idea of how out-of-hand the prescription frenzy got, take a look at John Temple's book American Pain, which describes the pill mills in South Florida. Towns in Appalachia used to send charter buses to these pharmacies. After a 12-20 hour bus ride, each passenger would pick up hundreds or thousands of pills, then ride home to sell them in their small country towns.

Some of the pill mills, which were fully licensed by the state of Florida, were cash-only and would haul garbage bags full of money to the bank each day at closing. For a summary of Temple's book, see https://adiamond.me/2020/01/american-pain-by-john-temple/

Actually, one software-developer-turned-writer did something close. Check out Chris Fox's book "Write to Market" on Amazon. He sat down and analyzed sales figures for different book genres on Amazon then chose a genre where he thought he could squeeze onto the top ten bestseller list. He's been cranking out books for years and selling pretty well. If you check his author page, you'll see he has quite a few titles out.

Basically any company offering special services that work with very large data sets. That could be a consumer backup system like Carbonite or a bulk photo processing service. In either case, legal agreements with customers are key, because you ultimately don't control the storage system on which your business and their data depend.

I work for a non-profit doing digital preservation for a number of universities in the US. We store huge amounts of data in S3, Glacier and Wasabi, and provide services and workflows to help depositors comply with legal requirements, access controls, provable data integrity, archival best practices, etc.

There are some for-profits in this space as well. It's not a huge or highly profitable space, but I do think there are other business opportunities out there where organizations want to store geographically distributed copies of their data (for safety) and run that data through processing pipelines.

The trick, of course, is to identify which organizations have a similar set of needs and then build that. In our case, we've spent a lot of time working around data access costs, and there are some cases where we just can't avoid them. They can really be considerable when you're working with large data sets, and if you can solve the problem of data transfer costs from the get-go, you'll be way ahead of many existing services built on S3 and Glacier.

Only the affluent can learn these things because ordinary people have real problems to worry about.

I can't say I agree with all the author writes, but that statement is spot on.

Interesting. Thank you! And the about page has some useful links to other projects. https://www.marginalia.nu/marginalia-search/about/

One of the great lines of 18th century theater comes from Richard Brinsley Sheridan's "The School for Scandal." Sir Peter, alarmed at his wife's extravagant spending, asks her why she buys such expensive things. She says, "Because I have taste." To which he replies, "You had no taste when you married me."

Thanks for posting that. That's pretty cool.

To all you folks complaining about the quality of POD books, you should know that the book designer has a lot to do with the book's readability. Many self-published authors who design their own interiors choose bad fonts out of ignorance, or make the font too small to reduce page count and cost. Print design is really an art. The big publishers take it seriously, and some self-published authors do as well.

As for blurry covers, a common mistake among self-published authors is not preserving the aspect ratio for cover art. They'll stretch an image, or use a low-resolution image. Again, this is done out of ignorance or carelessness. Major publishers know to pay attention to these details.

POD publishers like Ingram Spark also let you choose the type of paper and cover finish. They can produce good paperbacks. Maybe Amazon's POD system chooses cheaper paper by default. I'm not sure.

Ha! I remember slogging through that as an undergrad. Afterwards, I was hoping for something less dense and arcane. But the next book was Heidegger's Being and Time.

I think this is right. Some book stores in the US even have print-on-demand machines inside the shop. See https://www.ondemandbooks.com/ for more info about the printers.

Most publishers make good money on their backlists, with a slow trickle of sales across a huge number of titles. Amazon and other stores don't always want to stock these titles because they take up a lot of shelf space.

Publishers may not want to put them into book stores because they may be returned if unsold, and the return process is costly. So print-on-demand (POD) makes sense for both parties.

Amazon has been running a pretty big POD operation for years for its Kindle Direct Publishing arm, so they have the infrastructure to print books for others as well. And yes, depending on the machine, the covers may not look quite as nice as the covers from offset printers.

I second this advice. I've been in your position, and I can tell you that that kind of management simply doesn't care about the talent beneath them. If you leave, they'll replace you with someone else and treat them the same way. Do not work where you can't be effective.

Other good employees, including many of those you are currently managing, will likely follow suit over time. And management won't really notice or care when a first-rate team walks and is replaced by a second-rate team. Management will treat them all the same.

That said, I have seen good team leaders move to greener pastures and convince some their better developers, over time, to join the new team. This happens in almost all industries.

Don't let your mental health suffer. If you care more than the decision-makers above you about what you're doing, you can't win. Go somewhere that will let you do your best work.

BTW, when I was in your position, I hesitated a long time before making my move. It got so bad, I thought I didn't even want to work in the tech industry anymore. When I finally did make the move, the difference was night and day. Work was enjoyable again, and I was energized and full of motivation.

In case you're interested, here's a brief account of my situation and job change, from long ago: https://sheprador.com/2022/04/worst-manager-ever/

I love the succinct summation in the last line of the review: "if this is the output we can expect from one of our bright young things, we’re fucked."

I wonder if the author could boost sales by quoting that on the cover of his book.

If your management has been letting that wasteful situation go on for a long time, there's not much you can do. It sounds like there's some dysfunction in management. I've seen that in many businesses. Unfortunately, the money they're wasting on maintaining two environments is probably preventing the company from moving forward in other ways.

That said, a number of companies have left the cloud for cost reasons. 37 Signals is a recent high-profile example: https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47...

I've read of others leaving for cost reasons as well. Many run niche workloads that don't make sense in the cloud. One company, if I recall correctly, was doing scientific processing that required massive, but not unlimited, CPU and GPU resources. They found it was cheaper to buy a bunch of high-end workstations and run their jobs locally. They didn't need the cloud's 24/7 uptime guarantees, because they were doing offline processing with no public-facing apps.

Pick this apart, please!

Interesting that no one has commented on the central point of the article, that living entirely in your own head for too long is just plain unhealthy. This problem has been around for a long time. In Gulliver's Travels, the Laputans spend so much time thinking, they have servants called flappers who periodically rattle them into awareness of the actual physical world around them.

Our minds can and will go anywhere. Sometimes we need to be reminded of our physical, social, time-bound being, of its limits and where to focus our attention. That's called "being grounded."

I read an interview a few years ago where a reporter asked a writer what was the difference between authors who made it and authors who didn't. The writer said that the ones who made it stuck with it. They just kept on writing despite years and years without strong sales or recognition.

If you're going to stick with it, there has to be something there beyond a desire for sales or widespread praise.

Success, too, has its downsides. My wife is a painter--and if you think it's hard to make a living at writing, try painting. She met another painter who makes a huge income, but that painter complained, "My first sales were paintings of boats. And for years now, everyone comes to me wanting a boat. I'm bored to death of them. I want to paint other subjects, but this is what pays, so I keep doing it, as frustrating as it is."

Look into fiction writing, and you'll find few writers whose publishers will let them write outside their genres. Those writers may want to explore, but the publishers know what has sold in the past, and they want more of that. The only major author I can think of who sells big in multiple genres in Stephen King.

Self-publishing has been a godsend for those who like to explore. You can write anything you want and push it out at will. You might not make money at it, but you can do it.

At bottom, you have to choose your definition of success and failure. If you define success as being accepted by editors and critics, receiving praise and prizes from the publishing community and hitting the bestseller lists, all of that is beyond your control. It's like fishing for likes on Instagram.

If you define success as getting better at your craft, writing something that you and the few people you really respect like, then you can find fulfillment. I often think of gardeners who pour hours into their roses, not to get rich or to receive recognition, but just because it pleases them and fulfills a deep inner appreciation.

jasode's answer is spot on. At work, we run an digital preservation system that ingests large file sets into S3, Glacier and Wasabi. The files have through a pipeline of microservices to verify integrity, identify file formats, and extract other metadata.

We use AWS Elastic Container Service (ECS) to ensure all of the services are running and to scale when necessary. We use NSQ to make sure tasks are sent and re-sent to workers until they complete the entire pipeline. And we use Redis to store interim processing data from each worker.

Keeping that interim data (state) in a place where all workers can access it is key. ECS kills workers when it wants, and workers occasionally die due to out-of-memory exceptions. With the state info stored in Redis, a new worker can pick up any failed task where the last worker left off.

This system has been running well in production under heavy load. It replaces an older server-based system that used different technologies to handle the same responsibilities: supervisord to keep processes running and BoltDB to store interim processing data. That system worked, but could not scale horizontally because BoltDB is a local, disk-based store. Distributed workers need a shared, network-accessible store to share state info.

You'll find a detailed overview with diagrams of the new system at https://aptrust.github.io/preserv-docs/overview/

There's a shorter write-up of the goals and how we achieved them at https://sheprador.com/2022/12/architecting-for-the-cloud/

This stuff isn't too hard, as long as you get the pieces right. Try to stick to fredley's advice. Keep things simple and you'll save yourself many headaches going forward. Also, be sure your workers handle SIGTERM and SIGKILL explicitly, cleaning up their work as much as possible before they die.