HN user
building promptless.ai
Actually, you might find this interesting: we wrote a blog post a while back about how we gave our agent a way to voice its suffering. The first example is from the screenshot updating subagent, where it got stuck after 84 browser interactions trying to log into a customer's app :'(
Funnily enough, one of our early users (a solo technical writer) told us that the biggest value he got from Promptless was just finding out when the engineering team forgot to tell him about a change that would impact docs. The drafts from Promptless were just the cherry on top for him.
So yes, there's a workflow where you really just use Promptless for the notifications. That being said, the drafting is now quite good, so it's probably at least a good starting point to get past writer's block!
Yep. You give Promptless access to a test account and it navigates and interacts with your product to capture new/updated screenshots.
Here's an example from our own docs, where you can see the captured screenshots if you scroll down to the PNGs that Promptless added https://github.com/Promptless/docs/pull/233/changes
Getting the browser navigation to work was predictably hard, but the other thing that's very hard to get right is timing when Promptless takes the screenshots. Some teams want Promptless capturing screenshots on a staging environment, and sometimes it's on prod, and different teams have different release cycles and feature flagging processes.
For that reason, we still have to do a bit of manual set-up for each user that wants screenshot updating.
Agreed though that just asking nano banana to predict a new screenshot is negative value.
tl;dr. I'm one of the founders of Promptless, the AI agent that automatically updates docs. 12 months ago, we decided to give our agent a tool to let it "scream" when it was struggling, and we wanted to share our learnings and some cool examples of issues that it caught.
AMA!
A (charitable) interpretation of this is that the model understands "stuff that would embarrass Anthropic" to just be code for "bad/unhelpful/offensive behavior".
e.g. guiding against behavior to "write highly discriminatory jokes or playact as a controversial figure in a way that could be hurtful and lead to public embarrassment for Anthropic"
Promptless tries to check its generated docs against other sources to identify errors, but there are also some very cool documentation testing tools that we might integrate in the near future!
Happy to clarify! There are two main ways people are using the Slack integration: (1) You click the Promptless Slack message action (video below) on a thread that might warrant a doc update—Slack Connect channels are an extremely common use-case for this, and (2) You tag @Promptless in a DM or a channel with a brief instruction—people sometimes use this for simple requests, but after signing up for Promptless, you can also send Promptless a link to an old PR, and it'll use the Github API to read the content of the PR and draft an update.
Here's a video we made a month or so ago that shows how (1) works, hope that helps! https://www.tella.tv/video/cm5x8uj9f001d0al11vyxcnsc/view
Forgot to post: link to try Promptless out right now is https://gopromptless.ai/hn
Thank you!
Yeah, even with lots of full-time resources, companies' documentation regularly fall out of date. It's particularly ubiquitous in AI just because of how fast the general space is changing.
Thank you!
We're still seeing new customers use Promptless in new ways (hence pricing can be a bit bespoke sometimes).
But for our standard use-case, the number of docs pages (crude proxy for documentation surface area) ends up being the primary metric separating pricing tiers. We're in the process of standardizing a startup tier that probably fits most docs sites that are ~100 pages or fewer and <20 employees.
Totally. My cofounder and I are very excited about this idea.
Sort of a primitive version of this, but one Promptless user is using it to update in-app tutorials for various integrations they have, in addition to public docs. The tutorials in this case are just text, but there's a path from having that context to actually having an in-app agent that provides a product walkthrough tailored for the logged-in user's unique use-case.
Yes, Promptless already has the ability to be triggered from Slack channels (e.g. Slack Connect support channels with customers). Discord likely coming later this week or early next week!
Happy to get you set up on either!
Yeah, this is very valid. We opened up a self-service trial this week so that people can more easily give it a spin before deciding to join a paid plan, but we're still working on a lot of the customer experience things to offer a truly complete onboarding experience. Appreciate the feedback!
Yeah, great question! The earliest quantitative measurements people make are around how much time is being saved, but people are also measuring deflection rates in AI support chatbots (better docs => better support), which is strongly correlated with CSAT scores. Measuring reduction in churn or increased inbound takes several months to prove out, and we just haven't been around that long :)
We'll definitely be posting some case studies in the coming weeks, though!
Exactly. I think lots of people tend to underestimate how much documentation is out there. When I started working on Promptless, I assumed it was dominated by things like API docs and other developer docs. Turns out that's just a small fraction of software docs in general (they call them 'help centers' or 'knowledge bases' in this audience). And software docs in general is a fraction of docs that are published for other industries.
Very cool. Will reach out soon!
Love this! Those people are definite force multipliers, but the fact that they're rare can end up putting a lot of burden on their shoulders. And yes, I definitely overgeneralized—while it's a running joke that developers hate writing docs, definitely not universally true!
Yes, absolutely! Someone mentioned a similar situation where they had 100+ integration-specific landing pages in webflow (mostly for SEO) that became stale within months.
Definitely would love to pick your brain more about this if interested. prithvi@gopromptless.ai
Hmm, are you thinking about this in the context of, say, code documentation in Github repos, like README.md files? Or are there types of internal docs that you typically see in markdown?
If you're asking about when there are new commits on the source code after a PR is opened, yes that's what happens! It just updates the existing docs branch.
If you're asking if Promptless should just add doc updates to the same PR as the code PR, that's definitely an option, but people tend to just want them separate both because it fits better with their workflows and it's less contingent on CI/CD processes, if that makes sense.
For sure. It can be challenging to inculcate a culture of treating docs as a product, but since great docs can drive revenue growth and bad docs can increase churn, it's a very important mindset.
In terms of creating guides/examples for third-party tools, do you have a particular use-case in mind? e.g. if you're something like Zapier with hundreds of connections?
Will do Greg!
Thanks! Some of our early customers use Mintlify (we're big fans), which is an easy integration since there's always a Github sync. Let's chat soon!
Awesome!
Ha, I might also be weird. But yes it just opens PRs for doc updates, which still need to be reviewed. Maybe eventually people will want Promptless to be committing doc updates directly, but I still think human oversight is important for high-quality docs, both to prevent untruths but also to guide good information architecture.
Absolutely. There are steps in Promptless's agent flow that are designed to prevent this, but this is why users still review Promptless's suggestions to guides before committing/publishing them. I think people will still want to review Promptless's suggestions for a while, but the granularity of oversight will probably decrease as trust increases.
Great questions! It is possible to get Promptless to run only when PRs are merged, or when commits are made on `main`, but pretty much everyone wants the doc updates to be drafted when PRs are opened, because they'd like to review the doc updates in parallel (i.e. before it's actually released). If two PRs are created at the same time, Promptless will review them separately (and potentially create two docs PRs, if both have customer-facing impact).
Honestly, some of these workflow points are areas that we're probably going to adjust and add more configuration around. For example, some folks with very high commit velocity are asking for a "daily digest" docs PR from Promptless instead of individual docs PRs.
Oops. Fixed. Thank you!