Devs need system design tools, not diagramming tools

https://thenewstack.io/devs-need-system-design-tools-not-diagramming-tools/

by argoeris • 2 years ago

215 99 2 years ago

gtirloni• 2 years ago

I read the whole article and it has great points. Almost nodded the whole time. However, I couldn't identify whether an alternative exists today.

anotherhue• 2 years ago

Maybe checkout https://c4model.com/

imglorp• 2 years ago

It sounded like it was working up to a C4 pitch but never got there. It bears a look exactly for this post's title.

https://c4model.com

Veuxdo3• 2 years ago

Almost all of his complaints stem from using generic drag-and-drop diagramming tools. A modern take on system diagramming, like Ilograph[0], solves most of these issues (IMBO).

[0] https://www.ilograph.com

qznc4• 2 years ago

I believe the answer is to have a single model of everything.

You can build the model from whatever: Draw it with a GUI, query your server infrastructure, analyze binaries, collect it from databases, parse it from documentation, process your issues, etc. The important points is to merge all the information into a single model.

You cannot represent this model on a screen in any meaningful way. Instead you can only get certain "views" into it to understand an aspect or answer specific questions.

Does it exist today? Many tools and processes have been developed to achieve it. For example, "Model-based systems engineering" is a very formal approach. I have yet to see it realized in practice though.

I'm pretty sure it isn't a single tool or process though. By definition it integrates with nearly everything and that means a lot of customization. You won't get it get it off the shelf.

Veuxdo3• 2 years ago

You cannot represent this model on a screen in any meaningful way. Instead you can only get certain "views" into it to understand an aspect or answer specific questions.

It sounds like you're describing: https://www.ilograph.com/blog/posts/concrete-diagramming-mod...

AnimalMuppet• 2 years ago

The problem with "a single model of everything" is that it is an inferior tool.

It's a drawing or diagramming tool, but the stand-alone diagramming tools are better.

It's a text composition tool, but word processors are better.

It's a code writing tool, but IDEs are better.

In every single thing it tries to do, a specialized tool is better. So this single tool needs to be close enough to the best in every category in order to not be a boat anchor holding you back. So far, nothing has come close.

riiii• 2 years ago

UML had the idea. But instead of focusing on the visual aspect they went bananas on trying to turn it into some sort of a programming language.

Also the design tools for it were hilariously expensive and notoriously cumbersome.

kitd• 2 years ago

Does it exist today?

Not sure, but IMO a dataflow diagram from good ol' SSADM got a lot of the way there. Unlike a lot of the modern techniques which model (groups of) deployed components, DFDs were strictly logical and functional and included data stores and users as well as functional components. So it was possible to model data in flow and at rest, and the characteristics of each, including where it crossed system boundaries.

IMO this was the best diagram format to get an overall view of the system, or reveal which areas needed further analysis.

9dev2• 2 years ago

This is something I that’s been living in my head for a few years now—and no, it does not exist yet. I’m also convinced the only way to get it right is to have a single graph, modelling the entire system, and apply filters on the graph to get only the nodes you’re currently interested in, then have different diagrams produced from that as the output. How else would you describe the flow of packages from the internet through the firewall, what the logical network looks like, and which physical location things are located at? These questions all interfere with each other on a conceptual level, yet are all conceivable using different attributes of the same connected graph nodes.

It’s very complex, very interesting, and lots of work.

yoelhacks• 2 years ago

It sounds to me like what you're describing is a single *tool*, not a single model. Is that possible?

I agree that we need multiple views, but it isn't just a matter of filtering part of a single view - it's also different fundamental primitives that are needed to communicate different kinds of ideas or aspects of a given system.

To me, this seems parallel to the evolution of LLMs into multi-modality by default. Different queries will need different combinations of text, heat maps, highlights, arrows, code snippets, flow charts, architecture diagrams, etc.

It's certainly not easy, but it's exciting to consider :)

vajenetehais• 2 years ago

There is Capella which is a tool (software) and method to design systems. It used the Arcadia method (mbse). It is quite extensible.

I don't understand how you can write an article about system design without mention to enterprise architect or Capella. I've read this as "we need to come up with a solution that already exists".

argoeris13• 2 years ago

There are quite a few tools cropping up trying to solve this problem. Multiplayer.app is one example - they use OTel to gather distributed traces from your system and ensure you automatically get notified when there's drift.

dennisy• 2 years ago

Great post but lacking an answer. I guess we are yet to find one!

sam1r2• 2 years ago

Agreed!

If every acknowledgement of the(or any imo) issue at hand, was accompanied with a minimally viable solution (even if it doesn't work) -- as the defacto standard for providing critical feedback..

As long as some effort is put in towards a countering possible solution in response-- I truly feel that we would be closer to an end solution.

I guess should this "problem" truly exist.

Joel_Mckay• 2 years ago

I think the folks working on constraint programming with MiniZinc and Phoenix/LiveView have achieved a fairly practical solution.

https://github.com/bokner/solverview

https://github.com/bokner/solverl

https://www.minizinc.org/

YMMV, as I've yet to personally try a toy project with this approach. =3

argoeris13• 2 years ago

Check out https://www.multiplayer.app/ The author is one of the co-founders, but I understand not wanting to just promote his tool and just have a conversation about the problem.

RcouF1uZ4gsC• 2 years ago

Today, we have the technology and knowledge to create tools that prevent developers from wasting valuable development time deciphering static, outdated diagrams,

One issue is that diagrams in general are pretty universal. As long as your tool can makes shapes and connect them, you can use it for any kind of architecture. For example it will work flow charts for 50 year old Fortran to UML from the 90s to microservices diagrams from now and I bet to whatever is common 50 years from now.

sam1r2• 2 years ago

> >Today, we have the technology and knowledge to create tools that prevent developers from wasting valuable development time deciphering static, outdated diagrams,

"Outdated", imo, is subjective to the content and is bias'd by the time since creation.

As long as wireframed thought is the bare skeleton, the diagrams are merely a read-only user interface to communicate to a particular end-user audience.

argoeris13• 2 years ago

I don't think the issue is with diagrams per se, but with how we create them. They are super helpful in conveying meaning but why do we need to create and update them manually?

jawns2• 2 years ago

This reminds me of the following:

* Devs: We need a better language than Javascript as the lingua franca of the web!

* Users: Sure, use whatever language you want -- just make sure it compiles to Javascript, which is already the lingua franca of the web.

Keep in mind that the consumers of technical diagrams are often non-technical folks. And they don't care about how they get their diagrams. They just want to be able to understand, at a high level, what's going on in the black box.

You can either convince every single one of them that devs need to focus on better system design tools ... or you can continue to give them the diagrams they want, just using a smarter process to generate them.

Or you can treat them as entirely separate problems, because fundamentally system design tools are building tools, and system diagrams are communication tools. In most cases you can improve them independently.

BadHumans• 2 years ago

* Users: Sure, use whatever language you want -- just make sure it compiles to Javascript, which is already the lingua franca of the web.

I don't think users are saying this and I don't think they care as long as it works.

Jtsummers5• 2 years ago

Exactly. Users don't care about the underlying tech as long as it works and works on the platforms they're using and with other systems they're using. That's it. When users start caring about the underlying tech, you need to actively dissuade them unless there is a sound technical argument. Otherwise you end up with people insisting on using DOS for enterprise applications in 2024 because that's what they've used since 1984 (wish I hadn't experienced this).

jawns2• 2 years ago

Right, they aren't explicitly saying it.

They are, however, implicitly saying it with the position that "I don't care as long as it works."

Javascript is available on every modern browser. It's not going away any time soon. Migrating to some other frontend language, no matter how demonstrably better it is than Javascript, is a Herculean effort. Actually, it's more of an Atlassian effort, since it would require practically an Earth-wide shift.

factormeta• 2 years ago

Isn't that what WASAM suppose to be?

TheRealPomax• 2 years ago

* Users: Sure, use whatever language you want, as long as it runs in my browser, because that's what I'm already using.

qznc4• 2 years ago

Well said.

Users: Sure, use whatever presentation you want - just make sure it can be shown on a 2-dimensional screen, which the lingua franca of presentations.

bsder• 2 years ago

Yeah, the PowerPoint-ization is now so complete that people can't even conceive that there are other ways to give a presentation.

Detrytus• 2 years ago

Compiling to Javascript is a really low bar: all Turing-complete languages are equivalent, one can be translated to another. You could have a Lisp or Haskell compile to Javascript. The problem is: everyone has their favorite language, so such ecosystem would be fragmented, and Javascript is the lowest common denominator.

XorNot2• 2 years ago

I'm not sure I've met a consumer of diagrams who ever really wanted to know what's going on, so much as they wanted to have their pre-conceptions validated for the brief time they're interested in the problem.

I have had too many conversations with managers who want to talk about "pipelines" but then you ask how the system is handling dropped or failed messages and get a deafening silence or "just don't worry about it it's rare".

People love arrows showing the "direction" data is going, but do things like assume that direction means "security" (it doesn't - any channel other then a full on data-diode is 2-way) or implies other operations (i.e. about storage).

argoeris13• 2 years ago

The article does indeed argue that we need smarter ways to create diagrams - so that devs don't have to manually create / update them and other teams get the info that they need.

Saving time and effort for devs, while making dynamic visualizations with different levels of detail (i.e. more and less technical for different audiences) is possible.

canterburry2• 2 years ago

We have the tools and ways to express architecture. The problem is no one is bothering to actually learn it and do it well.

Everyone seems to look for answers involving not having to spend any time learning anything new.

UML, BPMN, 4+1 view, it can do it all. Just learn it and make sure everyone else in your team does too.

Supermancho• 2 years ago

Product people will not. So we have the inglorious situation of 2 sets of diagrams and notes on the translation between them. If the problem was easy to solve, it wouldn't be something you can express in a sentence, while simultaneously asserting how easy it is.

Some people do not adequately constrain the problem while others overly constrain it, conceptually.

canterburry2• 2 years ago

BPMN was created for product people. UML use case diagrams were created for product people.

I don't see the argument why product people don't need to learn this. They work in a software organization, they build software, they should be expected to know the diagrams pertinent to them.

wudangmonk• 2 years ago

Honestly ALL diagrams are a dead end, any system where there is no single source of truth will be a big hassle to maintain and therefore will not be updated which makes the whole thing useless.

The only which I can see working as it has already proven itself for certain tasks is something like the blueprints from the unreal engine but where each node represents something big enough to warrant its own node without any no low level details like variables, conditionals, loops etc. Otherwise you end up with literally spaghetti code.

argoeris13• 2 years ago

This is an excellent point. You need to pull all the information about a system in a single place so that then you can choose what level of abstraction or deep dive into the details you need.

Projects like Multiplayer.app are in their early days, but I can see the potential of focusing on concentrating this info and automating the maintenance of docs and diagrams.

SrslyJosh• 2 years ago

This would be more interesting if there was actually a proposed solution.

argoeris13• 2 years ago

Check out https://www.multiplayer.app/ That's the project the author is working on.

qwerty456127• 2 years ago

Also GUI design tools like the WinForms designer but for more modern toolkits please. VB/Delphi/WinForms were such a joy to design and XAMl (let alone web frontent) is such a pain to write.

ranger_danger2• 2 years ago

I feel like https://plantuml.com/ gets close to what I want by being able to make diagrams with code, but for system design what I really want is to be able to have diagrams generated directly from the code itself, maybe with some extra comments/annotations that help it along.

Does anything like that exist already?

nomoreipg3• 2 years ago

Commented this above, but I actually built a site to auto generate up-to-date interactive visual diagrams for codebases. It's pretty plug&play, using static analysis & LLMs we generate documentation + an interactive system diagram you can recursively explore.

https://vision.useadrenaline.com

All the auto-generated diagrams are exportable as mermaid!

ranger_danger2• 2 years ago

proprietary? no thanks

Veuxdo3• 2 years ago

You need to use diagramming tools that are meant for complex systems. Almost all of the complaints in this article are caused be using generic drag-and-drop diagramming tools.

hemloc_io• 2 years ago

What are those tools?

Every big tech job I've had just uses the more generic ones like draw.io or maybe an internal only version.

ABoredBirb• 2 years ago

C4 diagrams using Mermaid markdown. Easy to reason about, easily version-controlled.

argoeris13• 2 years ago

My vote is for https://www.multiplayer.app/

agumonkey• 2 years ago

I wonder if modeling/system design people didn't have new insights since the last 20 years (when Model Driven * started to sink).

Languages changed a bit, more types, more concurrency, more functional idioms. Yet I feel we're not going anywhere.

bunsenhoneydew• 2 years ago

As others have said the C4 model is a great way to address a number of these issues.

I can’t find the right video at the moment but Simon Brown (creator of C4) gives a great talk about creating his DSL, Structurizr, for C4, which he developed during COVID lockdown (if memory serves). There are many videos on YouTube of Simon talking about “C4 Models as Code” so I’m sure any one of those will suffice.

The focus is on creating the model of your system architecture, from which the diagrams you extract are a specific projection of that model. Rather than a diagram be the main artifact. It’s a simple but very powerful concept that I’m always surprised isn’t more widely used.

Structurizr models can also be exported to display as ilograph diagrams, mermaid diagrams and more. Also very much worth a mention is icepanel, a lovely tool for architectural model that implements the C4 model heavily.

I saw Simon talk at a conference in Sydney about 10-15 years ago and heard about C4 for the first time in that talk, it’s been one of the most influential talks I’ve been to in my career as it made a lot of fuzzy things in my head all start to come together in a way that just made sense.

https://c4model.com/

https://structurizr.com/

https://www.ilograph.com/

https://mermaid.js.org/

https://icepanel.io/

nomoreipg3• 2 years ago

I actually built a site to auto generate up-to-date interactive visual diagrams for codebases.

It's pretty plug&play, using static analysis & LLMs we generate documentation + an interactive system diagram you can recursively explore.

https://vision.useadrenaline.com

All the auto-generated diagrams are exportable as mermaid

_caw• 2 years ago

I really wanted to check this out (using redis as an example), but it appears I need an account.

Is there no public demo?

vegetablepotpie• 2 years ago

This is similar to the 4+1 views concept.

I’ve never seen modeling tools widely used. When it’s attempted it’s always a top-down initiative that always addresses management concerns, but always creates more risk for developers without generating value for them, and it never takes off outside of being a box on a process list to check off. Developers promptly stop using these tools as soon as their use is no longer enforced.

The long term destiny of models is that they will always diverge from the code, sometimes in ways that matter. In that situation, the best model of the code is the code it’s self because it’s a model that you can rely on. You can certainly update the diagrams, but that’s a priority call vs. what else you’re doing, and it doesn’t always win out over other tasks.

Some enterprising tool developers will make modeling tools that will generate code, therefore linking the diagrams and the model to the development of the code. When this is employed, it’s complex and it’s never a perfect implementation. Generated code can be buggy, or it may not do what you want. It may be a significant effort to make the models generate the code that is needed, and you may find your self further developing the code generation tools to generate the code you want. I’ve never seen this have a good benefit to cost ratio.

argoeris13• 2 years ago

The problem with any of these tools is that they solve only one part of the puzzle. Take Structurizr for example, it doesn't automatically create the diagrams for you or notify you when it detects architectural drift (and automatically update the diagram).

Others miss other pieces of the puzzle, such as having a list all your APIs, all your system docs in a single place (ADRs, reqs, etc.), connecting to your repos, etc.

I recommend checking https://www.multiplayer.app/ as an alternative.

simon_brown3• 2 years ago

Take Structurizr for example, it doesn't automatically create the diagrams for you

The Structurizr DSL is designed for manual authoring (which is what most people tend to do), but there's nothing preventing you from writing some code (using one of the many open source Structurizr compatible libraries) to reverse-engineer parts of the software architecture model from source code, binaries, your deployment environment, logs, etc.

or notify you when it detects architectural drift

If you do the above, there's then nothing preventing you from writing some tests to notify of architectural drift, etc.

simon_brown3• 2 years ago

Thank you! :-)

bitwize3• 2 years ago

In 1971, M. Bryce and Associates marketed PRIDE, the first commercial software methodology for general business use. It was originally an entirely manual process using paper forms, but it was already far more comprehensive than anything called a "methodology" we use today. For example, in PRIDE, every artifact of the systems design and implementation process -- each high-level requirement, each software module, each database table and column, and more -- is assigned a tracking number and extensively documented as to its purpose in a unified knowledge repository. This was decades before Git or JIRA, and at first it was all done by hand, but not for long.

In the 80s, they marketed PRIDE/ASDM, which combines PRIDE with Automated Systems Design Methodology, a suite of system design tools written in COBOL for mainframes. Far from being mere diagramming tools, they assisted in all aspects of an information systems design from initial requirements down through coding and database management. A key component of ASDM was the Information Resource Manager (IRM), a searchable central database of all of the information artifacts described above along with their documentation. Another component was Automated Instructional Materials (AIM), the online documentation facility, which not only provided instructions on how to use the system, it also provided step-by-step directions (called "playscripts" in PRIDE-speak) for each member of the development team to follow, in order to see a business system through from abstract design down to software implementation and deployment. Provided the directions were followed, it was next to impossible to screw up a system or subsystem implemented through PRIDE/ASDM.

This level of comprehensiveness and clarity is the gold standard for business IS development. And it seems to be nearly lost to time.

theamk2• 2 years ago

"every artifact of the systems design and implementation process [..] is assigned a tracking number and extensively documented", "it also provided step-by-step directions [...] for each member of the development team to follow"

Maybe this made sense back in times of COBOL, but with the modern high-level languages this sounds like bureaucratic hell, the worst kind of micromanagement.

bitwize3• 2 years ago

Well, what's happened in the years since is that programmers have been in the driver's seat when it comes to IS, and that's resulted in a lot of sloppy, disorganized thinking in the field. Programmers in general fancy themselves as artists and resist organization and discipline -- as true today as it was in the COBOL days -- so it's no surprise that a method based on improved process control seems like micromanagement. But that level of discipline is needed in order to do the work right. Ultimately it should come down to self-discipline.

Unfortunately, IS has morphed into a programmer-centric field, which leads us to Agile, a family of methodologies by programmers for programmers. Agile mythology has it exactly backward: Agile is not the thing that saved us from waterfall. Agile was the "traditional" software development process PRIDE was created as a response to: start programming right away, deploy rough versions to production, and keep iterating with code changes until you have something that kind of, sort of works. Compared to this, PRIDE delivers improved productivity and quality, reduced costs, and reduced administrative overhead. I've never been on an on-time, under-budget Agile project, and one of my former employers basically aborted their entire Agile transformation.

Once the industry has shaken off its Agile wine, especially as profitability becomes more important in the post-ZIRP economy, software development will look more like PRIDE than like anything we see today.

danjl• 2 years ago

Google Slides and Drawings (under + > More...) are great tools. They support the diagramming objects, connectors, shared collaboration, and integration with the other Google Doc tools. They also have great history and diff tools as well as team comments, with actions.

argoeris13• 2 years ago

.. but they are manual. It's great to have history, diff, comments, etc. But why do I have to spend time manually creating a diagram and updating it every time I add a new dependency when it can be automatically done for me?

digger495• 2 years ago

This is a useless and insipid editorial. I don't want the problem redeclared. We know this. We want solutions.

argoeris13• 2 years ago

Fair point. I'm sure we all agree on why using general-purpose diagramming tools is too much manual work for system design.

This is a possible solution: https://www.multiplayer.app/ It allows dynamic diagrams and keeps them automatically updated leveraging OTel.

yuliyp• 2 years ago

It's not so hard to do static and tracing-based analysis to capture all the calls that various systems make between themselves. Similarly, it's not so hard to graph all of the metrics of a system.

That's not really all that useful. Diagrams, like other forms of documentation, are a format for communicating something to the reader. It means that they should spend more space on the important flows rather than the exceptional ones. They'll explain the meaningful parts of a sequence rather than just giving a series of function calls. The various tools we have for doing this (boxes showing what systems talk to what, sequence diagrams, database schema diagrams) provide a rich enough language for that communication.

Death star diagrams are bad because they spend a bunch of time and effort to convey one piece of information: "there are a lot of systems here" rather than anything actually useful for someone attempting to navigate a specific part of the system.

steve19772• 2 years ago

I think at least part of the problem is that todays "agile" methodologies lure people into almost completely neglecting analysis and design, as if these steps are forbidden in agile. So for many dev teams I have seen, agile basically means "we have no plan".

spyspy• 2 years ago

I’d characterize it as the degradation of the product manager role. Too many project managers get promoted to product but don’t really know what that job entails so they just project manage harder.

And to a project manager “I’m gonna take a week and think about it” is not an acceptable answer.

argoeris13• 2 years ago

I agree that many teams, in an effort to move away from waterfall development and Big Design Up Front, have gone the opposite way and completely skip system design. Which is a mistake, because you need some upfront design.

As Dave Thomas said: “big upfront design is dumb. No upfront design is dumber”.

webprofusion• 2 years ago

If we had a complete and evolving set of requirements for any reasonably complex system, we wouldn't really read them once they no longer match the system we have delivered.

Systems and requirements go hand in hand but the reality is requirements are eventually codified, and after a point only the system matters because the system is the reality and the requirements were just the formal inspiration.

I would love to see automated requirement analysis and verification from code.

_xiaz• 2 years ago

No, I need a diagramming tool. Excalidraw lives on

argoeris13• 2 years ago

Excalidraw is great for brainstorming and sketching. But I don't exclude pairing it with a tool that also automatically shows me all the metadata of system components, automatically detects architecture drift, etc.

tomjohnson3• 2 years ago

Thank you everyone for reading my article! I’m the author, Thomas Johnson. This article stems from my frustration with the typical approach of asking, “What diagramming tool should we use?” instead of addressing the root problem: the need for up-to-date, easily accessible system architecture information. That’s why I co-founded Multiplayer. We focus on automating the creation and maintenance of system architecture diagrams and creating a single source of truth for system information. This includes your individual components, APIs, dependencies, and repositories.

We’re language and environment agnostic and you can start with a napkin sketch or a photo of your whiteboard. And this is just the start, we have many plans for how to evolve system design tooling including supporting popular integrations and models like C4.

It’s early days for us. I’d love to hear what you think: https://www.multiplayer.app/ (It’s free!)

kmerroll• 2 years ago

To take this a different way, Devs need AI-driven system design tools, not diagramming tools. The work to automate the process to define, code, deploy, and document tech/app solutions using GenAI is arguably well underway and the days of humans producing layers of boxes and arrows documentation is thankfully numbered.

More directly, the need for the special snowflakes and the ensuing complex system design processes will be reduced as we drive towards more automation N(A) frameworks.

airbreather• 2 years ago

There is a potentially interesting standard many may not know - IEC 61499 - Standard for Distributed Automation, first ed 2005!!! and related to PLC type controllers.

It was meant to be the next step forward from IEC 61131 that described the five PLC control languages, but far ahead of it's time.

61499 introduced the concept of an executable specification, and went on develop tie ins with XML and OPC. The executable specification is a particularly interesting idea, effectively emerging as no code these days

But two decades on is still only starting to get traction, seemingly because it was so far ahead of it's time, and the target audience was potentially mostly only just branching out into software because they were now programming ladder logic, instead of designing panel boards with hundred of relays and timers.

But, it is worth a look for some ideas in the space, particularly the executable specification concept (you know this can only lead to a DSL of some kind), but also including their diagramming that looks a lot like UML but maybe works better, along with 4Diac software package, consideration of how some of the more esoteric aspects of OPC fit into all this, and more slightly obliquely, TLA+.

Devs need system design tools, not diagramming tools

Related Stories