HN user
I have interviewed CTOs & EMs of > 50 new-age companies about their testing cultures And learned something completely opposite to what I believed while working at Morgan Stanley.
We have a anonymiser which identifies common sensitive /Personally identifiable data like credit card, zip code and replaces them with anonymised data.
We also provide configuration option to specify additional fields are needed to be anonymised
Are you somehow automatically hooking into those function calls, recording their return values, and then mocking the functions in the replays? - this is correct.
As of now, no automated mutation although we do give option to modify the request by developers
In my experience, fuzzy testing is more helpful from Dast / security testing perspective and we were thinking of adding these later.
No, it doesn’t have by default but can be extended to support it.
Yes, I meant packages not part of opentelemetry - example python has lot of DB packages which don’t have support yet.
Go replay has been one of the inspiration Leonid, so glad you checked out CodeParrot :)
Typical Otel implementation don’t capture some request data esp parameters and replay part is missing among few other issues, so we need to extend it.
Thank you! Will keep you posted on open source version.
We have come across Hypertest, seems pretty cool and useful.
Yes, we rely a lot on openTelemetry for this. They have really good support for most libraries in Java, node and are progressing quickly in others. We are also contributing to it by extending support for other languages, which we'll be open sourcing soon.
I can relate to this perspective, however, some complexities we have come across in building this so far:
- Support for high number of languages, downstream dependencies - Intelligent sampling to choose requests with high coverage and auto update them over time - Performance, safety and data compliance guarantees
Yes, and Ken from speedscale is a very helpful person too.
Nice! Happy to share our experience if it helps :)
Good observation, it's challenging to solve these problems, here's how we are going about it -
To reduce false positives - we run the same request twice to eliminate flaky fields in response like current timestamp, mock the downstream dependencies as they behaved in prod env and are providing options to ignore / modify the sampled requests
To make integration easier - we are building on top of opentelemetry which has seen remarkable increase in support across languages / frameworks, which makes it easier for us to support different tech stacks.
Regression testing - our primary goal is to provide regression tests. We have come across two type of teams where this makes sense - companies with low test coverage and companies which high number of micro-services as they find it hard to cover every production scenario in tests
Nice catch, thanks for taking the time to review the website! Have updated it, should reflect in sometime.
it's similar in the sense that both rely on production traffic and user sessions to generate tests. However, we are focusing on API testing and I think Meticulous is building for UI testing.