Apologies for the delay. Now I do notice a delay when using the asdf shim instead of running the binary directly. But it's nowhere near 900ms.
I'm on a T480s with i5 CPU and SSD:
$ time node --version
v18.13.0
real 0m0.080s
user 0m0.079s
sys 0m0.027s
$ time ~/.asdf/installs/nodejs/lts/bin/node --version
v18.13.0
real 0m0.009s
user 0m0.006s
sys 0m0.003s