Ask HN: How to train a neural network based on pose estimation landmarks?

I am experimenting with pose estimation. In this case, the pose estimation uses your phone camera to detect the positions of your body's joints.

Now I will film myself performing simple actions, like, say, a pushup, or perhaps throwing a punch, several dozen times. I will process these videos in order to export the frame-by-frame positions of each joint.

What's the easiest way for me to take these time-series arrays and train a small neural network on them, so that given a new video, or rather, the last N frames of a live video feed, I can detect the action that was performed, if any?

I can already imagine how to preprocess/normalize the training data. I just need someone to point me in the right direction to learn how to train a simple model and perform inference.

I am using Python.

Thanks for any help!