One small thing is that in the proof they use n-1 instead of n as the denominator for sample variance which is different from my/Kevin's settings. Although this might be trivial, I will further look into that to see if it makes any difference. Overall this (full) derivation I would say is non-trivial, although it is not some discoveries.
Thank you very much CrazyStat. I was convinced that "x-bar and s are independent for data coming from a Normal distribution". So the conclusion is for my example, p(x|mu) is normal distribution, p(x-bar|mu), after derivation, is also normal distribution. If p(x|mu) comes from some other distributions, p(x-bar|mu), however, might not be normal.
Thanks for providing the proof, and I will only be convinced by the proof :) Will update the post shortly. Thanks for the great conversation and suggestions.
Thanks CrazyStat. "sigma and mu are independent". But "s and x-bar are independent" is an assumption which does not hold in practice. It should also be noted that X and x are different. I defined p(x|mu) to be a Normal distribution but p(X|mu) was never defined to be normal.
However, I did make a mistake, I should have said p(x-bar|mu), instead of p(X|mu), follows normal distribution, when N->infty. This means that p(x-bar|mu) only approximates normal distribution but it will never be the exact normal distribution. This also matches the conclusion from central limit theorem.
I am going to fix the "p(x-bar|mu)" typo. Thanks for the discussion.