First moment
Population: $ \Omega = \{ x_1, \dots, x_N \} $
Collection of $n$-samples:
$\mathcal{S} = \{ s \in \Omega^n \mid \forall i,j \in s, i \ne j \} $
Collection of $n$-samples containing $x$:
$ \mathcal{S}_x = \{ s \in \mathcal{S} \mid x \in s \} $
Observe that $ |\mathcal{S}_x| = \binom{N-1}{n-1} $.
Let population mean be zero. $\mu = 0$, i.e. $ \sum_{i = 1}^N x_i = 0 $
Fix an order for $\mathcal{S}$:
$ \mathcal{S} = \{ s_1, s_2, \dots, s_{|\mathcal{S}|} \} $.
$j$-th $n$-sample mean $ m_j = \frac1n \sum_{x \in \mathcal{S}_j} x $
Remark: I don’t use $ \sum s_j $ as in $ \cup \mathcal{T} $ in topology to
avoid misreading the $n$-sample $ s_j $ as an element.
mean of $n$-sample mean
$ m = \frac{1}{|\mathcal{S}|} \sum_{s_j \in \mathcal{S}} m_j $
$$
\begin{aligned}
m \binom{N}{n} &= \sum_{s_j \in \mathcal{S}} m_j \\
nm \binom{N}{n} &= \underbrace{\sum_{s_j \in \mathcal{S}}}_{\text{fix sample}}
\underbrace{\sum_{x \in s_j}}_{\text{find item}} x \\
mN \binom{N-1}{n-1} &= \underbrace{\sum_{x \in \Omega}}_{\text{fix item}}
\underbrace{\sum_{s \in \mathcal{S}_x}}_{\text{find sample}} x
\end{aligned}
$$
The transition of the double summation on the RHS can be explained by the
bijection
$$
\{(s_j, x) \mid s_j \in \mathcal{S}, x \in s_j \} \leftrightarrow
\{(x, s) \mid x \in \Omega, s \in \mathcal{S}_x \}.
$$
Then the RHS of the last equality is
$$
\sum_{x \in \Omega} \sum_{s \in \mathcal{S}_x} x =
\sum_{x \in \Omega} |\mathcal{S}_x| x = \binom{N-1}{n-1} \sum_{x \in \Omega} x.
$$
Take out $\binom{N-1}{n-1}$ on both sides to get $ mN = \sum_{x\in\Omega} x $.
Hence $m = 0$.
Second moment
Population variance: $ \sigma^2 = \frac1N \sum_{x \in \Omega} x^2 $
Variance of $n$-samples:
$ \sigma_m^2 = \frac{1}{\binom{N}{n}} \sum_{s_j \in \mathcal{S}} m_j^2 $
$$
\begin{aligned}
& \binom{N}{n} \sigma_m^2 \\
=& \frac{1}{n^2} \sum_{s_j\in\mathcal{S}} \left(\sum_{x\in s_j} x \right)^2 \\
=& \frac{1}{n^2} \left[ \sum_{s_j\in\mathcal{S}} \left(
\sum_{x \in s_j} x^2 + \sum_{\small \begin{array}{l} & x,y \in s_j \\ & x \ne y\end{array}} xy \right) \right] \\
=& \frac{1}{n^2} \left[ \binom{N-1}{n-1} \sum_{x \in \Omega} x^2 + \binom{N-2}{n-2} \sum_{\small \begin{array}{l} & x,y \in \Omega \\ & x \ne y\end{array}} xy \right] \\
=& \frac{1}{n^2} \left[ \binom{N-1}{n-1} - \binom{N-2}{n-2} \right]
\sum_{x \in \Omega} x^2 \\
=& \frac{1}{n^2} \frac{N-n}{n-1} \binom{N-2}{n-2} N \sigma^2 \\
=& \frac{\sigma^2}{n} \frac{N-n}{N-1} \binom{N}{n}
\xrightarrow[N \to \infty]{} \frac{\sigma^2}{n},
\end{aligned}
$$
which is the standard error of sample mean.