Why self-adjoint operators?
What is self-adjoint anyway? The adjoint $A^*$ of an operator $A$ (always densely defined on a Hilbert space $\mathcal{H}$) is given by all pairs $(y,z)$ (the graph of $A^*$) that obey \[ \langle y,Ax \rangle = \langle z,x \rangle \quad\forall x \in D(A) \] and thus it holds $z = A^*y$ and $y \in D(A^*)$. If $A=A^*$ on $D(A)$ the operator is called symmetric and one easily gets $D(A) \subseteq D(A^*)$. If further the domains coincide $D(A) = D(A^*)$ the operator is self-adjoint. For bounded operators we always have $D(A)=\mathcal{H}$ so the two notions are equivalent in that case.
Example. $i\partial_x$ with its natural domain $H^1([0,1]) = \{ f \in L^2([0,1]) \mid \partial_x f \in L^2([0,1]) \}$ (see Sobolev space) with all derivatives assumed weak. We study: \[ \langle f,i\partial_x g \rangle = i\int_0^1 \bar{f}\partial_x g \,dx = i \bar{f}(1)g(1)-i \bar{f}(0)g(0) - i\int_0^1 \partial_x\bar{f} g \,dx = i \bar{f}(1)g(1)-i \bar{f}(0)g(0) + \langle i\partial_x f,g \rangle. \]
The operator thus fails to be symmetric. On $H_0^1([0,1])$ instead it is symmetric, but having $g \in H_0^1([0,1])$ is enough for the boundary terms to vanish, so $f \in D((i\partial_x)^*) \supsetneq D(i\partial_x)$. To get a self-adjoint operator we must make $D(i\partial_x)$ larger which lets $D((i\partial_x)^*)$ shrink, until they match. A boundary condition $f(1)=\alpha f(0)$, $\alpha \in \mathbb{C}$, yields \[ \bar{f}(1)g(1)-\bar{f}(0)g(0)=|\alpha|^2\bar{f}(0)g(0)-\bar{f}(0)g(0) \] that vanishes if $|\alpha|^2 = 1$. Any such choice of $\alpha$ gives a self-adjoint extension of the original operator but there is no unique such self-adjoint extension.
Physical note. Clearly different $\alpha$ describe different “phase jumps” at the boundary that is now actually the joint where the two endpoints of the interval $[0,1]$ are identified. Without periodicity (or an infinitely large configuration space $\mathbb{R}$) there cannot be a momentum eigenstate (it would hit the wall), which somehow is supposed to be connected to self-adjoint operators.
Why self-adjoint operators? Sometimes it is argued that observables are represented by self-adjoint operators because they have real eigenvalues exclusively that are identified as the possible outcomes of a measurement. But why should a $z\in\mathbb{C}$ not be the outcome of a measurement, e.g. the pointer position of a clock? On the other side a non-self-adjoint operator may have only real eigenvalues. So this cannot really be the reason.
Looking back at the old matrix mechanics of Heisenberg one identifies the central method of diagonalization. For an arbitrary linear operator $A$ on a Hilbert space it is asked, whether there is a unitary $U$ such that $UAU^*$ is diagonal, i.e. a multiplication operator (a function $x \mapsto f(x)$, after all it's functional analysis) on $L^2(\Omega,\mathcal{A},\mu)$ on some measure space $(\Omega,\mathcal{A},\mu)$. In the case of $A=i\partial_x$ this can be achieved through Fourier transform. In general the diagonalization is achieved through a resolution of identity (the generalization of an orthonormal eigenbasis), i.e. every Hilbert space state can be assigned to one or more specific outcomes.
The mathematical result telling us which operators can be diagonalized is called the spectral theorem. Needless to say self-adjoint operators allow for such a diagonalization but also the more general normal operators do. They are to self-adjoint operators what complex numbers are to reals: \[ \begin{array}{l|ll} N^*N=NN^* & \bar{z}z = z\bar{z} = |z|^2 & z \in \mathbb{C} \\ \hline A=A^* & z = \bar{z} & z \in \mathbb{R} \end{array} \]
So should all observables be represented by normal operators? (Or by an even larger set?1)) Normal operators do not really add anything, since one can always write $N=A+iB$ with $A,B$ self-adjoint and $[A,B]=0$ just like with complex numbers. One can even find a polar decomposition $N=PU$ with $P=(N^*N)^{1/2}$ positive and $U$ unitary.
Eventually the real reason behind the restriction of observables (questions posed at the system) to self-adjoint operators seems to be the applicability of the spectral theorem, which in turn guarantees that the states belonging to all possible answers (the spectrum) span the whole Hilbert space (resolution of identity). In the case of an orthogonal projector, always a self-adjoint operator, the possible answers are only true/false, they represent binary questions. In QM the different answers are of course assigned only probabilities, so this principle of completeness makes sure that all the probabilities of all possible, mutually exclusive outcomes add up to 1.