A mathematical formulation for the guided modes of an optical fiber is derived from Maxwell's equations: this formulation leads to an eigenvalue problem for a family of self-adjoint noncompact operators. The main spectral properties of these operators are established. Then the min-max principle provides an expression of the nonlinear dispersion relation, which connects the propagation constants of guided modes to the frequency. Various existence results are finally proved and a complete description of the dispersion curves (monotonicity, asymptotic behavior, existence of cutoff values) is carried out.