In some cases the geometry of the structure in the port plane leads to degenerate modes, or multiple modes with the same wave number. For instance, the fundamental mode in a circular waveguide actually includes two modes with electric fields that are orthogonal to each other. In this case you may wish to request multiple modes to allow the degeneracy to be resolved.
These figures show the real part of the electric field at the port, for the first two modes in the waveguide. The second field is a rotation of the first one, and the two modes are degenerate, that is, they have the same wavenumber. In such cases the mode degeneracy is broken with the help of a local sampling point and an "up" direction that either you can define, or that the solver can auto-generate. These are used to force the electric field vector in the first mode in a degenerate pair to point in the direction of the "up" vector, at the sample point. In the circular waveguide above, the "up" vector points in the positive z direction, and as a result the field for the first of these two modes is directed along z. A given mode may have a field null at the default sample point, so that RF3p is not able to enforce mode consistency between solves. In this case, you will get better results by intelligently defining the sample point to be a location in the port where the fields are large, and by defining the sample direction to point in a major direction of the field at that location.
If you wish to resolve the degeneracy, you must request as many modes in the port definition as there are degenerate modes at that port (two in the case of the circular waveguide). If you ask for a single mode, then the result will be an essentially arbitrary combination of the two degenerate modes. The net effect in this case will be that the orientation may not be as desired and may change with each adaptive mesh refinement step, leading to inconsistencies in the S-matrix (depending on the orientation of the corresponding mode at other ports and mode coupling in the structure). The modes computed for a port are sorted in order of cutoff frequency, so if you are unsure about the mode spectrum at a particular frequency you can always request a Ports Only analysis and ask for additional modes. Degenerate modes can be identified by their common propagation constant, and non-propagating modes will have an imaginary propagation constant. Once you have asked for enough modes that at least one is non-propagating it is not necessary to look further - all higher-order modes will be non-propagating.