The generation of optical frequency combs in microresonator devices is currently an area of intense research. Microresonator based frequency comb technologies permits the creation of new, compact and efficient light sources that are finding increasing applications for multiple fields such as spectroscopy, metrology and quantum optics. The physical mechanism behind Kerr comb generation is the instability of the continuous wave (CW) intracavity field to periodic modulations, i.e. the modulational instability (MI). MI is responsible for the generation of the primary sidebands around the pump wavelength, and the spectral properties of the MI gain is therefore paramount in determining the kind of comb states that can form inside the resonator cavity. This process is by now well understood for the case of a single field with a definite polarization. However, it is less well understood in the vector case when two orthogonal polarization modes are simultaneously excited.

Nonlinear mode coupling between the modes can, similar to polarization mode coupling in non-resonant structures such as optical fibers, lead to new types of vector instabilities. These instabilites can in turn bring about the formation of new types of comb states that have no counterpart for the scalar case. In this work, we consider two orthogonally polarized quasi-TE/TM mode families that are nonlinearly coupled through cross-phase modulation.

The dynamic of the system can in the mean-field limit be modelled by two normalized Lugiato-Lefever type equations [1]. These equations admit a family of homogenuous CW steady-state solutions, that can be characterized by the total power and relative detuning between the two modes. These will, for the non-degenerate case, display Kerr tilted resonances with a split resonance shape that can feature multi-stability.

We have performed an in-depth analysis of the MI of these solutions. The potentially unstable eigenvalues are found to be roots of a characteristic equation. We show that the eigenvalues can be written explicitly in two complementary limits, and that they depend on two phase-matching conditions. These can be used to describe the asympototic instability development, and give simple rules for the conditions that allow for the occurance of different kinds of MI. An example showing the MI growth rate for a case when the two polarization modes have opposite signs of dispersion is shown in Fig. 1. The regions in which the different instability conditions are satisfied are highlighted in the figure, and can be associated with the growth of different combinations of eigenvalues and normal modes. Also shown in the figure, is a numerical simulation of the coupled mean-field equations, that demonstrates the development of a frequency comb from a vector instability that is characterized by a complex conjugated pair of eigenvalues. The simulation is made for a fixed parameterized detuning, and shows that the instability development eventually results in the generation of a novel group-velocity locked temporally periodic pattern state where the peak intensity alternates between the two polarizations components.

In summary, we have demonstrated that nonlinear polarization mode coupling can give rise to MI with spectral properties that have no scalar counterpart.We have found some simple phase-matching conditions that can be used to predict their occurence, and shown that the development of the instability can lead to the generation of novel frequency comb states. We anticipate that these results will find importance in the design of future microresonator devices that use mode coupling to aid in comb generation, e.g., in the normal dispersion regime.