Sensory neurons with related functions form ganglia, but how these precisely positioned clusters are assembled has been unclear. Here, we use the zebrafish trigeminal sensory ganglion as a model to address this question. We find that some trigeminal sensory neurons are born at the position where the ganglion is assembled, whereas others are born at a distance and have to migrate against opposing morphogenetic movements to reach the site of ganglion assembly. Loss of Cxcr4b-mediated chemokine signaling results in the formation of mispositioned ganglia. Conversely, ectopic sources of the chemokine SDF1a can attract sensory neurons. Transplantation experiments reveal that neuron-neuron interaction and the adhesion molecules E- and N-Cadherin also contribute to ganglion assembly. These results indicate that ganglion formation depends on the interplay of birthplace, chemokine attraction, cell-cell interaction, and cadherin-mediated adhesion.