Distant organ metastasis is linked to poor prognosis during cancer progression. The expression level of the focal adhesion adapter protein paxillin varies among different human cancers, but its role in tumor progression is unclear. Herein we utilize a newly generated PyMT mammary tumor mouse model with conditional paxillin ablation in breast tumor epithelial cells, combined with in vitro three-dimensional (3D) tumor organoids invasion analysis and 2D calcium switch assays, to assess the roles for paxillin in breast tumor cell invasion. Paxillin had little effect on primary tumor initiation and growth but is critical for the formation of distant lung metastasis. In paxillin-depleted 3D tumor organoids, collective cell invasion was substantially perturbed. The 2D cell culture revealed paxillin-dependent stabilization of adherens junctions (AJ). Mechanistically, paxillin is required for AJ assembly through facilitating E-cadherin endocytosis and recycling and HDAC6-mediated microtubule acetylation. Furthermore, Rho GTPase activity analysis and rescue experiments with a RhoA activator or Rac1 inhibitor suggest paxillin is potentially regulating the E-cadherin-dependent junction integrity and contractility through control of the balance of RhoA and Rac1 activities. Together, these data highlight new roles for paxillin in the regulation of cell-cell adhesion and collective tumor cell migration to promote the formation of distance organ metastases.