Small GTPases play a critical role in membrane traffic. Among them, Arf6 mediates transport to and from the plasma membrane, as well as phosphoinositide signalling and cholesterol homeostasis. Here we delineate the molecular basis for the link between Arf6 and cholesterol homeostasis using an inducible knockout (KO) model of mouse embryonic fibroblasts (MEFs). We find that accumulation of free cholesterol in the late endosomes/lysosomes of Arf6 KO MEFs results from mistrafficking of Niemann-Pick type C protein NPC2, a cargo of the cation-independent mannose-6-phosphate receptor (CI-M6PR). This is caused by a selective increase in an endosomal pool of phosphatidylinositol-4-phosphate (PI4P) and a perturbation of retromer, which controls the retrograde transport of CI-M6PR via sorting nexins, including the PI4P effector SNX6. Finally, reducing PI4P levels in KO MEFs through independent mechanisms rescues aberrant retromer tubulation and cholesterol mistrafficking. Our study highlights a phosphoinositide-based mechanism for control of cholesterol distribution via retromer.