The transforming growth factor-β (TGF-β) superfamily of ligands signals along two intracellular pathways, Smad2/3-mediated TGF-β/activin pathway and Smad1/5/8-mediated bone morphogenetic protein pathway. The C terminus of Hsc70-interacting protein (CHIP) serves as an E3 ubiquitin ligase to mediate the degradation of Smad proteins and many other signaling proteins. However, the molecular mechanism for CHIP-mediated down-regulation of TGF-β signaling remains unclear. Here we show that the extreme C-terminal sequence of Smad1 plays an indispensable role in its direct association with the tetratricopeptide repeat (TPR) domain of CHIP. Interestingly, Smad1 undergoes CHIP-mediated polyubiquitination in the absence of molecular chaperones, and phosphorylation of the C-terminal SXS motif of Smad1 enhances the interaction and ubiquitination. We also found that CHIP preferentially binds to Smad1/5 and specifically disrupts the core signaling complex of Smad1/5 and Smad4. We determined the crystal structures of CHIP-TPR in complex with the phosphorylated/pseudophosphorylated Smad1 peptides and with an Hsp70/Hsc70 C-terminal peptide. Structural analyses and subsequent biochemical studies revealed that the distinct CHIP binding affinities of Smad1/5 or Smad2/3 result from the nonconservative hydrophobic residues at R-Smad C termini. Unexpectedly, the C-terminal peptides from Smad1 and Hsp70/Hsc70 bind in the same groove of CHIP-TPR, and heat shock proteins compete with Smad1/5 for CHIP interaction and concomitantly suppress, rather than facilitate, CHIP-mediated Smad ubiquitination. Thus, we conclude that CHIP inhibits the signaling activities of Smad1/5 by recruiting Smad1/5 from the functional R-/Co-Smad complex and further promoting the ubiquitination/degradation of Smad1/5 in a chaperone-independent manner.