While traditional crystallographic representations of structure play an important role in materials science, they are unsuitable for efficient machine learning. A range of effective numerical descriptors have been developed for molecular and crystal structures. We are interested in a special case, where distortions emerge relative to an ideal high-symmetry parent structure. We demonstrate that irreducible representations form an efficient basis for the featurization of polyhedral deformations with respect to such an aristotype. Applied to a data set of 552 octahedra in ABO3 perovskite-type materials, we use unsupervised machine learning with irreducible representation descriptors to identify four distinct classes of behaviors, associated with predominately corner, edge, face, and mixed connectivity between neighboring octahedral units. Through this analysis, we identify SrCrO3 as a material with tunable multiferroic behavior. We further show, through supervised machine learning, that thermally activated structural distortions of CsPbI3 are well described by this approach.