Metal halide perovskites are promising candidates for next-generation photovoltaic and optoelectronic applications. The flexible nature of the octahedral network introduces complexity when understanding their physical behavior. It has been shown that these materials are prone to decomposition and phase competition, and the local crystal structure often deviates from the average space group symmetry. To make stable phase-pure perovskites, understanding their structure-composition relations is of central importance. We demonstrate, from lattice dynamics calculations, that the 24 inorganic perovskites ABX3 (A = Cs, Rb; B = Ge, Sn, Pb; X = F, Cl, Br, I) exhibit instabilities in their cubic phase. These instabilities include cation displacements, octahedral tilting, and Jahn-Teller distortions. The magnitudes of the instabilities vary depending on the chemical identity and ionic radii of the composition. The tilting instabilities are energetically dominant and reduce as the tolerance factor increases, whereas cation displacements and Jahn-Teller type distortions depend on the interactions between the constituent ions. We further considered representative tetragonal, orthorhombic, and monoclinic perovskite phases to obtain phonon-stable structures for each composition. This work provides insights into the thermodynamic driving force of the instabilities and will help guide computer simulations and experimental synthesis in material screening.