DNA methylation is an epigenetic modification that regulates gene expression by DNA methyltransferases (DNMTs). Inhibition of DNMTs is a promising approach for cancer therapy. Recently, novel classes of the quinolone-based compound, SGI-1027, and RG108-procainamide conjugates, CBC12, have been identified as potent DNMT inhibitors. In this work, we report comprehensive studies using induced-fit docking of SGI-1027 and CBC12 with human DNMT1 and DNMT3A. The docking was performed in the C-terminal MTase catalytic domain, which contains the substrate and cofactor binding sites, in the presence and absence of other domains. Induced-fit docking predicts possible binding modes of the ligands through the appropriate structural changes in the receptor. This work suggests a hypothesis of the inhibitory mechanisms of the new inhibitors which is in agreement with the reported autoinhibitory mechanism. The insights obtained in this work can be used to design DNMT inhibitors with novel scaffolds.