On sufficient dimension reduction methods based on a graphical model with non-concave penalty

Ultra high-dimensional datasets, which refer to scenarios where the number of covariates grows at an exponential rate relative to the sample size, are frequently encountered in modern data analysis across fields such as genomics, finance, and social sciences. These datasets pose significant challenges due to the large number of variables relative to the number of observations, potentially resulting in issues such as multicollinearity, overfitting, and computational difficulties. Traditional sufficient dimension reduction (SDR) methods struggle with these challenges, making it necessary to develop new approaches. To address these limitations, we introduce a graphical model-based SDR method that incorporates a smoothly clipped absolute deviation (SCAD) penalty. This method effectively reduces dimensionality while managing sparsity in the dataset. Additionally, we extend directional regression for high-dimensional data by integrating them with graphical LASSO, which enhances the model’s ability to estimate sparse precision matrices. This combined approach not only mitigates computational infeasibility in estimating covariance matrices but also helps avoid overfitting, making it particularly suitable for high-dimensional contexts. Through extensive simulation studies and real-world data analyses, we validate the robustness and effectiveness of our proposed methods. Moreover, we provide a theoretical framework that discusses the convergence rate of these methods, offering insights into their performance under various conditions. Finally, we outline potential avenues for future research, including exploring alternative penalty functions and expanding the applicability of these methods to other types of data structures.