Self-reported health literacy measures have seen increased application throughout the last years, among those are the brief health literacy screeners (BHLS) developed by Chew and colleagues (2004). There has been little systematic research on the performance of these measures across different contexts, including countries and languages, to draw conclusions about their predictive power outside of the United States. This study aimed at replicating the original validation of the BHLS. Receiver operating characteristic (ROC) analysis was applied to data from Hungary, Italy, Lebanon, Switzerland, and Turkey. In addition, logistic regression models incorporating ROC analysis using BHLS as predictors were compared to models using socio-demographics only to identify individuals with inadequate and inadequate or marginal health literacy as measured with the Short Test of Functional Health Literacy in Adults. Analyses showed that in all cases the BHLS were not sufficiently able to identify individuals with different health literacy levels. Logistic regression models using socio-demographics only as predictors outperformed models using the BHLS. The findings highlight the limitations of using the BHLS outside the United States. Further, they question in how far self-reported health literacy measures are comparable across different contexts and whether thresholds for different health literacy levels are universally applicable.