Beyond Self-Report: Evaluating Authentic AI Literacy in a Continuing Education Context

MARIKA MASCITTI, Paola Rigoni, Chiara Panciroli

Abstract


Research on AI literacy has predominantly relied on self-report instruments (Lintner, 2024), which correlate weakly with objective performance-based measures (Zhang et al., 2026). This study examines how AI literacy can be reliably assessed among adults engaged in continuing education. Participants (N = 141) completed the AILQ (Ng et al., 2024), then performed an authentic task based on a real-world employer branding scenario. Performance was assessed using a four-dimension analytic rubric. A Disposition-Performance Gap (DPG) index quantified the misalignment between self-perceived disposition and demonstrated performance: 68.8% of participants were aligned, while 27.7% showed a Disposition Surplus. This pattern was strongly stratified by competence: all participants at the lowest performance level overestimated their competence, whereas proficient participants were almost entirely aligned (χ²(4) = 78.413, p < .001). Findings support performance-based assessment as a necessary complement to self-report instruments.

Keywords


AI literacy; Performance-based assessment; Continuing Education; DPG

Full Text:

PDF PDF

References


Beckert, J., & Koch, T. (2025). Corporate perspectives on diversity: Engagement, communication motives, and the Diversity-Washing Dilemma. Journal of Marketing Communications, 31(5), 538–555. https://doi.org/10.1080/13527266.2025.2470346

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? FAccT 2021 - Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. https://doi.org/10.1145/3442188.3445922

Bick, A., Blandin, A., & Deming, D. J. (2024). The rapid adoption of generative AI (NBER Working Paper No. 32966). National Bureau of Economic Research. https://doi.org/10.3386/w32966

Council of the European Union. (2000). Council Directive 2000/78/EC of 27 November 2000 establishing a general framework for equal treatment in employment and occupation. Official Journal of the European Communities, L 303, 16–22. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A32000L0078

Dell’Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023). Navigating the jagged technological frontier: Field experimental evidence of the effects of artificial intelligence on knowledge worker productivity and quality (Harvard Business School Working Paper No. 24-013). Harvard Business School. (Published 2026 in Organization Science, advance online publication, https://doi.org/10.1287/orsc.2025.21838 )

Eva, K.W. and Regehr, G. (2005) Self-Assessment in the Health Professions: A Reformulation and Research Agenda. Academic Medicine, 80, S46-S54.

https://doi.org/10.1097/00001888-200510001-00015

Fernandes, D., Villa, S., Nicholls, S., Haavisto, O., Buschek, D., Schmidt, A., Kosch, T., Shen, C., & Welsch, R. (2026). AI makes you smarter but none the wiser: The disconnect between performance and metacognition. Computers in Human Behavior, 175, Article 108779. https://doi.org/10.1016/j.chb.2025.108779

Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020). RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020, 3356–3369. https://doi.org/10.18653/v1/2020.findings-emnlp.301

Girrbach, L., Alaniz, S., Smith, G., & Akata, Z. (2025). A Large Scale Analysis of Gender Biases in Text-to-Image Generative Models. https://doi.org/https://doi.org/10.48550/arXiv.2503.23398

Glazko, K., Mohammed, Y., Kosa, B., Potluri, V., & Mankoff, J. (2024). Identifying and Improving Disability Bias in GPT-Based Resume Screening. 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT 2024, 1, 687–700. https://doi.org/10.1145/3630106.3658933;CSUBTYPE:STRING:CONFERENCE

Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127. https://doi.org/10.1136/amiajnl-2011-000089

Gulikers, J.T.M., Bastiaens, T.J. & Kirschner, P.A. (2004). A five-dimensional framework for authentic assessment. ETR&D 52, 67–86. https://doi.org/10.1007/BF02504676

Kamruzzaman, M., Minul, M., Shovon, I., & Kim, G. L. (2023). Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models. https://arxiv.org/pdf/2309.08902

Koch, M. J., Carolus, A., Wienrich, C., & Latoschik, M. E. (2024). Meta AI literacy scale: Further validation and development of a short version. Heliyon, 10(21), Article e39686. https://doi.org/10.1016/j.heliyon.2024.e39686

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.

Martens, L., Virgin, N., Hoffarth, P., Goodwill, J., Derenne, N., Eck, R. Van, Klug, M., Manocha, G. D., & Jurivich, D. (2025). Recapitulation of Ageism in Artificial Intelligence–Generated Images: Longitudinal Comparative Study. Journal of Medical Internet Research, 27(1), e68428. https://doi.org/10.2196/68428

Muñoz-Ortiz, A., Gómez-Rodríguez, C., & Vilares, D. (2024). Contrasting Linguistic Patterns in Human and LLM-Generated News Text. Artificial Intelligence Review, 57(9), 265. https://doi.org/10.1007/s10462-024-10903-2

Naik, R., & Nushi, B. (2023). Social biases through the text-to-image generation lens. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (pp. 786–808). https://doi.org/10.1145/3600211.3604711

Ng, D. T. K., Wu, W., Leung, J. K. L., Chiu, T. K. F., & Chu, S. K. W. (2024). Design and validation of the AI literacy questionnaire: The affective, behavioural, cognitive and ethical approach. British Journal of Educational Technology, 55(3), 1082–1104. https://doi.org/10.1111/bjet.13411

Phutane, M., Jung, H., Kim, M., Mitra, T., & Vashistha, A. (2025). ABLEIST: Intersectional Disability Bias in LLM-Generated Hiring Scenarios. https://arxiv.org/pdf/2510.10998

Sandelowski, M.J., Voils, C.I., & Knafl, G.J. (2009). On Quantitizing. Journal of mixed methods research, 3, 208 - 222.

Tao, Y., Viberg, O., Baker, R. S., & Kizilcec, R. F. (2024). Cultural bias and cultural alignment of large language models. PNAS Nexus, 3(9). https://doi.org/10.1093/pnasnexus/pgae346

Wang, B., Rau, P.-L. P., & Yuan, T. (2022). Measuring user competence in using artificial intelligence: Validity and reliability of artificial intelligence literacy scale. Behaviour & Information Technology. Advance online publication. https://doi.org/10.1080/0144929X.2022.2072768

Wingerter, T. L., Straub, T., & Schweitzer, S. (2025). Mitigating automation bias in generative AI through nudges: A cognitive reflection test study. Procedia Computer Science, 270, 2106–2114. https://www.sciencedirect.com/science/article/pii/S1877050925030042

Zhang, S., Xiao, R., Botelho, A. F., Liao, G., Chiu, T. K. F., Stamper, J., & Koedinger, K. R. (2026). How to assess AI literacy: Misalignment between self-reported and objective-based measures. In LAK26: Proceedings of the 16th International Learning Analytics and Knowledge Conference. Association for Computing Machinery. https://doi.org/10.1145/3785022.3785088




DOI: https://doi.org/10.32043/gsd.v10i4.1872

Refbacks

  • There are currently no refbacks.


Copyright (c) 2026 ITALIAN JOURNAL OF HEALTH EDUCATION, SPORT AND INCLUSIVE DIDACTICS

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Italian Journal of Health Education, Sports and Inclusive Didactics 

Publisher: Cares srl – REN Press (since 2026). Previously published by Gaia srl – Edizioni Universitarie Romane (2017–2025).

ISSN: 2532-3296 E-ISSN: 3103-5744