Assessing skill and competency is a critical aspect of any micro-credential process. As technology advances, the use of artificial intelligence (AI) in various fields has become increasingly prevalent. However, when it comes to assessing micro-credential submissions (often similar to performance assessments), using AI may not always be the best practice. In this blog post, we will explore the limitations of AI, particularly in the context of micro-credentialing.
We aren’t the first to wonder how innovative technologies and tools can help speed up the assessment process, and AI is just the latest tool that folks hope to leverage. Digital Promise, and the Credentials team specifically, have been researching AI with regard to assessments. Current large language models (LLMs), which are driven by deep learning, do not have the ability to understand content and context and may have learned bias from their training data. This is holding up their ability to effectively assess competency-based work.
While AI has made remarkable strides in natural language processing within certain contexts, it still lacks the subject matter expertise and specific contextual understanding of the learner being assessed. Without this understanding, AI cannot accurately evaluate the nuances and complexities of competency-based work. This is especially true when assessing complex evidence-based submissions that might include a variety of artifacts that may include text, audio, video, and images to provide nuanced context and information about skill implementation.
There are no micro-credentials utilizing AI that have been developed in partnership with Digital Promise and/or using the Digital Promise micro-credential framework, and this has been by design. Competency-based assessments require an evaluator to gauge a person’s skills, knowledge, and abilities within a specific domain. AI’s assessment capabilities are limited to pattern recognition and statistical analysis, which may not be sufficient to evaluate complex competencies comprehensively.
AI has great utility for supporting many types of workflows. In fact, we used ChatGPT to help us think about how to write this blog. However, while it allowed us to overcome the writer’s block of a blank page, we still had to carefully review the content and evaluate accuracy and alignment with our stance and values. So why is it okay to use AI for this task, but not competency-based assessment?
Language Models (LM) rely on large training data sets that are often derived from the Internet, and as a result, AI tools tend to replicate the dominant patterns, biases, and other issues in the data (Bender, et al, 2021). The authors use stochastic parrot to refer to the ability of these tools to sound like a human, but lacking the ability to produce “meaningful text” from which humans gain a coherent understanding of a speaker’s intent, beliefs, and context. Further, Bender and colleagues caution about “mistak[ing] LM-driven performance gains for actual natural language understanding” which can lead to real risks and harm to human beings. More recent research has shown that popular tools such as ChatGPT demonstrate rudimentary levels of creative reasoning and technical proficiency while lacking the ability to critically reason (Bubek et al., 2023). Further, the authors state that these tools are still prone to errors, bias, and misinformation.
When considering AI in the context of learning, assessment, and credentialing, the risk of harming learners does not outweigh the cost of increasing efficiency or decreasing costs.
Performance assessments are not just about scoring or evaluating; they are also an opportunity for valuable feedback and personalized guidance. Human assessors can provide constructive criticism, motivational feedback, and individualized recommendations for improvement. AI, on the other hand, lacks empathy and cannot offer the human touch required to foster growth and development in learners.
AI models are trained on large datasets, and if those datasets contain biases, the AI’s assessments may also inherit those biases. This can lead to unfair evaluations, disadvantaging certain groups or individuals. Ensuring fairness and impartiality in assessments is a crucial aspect, and human assessors are better equipped to address these concerns with sensitivity and understanding. While people may also be prone to bias, it can be more easily corrected with appropriate systems, processes, or training.
Micro-credential assessments often involve real-world scenarios and tasks that require creativity, critical thinking, and problem-solving abilities. These complex assessments are challenging for AI to handle because they demand a deep understanding of context and context-specific knowledge. Human assessors can apply their experience and expertise to adapt to unique situations and evaluate learners more comprehensively, while also connecting with learners in a way that enhances their learning process.
The field of education and professional development is constantly evolving, with new skills and knowledge emerging regularly. AI models require updates and retraining to keep up with these changes. In contrast, human assessors can continuously learn and adapt their assessment methods to stay current with the latest trends and developments in their respective fields.
While AI is a remarkable tool with immense potential when used correctly, using it for performance assessments, particularly in the context of micro-credentialing, comes with inherent limitations. The lack of content understanding, ineffective assessment of competency, absence of human touch, bias concerns, complexity of assessments, and the inability to adapt are significant drawbacks of relying solely on AI for assessments. A balanced approach that combines AI’s efficiency with human assessors’ expertise is likely the best practice for fair and accurate performance evaluations. As technology progresses, AI may indeed become more proficient in assessments, but for now, human assessors remain indispensable for providing the valuable insights and guidance needed for personal and professional growth.
This blog post is part of a series exploring how to design micro-credentials for equity and inclusion throughout the year. If you are interested in learning more about micro-credentials, check out our current offerings on the Micro-credential Platform or visit our website to learn more about our services.
Sign up for our quarterly newsletter to stay updated on micro-credentials.
