Efficacy of an Intensive Generative AI Professional Development Program on Pedagogical Content Knowledge (AI-PCK) and the Comparative Analysis of Learning Gain between Experienced and Pre-service Teachers

Synthesis: This quasi-experimental pretest-posttest study of an intensive 8-hour generative-AI professional development program with 163 teachers and pre-service teachers found significant gains across all five AI-PCK components (overall d = 2.36, p < 0.001), with pre-service teachers showing statistically higher learning gains than experienced teachers (p = 0.033). The Assessment Rubrics component (the lowest pretest area) recorded the largest effect size (d = 2.19), identifying assessment tools rather than teaching methods as the sharpest gap in teachers' knowledge for an AI era.

Key Findings

1. Very large overall effect. The intensive 8-hour program produced significant gains across all five AI-PCK components (p < 0.001) with an overall effect size of d = 2.36, confirming the efficacy of short-format, skills-focused professional development.
2. Rubrics showed the largest gain. The Assessment Rubrics component — which had the lowest pretest mean — recorded the highest effect size (d = 2.19), pointing to assessment tools as the most acute area of teacher deficiency.
3. Pre-service teachers gained more. Pre-service teachers showed a significantly higher overall learning gain than experienced teachers (p = 0.033), driven chiefly by the scenario-based task-design (p = 0.024) and rubric (p = 0.044) components.
4. Qualitative convergence. Participants reported transforming theoretical knowledge into procedural knowledge, reduced cheating anxiety, and a shift from viewing AI as a threat to a process-management assistant; experienced teachers reported needing longer, sustained support.

Background: From PCK to AI-PCK

Shulman's (1986) Pedagogical Content Knowledge (PCK) captured how teachers transform subject matter into accessible learning experiences. With the arrival of educational technology this was extended into the Technological Pedagogical Content Knowledge (TPACK) framework by Koehler and Mishra (2009), which stresses the intersection of content, pedagogy, and technological knowledge. The authors argue that Generative AI represents a paradigmatic shift: rather than a static supplementary tool, it enters the teaching-learning process as an active, generative, and automating agent, challenging TPACK and motivating a move toward AI-PCK (Mishra et al., 2023; Sarabi, 2024).

The Five AI-PCK Components

The study focuses on five components of Teacher AI Competency most directly impacted by Generative AI:

1. Designing scenario-based, AI-resistant tasks. Because AI can produce high-quality textual responses to standard prompts, teachers need to design Authentic Assessment tasks where students use AI as a data-gathering or analytical tool and apply findings in local or ethical contexts (Problem-Based Learning).
2. Personalized and differentiated content. AI supports Personalized Learning, letting teachers tailor supplementary materials to comprehension levels and interests while focusing their own effort on pedagogical interpretation.
3. Automation of assessment and learning analytics. AI automates repetitive grading and extracts learning data from assignments to surface error patterns and target Feedback (Automated Assessment).
4. Constructive feedback. Teachers can use AI-generated analytics and improvement suggestions, but must humanize and contextualize them so feedback stays process-oriented rather than a mere score.
5. Assessment rubrics. Transparent, analytical rubrics assign grades and evaluate process, Creativity, and depth of analysis — reducing cheating and grounding assessment in objective evidence rather than personal judgment (Summative Assessment).

Methodology

The study used a quasi-experimental pretest-posttest design with a non-equivalent control group and 163 participants: 112 experienced teachers (≥1 year) and 51 pre-service teachers. The intervention was an online, nationally-delivered program of four 2-hour sessions (8 hours total) covering the five components. Data were collected with a researcher-developed 5-point Likert questionnaire (10 items per component; total range 50–250) whose content and face validity were confirmed via Content Validity Index, with reliability at Cronbach's alpha 0.88. Quantitative data were analyzed with paired and independent t-tests and Cohen's d effect sizes; open-ended responses were analyzed with inductive content analysis (Research Methods in AIED).

Findings

The program produced statistically significant pretest-posttest gains on every component (p < 0.001). Total AI-PCK rose from M = 151.65 to M = 220.14, t(162) = 30.48, d = 2.36. Component-wise, the largest effect was for Assessment Rubrics (d = 2.19), followed by Assessment Automation (d = 1.95), Scenario-Based Tasks (d = 1.82), Differentiated Content (d = 1.75), and Constructive Feedback (d = 1.70). Descriptively, teachers entered with the most prior knowledge in task design and the sharpest deficit in objective assessment tools.

Comparative Learning Gains by Experience

Pre-service teachers showed a higher overall learning gain (M = 72.81, SD = 14.43) than experienced teachers (M = 66.72, SD = 14.42), t(161) = 2.15, p = 0.033. Significant group differences also appeared on Scenario-Based Tasks (p = 0.024) and Assessment Rubrics (p = 0.044), while Components Two, Three, and Four did not differ significantly.

Qualitative Themes

Three themes emerged from inductive content analysis:

1. Operationalizing PCK and alleviating cheating anxiety. Participants transformed theoretical knowledge into procedural knowledge, reporting that mastery of rubric and scenario-based-task design reduced anxiety about AI-generated cheating and restored confidence in Academic Integrity.
2. Changing perceptions of AI. Teachers moved from viewing AI as a threat to a productivity tool and process-management assistant, freeing time for human interaction and student engagement.
3. Different needs for professional development. Pre-service teachers found 8 hours sufficient for starting their careers, while experienced teachers reported the course was too short to change deep-seated habits and requested long-term, mentor-based support.

Discussion

The authors interpret the large effects through the lens of focus on immediate, operational classroom needs (such as AI-cheating) and the program's success in converting conceptual into procedural knowledge. The rubric component functions as a "defensive mechanism" against AI challenges, confirming the priority of process-oriented assessment in the AI era. The experience-based difference aligns with Rogers' (2003) Diffusion of Innovations theory: pre-service teachers act as early adopters with high cognitive flexibility who add a new skill, whereas experienced teachers must unlearn and replace established PCK — an inherently slower change in Adult Learners contexts.

What this means for practice

• Faculty developers. Lead with assessment, not tool fluency: teachers entered with the lowest pretest score on Assessment Rubrics (M = 27.51) and that component recorded the largest gain (d = 2.19), ahead of Assessment Automation (d = 1.95) and Scenario-Based Tasks (d = 1.82).
• Faculty developers. Run intensive short-format training when the need is operational: four 2-hour online sessions (8 hours total) raised total AI-PCK from M = 151.65 to M = 220.14, t(162) = 30.48, d = 2.36.
• Faculty developers. Differentiate by career stage: pre-service teachers gained significantly more than experienced teachers (M = 72.81, SD = 14.43 vs. M = 66.72, SD = 14.42, p = 0.033), so embed rubric and scenario-based-task design in initial Professional Development and give experienced staff sustained, mentor-based support instead of one-off sessions.
• Administrators. Adopt AI tools that cut administrative workload to motivate experienced teachers, whose change requires unlearning established practice rather than adding a skill.
• Researchers. Report component-level pretest profiles alongside the total score, since a total-score effect size hides which areas of AI-PCK teachers enter with least.

Limitations

• The design is quasi-experimental: a pretest-posttest with a non-equivalent control group and no full random assignment across 163 participants (112 experienced teachers, 51 pre-service), which the authors state limits generalization to a broader population.
• No follow-up measurement was taken after the 8-hour program, so the study cannot say whether gains persist after 6 months or a year — the sustainability question its own authors raise.
• Every AI-PCK outcome is self-reported on a researcher-developed 5-point Likert questionnaire (10 items per component, total range 50–250; Cronbach's alpha = 0.88), with no classroom observation or student-outcome measure.
• Teachers from all disciplines were pooled and analyzed together, with no discipline-specific breakdown of AI-PCK needs.

Citation

Talebzadeh, H. (2026). Efficacy of an Intensive Generative AI Professional Development Program on Pedagogical Content Knowledge (AI-PCK) and the Comparative Analysis of Learning Gain between Experienced and Pre-service Teachers. EdArXiv preprint.