Reading QizMaker

ReadingQuizMaker: A Human-NLP Collaborative System that Supports Instructors to Design High-Quality Reading Quiz Questions.

201250035 邓尤亮

2023.11.10

---

0. Abstract

• Purpose: support instructors to conveniently design high-quality questions to help students comprehend readings
• Adapt to natural workflow and offer NLP-based support
• Evaluation Study:
  • questions generated with ReadingQuizMaker comparable to their manually designed quizzes
  • NLP suggestions satisfying and helpful
  • show the importance of allowing users to control the use of AI and providing immediate previews of AI-generated outcomes​

---

1. Introduction

1. Assigned readings are crucial in college education. But a persistent issue in higher education is that only 20-30% of students complete their reading assignments, a trend that has worsened with the rise of social media.
2. Various strategies and tools have been implemented to enhance students’ reading experience, like social annotation tools, which, however, work better for students with self-regulated learning skills.

---

3. Designing high-quality, thought-provoking questions is time-consuming and challenging, and existing NLP-based automatic question generation systems have low adoption rates in classrooms due to their domain specificity and poor quality.
4. Introduce ReadingQuizMaker
5. The paper claims three main contributions:
   1. revealing the challenges instructors face in creating reading quiz questions
   2. introducing ReadingQuizMaker with its NLP-based support
   3. An evaluation study of the system to demonstrate its usability and the advantages of a human-AI teaming approach​

---

2. Related Work

1. Low Compliance in College Reading Assignments
2. Strategies to Support Reading Practices
3. Interfaces to Support Active Reading
4. Question Generation Techniques for Educational Purposes
5. Human-AI Systems for Education
6. Human-AI System Design Guidelines

---

3. Formative Investigations

1. Unique Challenges in Question Creation
2. Desire for High-Quality Questions
3. NLP Tool Support

---

4. Design Requirements
   • Support for creating convincing distractors.
   • Process-oriented support allowing for the incorporation of instructors’ expertise.
   • Integration with current workflows for quick question creation.
   • Easy feedback mechanism for the questions.
   • Ensuring instructors have control when interacting with AI.

---

4. Reading Quiz Maker

Detailed Design

1. Compatibility with HTML Articles
2. Navigation Bar
3. Immediate Preview of NLP Suggestions
4. User Natural Workflows and Feedback
5. NLP Toolbox
6. Review and Output

---

NLP Models

1. Abstractive Summarization Model: Utilizes a fine-tuned BART (Bidirectional and Auto-Regressive Transformers) model that is trained on the CNN-DailyMail dataset.
2. Paraphrase Model: This model is pretrained on PEGASUS (Pre-training with Extracted Gap-sentences for Abstractive Summarization Sequence-to-sequence models) and is used to rephrase sentences while retaining their semantic information
3. Negation Model: A BART-based model fine-tuned on WikiFactCheck-English for negative claim generation.

---

5. Evaluation Study

Four key research questions:

1. Usability of ReadingQuizMaker: Instructors could use ReadingQuizMaker to create satisfactory questions.
2. Perception of AI Suggestions: Instructors perceived AI suggestions positively, finding them useful and not distracting, and they were able to control when and how to use these suggestions.
3. Comparison with Automatic Question Generation: The study found a strong preference among instructors for the human-AI teaming approach of ReadingQuizMaker over the automatic question generation approach.
4. Challenges and Design Implications: Instructors reported challenges in creating distractors and finding appropriate question stems.

---

6. Discussion

1. Positive Reception of ReadingQuizMaker
2. Preference for Human-AI Collaboration
3. Challenges and Future Directions: The paper identifies several challenges that remain in the question creation process:
   • Discoverability and Adoption of AI Suggestions
   • Visualization and Explainability of AI Output
4. Automated versus Human-AI Collaborative Approach

---

7. Limitation

1. Small Participant Sample
2. Time-saving Self-reporting
3. Teacher-Facing Evaluation Study

---

8. Conclusion

• English exam
• source code?