Publications

2024

1. Preprint

   

   Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach

   Qingchuan Li, Jiatong Li, Tongxuan Liu, and 4 more authors

   2024

   HTML
2. NeurIPS ’24

   

   PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbations (Spotlight)

   Jiatong Li, Renjun Hu, Kunzhe Huang, and 5 more authors

   In Advances in Neural Information Processing Systems, 2024

   HTML Code
3. NeurIPS ’24

   

   Computerized Adaptive Testing via Collaborative Ranking

   Zirui Liu, Yan Zhuang, Qi Liu, and 5 more authors

   In Advances in Neural Information Processing Systems, 2024
4. WWW ’24

   

   Towards the Identifiability and Explainability for Personalized Learner Modeling: An Inductive Paradigm

   Jiatong Li, Qi Liu, Fei Wang, and 5 more authors

   In Proceedings of the ACM Web Conference 2024, Singapore, Singapore, 2024

   Abs HTML Code

   Personalized learner modeling using cognitive diagnosis (CD), which aims to model learners’ cognitive states by diagnosing learner traits from behavioral data, is a fundamental yet significant task in many web learning services. Existing cognitive diagnosis models (CDMs) follow theproficiency-response paradigm that views learner traits and question parameters as trainable embeddings and learns them through learner performance prediction. However, we notice that this paradigm leads to the inevitable non-identifiability and explainability overfitting problem, which is harmful to the quantification of learners’ cognitive states and the quality of web learning services. To address these problems, we propose an identifiable cognitive diagnosis framework (ID-CDF) based on a novelresponse-proficiency-response paradigm inspired by encoder-decoder models. Specifically, we first devise the diagnostic module of ID-CDF, which leverages inductive learning to eliminate randomness in optimization to guarantee identifiability and captures the monotonicity between overall response data distribution and cognitive states to prevent explainability overfitting. Next, we propose a flexible predictive module for ID-CDF to ensure diagnosis preciseness. We further present an implementation of ID-CDF, i.e., ID-CDM, to illustrate its usability. Extensive experiments on four real-world datasets with different characteristics demonstrate that ID-CDF can effectively address the problems without loss of diagnosis preciseness. Our code is available at https://github.com/CSLiJT/ID-CDF.

5. KDD ’24

   AdaRD: An Adaptive Response Denoising Framework for Robust Learner Modeling

   Fangzhou Yao, Qi Liu, Linan Yue, and 4 more authors

   In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Barcelona, Spain, 2024

   Abs HTML

   Learner modeling is a crucial task in online learning environments, where Cognitive Diagnosis Models (CDMs) are employed to assess learners’ knowledge mastery levels based on recorded response logs. However, the prevalence of noise in recorded response data poses significant challenges, including various behaviors such as guess and slip, casual answers, and system-induced errors. The existence of noise degrades the accuracy of diagnosis results and learner performance predictions. In this work, we propose a general framework, Adaptive Response Denoising (AdaRD), designed to salvage CDMs from the influence of noisy learner-exercise responses. AdaRD extends existing CDMs, incorporating primary training for denoised CDMs and auxiliary training for additional denoising support. The primary training employs binary Generalized Cross Entropy (GCE) loss to slow down the large update of learner knowledge states caused by noisy responses. Simultaneously, we utilize the variance of diagnosed knowledge mastery levels between primary and auxiliary diagnosis modules as a criterion to downweight high-variance responses that are likely to be noisy. In this manner, the proposed framework can prune noisy response learning during training, thereby enhancing the accuracy and robustness of CDMs. Extensive experiments on both real-world and synthetic datasets validate AdaRD’s effectiveness in mitigating the impact of noisy learner-exercise responses.

6. Preprint

   Survey of Computerized Adaptive Testing: A Machine Learning Perspective

   Qi Liu, Yan Zhuang, Haoyang Bi, and 12 more authors

   2024

   HTML
7. Preprint

   A Survey of Models for Cognitive Diagnosis: New Developments and Future Directions

   Fei Wang, Weibo Gao, Qi Liu, and 8 more authors

   2024

   HTML
8. DASFAA ’24

   Reformulating Sequential Recommendation: Learning Dynamic User Interest with Content-enriched Language Modeling

   Junzhe Jiang, Shang Qu, Mingyue Cheng, and 8 more authors

   2024

   HTML

2023

1. Preprint

   

   Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluation

   Jiatong Li, Rui Li, and Qi Liu

   2023

   HTML

2022

1. KDD ’22

   

   HierCDF: A Bayesian Network-based Hierarchical Cognitive Diagnosis Framework

   Jiatong Li, Fei Wang, Qi Liu, and 6 more authors

   In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington DC, USA, 2022

   Abs HTML Code

   Cognitive diagnostic assessment is a fundamental task in intelligent education, which aims at quantifying students’ cognitive level on knowledge attributes. Since there exists learning dependency among knowledge attributes, it is crucial for cognitive diagnosis models (CDMs) to incorporate attribute hierarchy when assessing students. The attribute hierarchy is only explored by a few CDMs such as Attribute Hierarchy Method, and there are still two significant limitations in these methods. First, the time complexity would be unbearable when the number of attributes is large. Second, the assumption used to model the attribute hierarchy is too strong so that it may lose some information of the hierarchy and is not flexible enough to fit all situations. To address these limitations, we propose a novel Bayesian network-based Hierarchical Cognitive Diagnosis Framework (HierCDF), which enables many traditional diagnostic models to flexibly integrate the attribute hierarchy for better diagnosis. Specifically, we first use an efficient Bayesian network to model the influence of attribute hierarchy on students’ cognitive states. Then we design a CDM adaptor to bridge the gap between students’ cognitive states and the input features of existing diagnostic models. Finally, we analyze the generality and complexity of HierCDF to show its effectiveness in modeling hierarchy information. The performance of HierCDF is experimentally proved on real-world large-scale datasets.