publications

2023

1. Data Selection for Fine-tuning Large Language Models Using Transferred Shapley Values

   Schoch, Stephanie, Mishra, Ritwick, and Ji, Yangfeng

   In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop) Jul 2023

   Abs

   Although Shapley values have been shown to be highly effective for identifying harmful training instances, dataset size and model complexity constraints limit the ability to apply Shapley-based data valuation to fine-tuning large pre-trained language models. To address this, we propose TS-DShapley, an algorithm that reduces computational cost of Shapley-based data valuation through: 1) an efficient sampling-based method that aggregates Shapley values computed from subsets for valuation of the entire training set, and 2) a value transfer method that leverages value information extracted from a simple classifier trained using representations from the target language model. Our experiments applying TS-DShapley to select data for fine-tuning BERT-based language models on benchmark natural language understanding (NLU) datasets show that TS-DShapley outperforms existing data selection methods. Further, TS-DShapley can filter fine-tuning data to increase language model performance compared to training with the full fine-tuning dataset.

2022

1. CS-Shapley: Class-wise Shapley Values for Data Valuation in Classification

   Schoch, Stephanie, Xu, Haifeng, and Ji, Yangfeng

   In NeurIPS 2022 Nov 2022
2. Ask and You Shall Receive?: A LibAnswers FAQ Usability Study

   Schoch, Stephanie, and VerMeulen, Amanda

   In Weave: Journal Of Library User Experience Oct 2022

2021

1. Underreporting of errors in NLG output, and what to do about it

   Miltenburg, Emiel, Clinciu, Miruna, Dušek, Ondřej, Gkatzia, Dimitra, Inglis, Stephanie, Leppänen, Leo, Mahamood, Saad, Manning, Emma,  Schoch, Stephanie, Thomson, Craig, and Wen, Luou

   In Proceedings of the 14th International Conference on Natural Language Generation Aug 2021
2. Contextualizing Variation in Text Style Transfer Datasets

   Schoch, Stephanie, Du, Wanyu, and Ji, Yangfeng

   In Proceedings of the 14th International Conference on Natural Language Generation Aug 2021

2020

1. “This is a Problem, Don’t You Agree?” Framing and Bias in Human Evaluation for Natural Language Generation

   Schoch, Stephanie, Yang, Diyi, and Ji, Yangfeng

   In Proceedings of the 1st Workshop on Evaluating NLG Evaluation Dec 2020