Student Research Executive Summary

In this pass, all citations were moved from de-identified papers stored as .pdf files to a single spreadsheet. Each citation was given a unique identifier in combination with its parent research paper and source number within that paper. In this process, 4 papers had no bibliography or 0 sources and were removed from subsequent passes. The year of each citation was also collected in this pass.

In this pass researchers categorized citations by source-type according to a methodology loosely adapted from previous research (Swales, Howard, Perruso, Hendley, Wilson) to answer the question “What types of sources are students citing?”. Because the length, extent, scope, etcetera of each citation analysis is different, while broad themes in our categorization are similar to other researchers’ citation analysis examples, ours required distinct categories with distinct definitions. These categorical definitions are outlined below.

Researchers were instructed on these categorical definitions during 2 separate training and Q&A sessions. To ensure categorical clarity, 11 researchers were split into two teams that each completed a distinct pass on the first ~1700 items in the data set (3400 individual categorizations). With these citations using Cohen’s kappa, which determines inter-rater reliability and factors for the possibility of agreement due to chance, our inter-rater reliability was determined to be 82%. According to Cohen’s kappa, this score fits into the “almost perfect” category at 82%; however, it should be noted that 82% is the cutoff for “almost perfect,” with a score of 81% being considered “strong.” Additional limitations of Cohen’s kappa are discussed in the linked article. After inter-rater reliability had been assessed, the two teams of researchers were dissolved to work on the remaining citations as one unit. 

In this pass researchers assessed the accuracy of sources according to MLA standards. This would answer the question, “How accurately are students citing sources?”. In addition to the broad categories, researchers also noted when a citation had url problems or contained “downstream info.” This pass was broken into broad categories defined below: 

In this pass researchers extracted the publisher container from each citation peer-reviewed, popular, self-published, and non-academic citation. This would answer the question, “What sources are students citing the most?”.