Abstract:

Modern text analysis tools enable researchers to investigate large text masses quicker and more easily than before. Keyword in context -searches have become easier and researchers can construct their own dictionaries (i.e., the sets of content categories) describing the themes, which have been found to be characteristic according to the preceding qualitative analysis of the text or according to theory-based reasoning.

In the study, we applied computer-assisted quantitative content analysis to assess the cognitive demand level of digital mathematics textbook texts in Finnish. The research data consisted of 9 digital textbooks for grades 7 to 9 dealing with different mathematics topics. The study sought answers to the following two problems:
1) Do the cognitive demand levels required by textbooks differ in different content areas?
2) Does the cognitive demand level of learning materials increase as the class level gets higher?

The comparison of the textbooks was based on a linguistic analysis of the texts. Cognitive demand levels were assessed by two different methods, both based on a quantitative analysis of the large collection of keywords calculated from the texts. The first keyword dictionary was based on the classification of action verbs at the revised Bloom levels (Anderson & Krathwohl 2001) and the second on the classification of typical mathematical prompts in tasks - such as calculate, solve, examine, define - into four different cognitive demand levels. Both so-called Yoshikode-dictionaries were developed for this study. In the study we also discuss what criteria can be used to assess the validity of such dictionaries and how reliable the measurement of cognitive demand is by this type of method.

The text mass contained a total of about 2000 pages of Finnish text. The language posed its own challenge to the choice of the applied software, as In Finnish language each word usually has numerous inflected forms and many words several synonyms as well. The analysis of the data was performed by the software Yoshikoder (Lowe 2015), which allows the keyword to be truncated to the stem of the word by replacing the end of the word with the wildcard entry.

The results revealed that the distributions of the measured demand levels of the textbooks were quite consistent (Research problem 1). Based on the revised Bloom’s levels, at each grade, lower levels were emphasized. Surprisingly, no clear increase in cognitive demand levels were observed in higher grade-levels (Research problem 2). Analyzing the data with a mathematical keyword dictionary yielded very similar results. Based on the results, the main emphasis in the design of learning materials, at this school level, seems to be on learning different mathematical contents. The learning materials do not seem to explicitly aim to develop a level of mathematical thinking from calculation skills and routines to an emphasis on reasoning and problem solving.

References:
[1] Anderson, L. W., & Krathwohl, D. R. (Eds.) (2001). A Taxonomy for Learning, Teaching and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. Complete Edition, New York: Longman.
[2] Krathwohl, D. R. (2002). A Revision of Bloom's Taxonomy: An Overview. Theory Into Practice, 41(4), 212–218.
[3] Lowe W. (2015). Yoshikoder: Cross-platform multilingual content analysis. Java software version 0.6.5. Retrieved May 1, 2016, from http://www.yoshikoder.org.

Keywords:

Textbooks, mathematics, computer-aided, content analysis.