Development of the Distance Education Learning Environments Survey (DELES) for Higher Education
Instrument Development
The development of the Distance Education Learning Environments Survey (DELES) follows a three-stage strategy outlined in Fraser (1986). Stage one included identification of salient scales within Moo's (1974) dimensions of Relationship, Personal Development, and System Maintenance and Change outlined above. However, this study goes one step beyond the learning environment exclusively and includes an added affective measure of student satisfaction-the data from which can be analysed for strength in association with the learning environment. Stage two involved writing individual items within the scales, and stage three called for field testing items followed by an item analysis. This section discusses in detail the steps taken to develop the DELES.
Stage One - Identification and Development of Salient Scales
Stage one development took three primary components into consideration: (1) review of and consistency with previously developed instruments (Fraser, Treagust, & Dennis, 1986; Jegede, Fraser, & Fisher, 1998); (2) review and consistency with the literature related to distance education learning environments and student satisfaction; and (3) review of preliminary scales by a panel of experts (Jegede, Fraser, & Fisher, 1998).
Review and consistency with previously developed instruments. From the classical perspective of survey development, Hase and Goldberg's National Science Foundation-supported study on differing strategies for the development of personality inventory scales purports that there are three primary strategies from which to consider when developing scales: (1) Internal, (2) External, and (3) Intuitive (1967). Within the internal scale development strategy falls the factor-analytical strategy. In this case items are given to a significant population and are then factor analysed. Items with high factor loading are then used for the scales. Accordingly, this strategy selects the best properties occurring from the internal structure of the original item pool. Within the external scale development strategy falls the empirical group discriminative strategy. In developing scales from this perspective, the researcher administers the instrument to persons falling on opposite poles of the personality trait under measure. Items within these scales are examined from the point of view of discriminating between the two groups at some level of significance. Those items with significant discriminating power remain in the scale. Likewise, within the intuitive strategy falls the intuitive-rational and the intuitive-theoretical strategies for scale development. The categorisation of the scales developed for the DELES fall under the intuitive-rational development of scales category following Fraser (1986). This categorisation is termed as such due to the intuitive understanding of the subject matter by the researcher. Validity within this categorisation of scales is contingent upon subjective opinion of the researcher. While no specific psychological theory is followed under this category of scale development, the scales are filtered by selecting items of highest internal consistency and only those items remain in the revised scales. Opposite the intuitive-rational strategy is the intuitive-theoretical strategy that, while still relying on filtering by means of high internal consistency, is based upon a given psychological theory.
Hase and Goldberg found that when constructing scales using these strategies, there was little difference in the outcomes and that they were "equivalent in their validity across 13 diverse criteria" (1967, p. 242). They went on to state, "dogmatic assertions of the superiority of one strategy over another are premature" (pp. 242-243).
Following the selection of strategy for scale development is the identification of salient scales. As discussed by Fisher and Fraser (1990) regarding the School Level Environment Questionnaire (SLEQ), new inventories require scales that distinguish important aspects of the environment under study. However, while new scales should be developed, similar studies of learning environments not only develop new scales, the precedent is set for modification and adaptation of scales from previously validated and tested questionaries (Fish & Dane, 2000; Fisher, Rickards, & Fraser, 1996; Maor, 1999; Newby, & Fisher, 1997). This study was influenced by previous inventories including the College and University Classroom Environment Inventory (Fraser, Treagust, & Dennis, 1986), the Constructivist Learning Environment Survey (Taylor, Fraser & Fisher, 1997), the Distance and Open Learning Environment Scale (Jegede, Fraser, & Fisher, 1998), and the Test of Science-Related Attitudes (Fraser, 1981).
Review and consistency with the literature related to distance education learning environments and student satisfaction. Literature related to distance education is prolific. However, when filtered for indications of the relationship of distance education to psychosocial learning environments, the number of sources drops to a level that allows for the character of quality distance education to be summarized and categorised under Moos' dimensions. Likewise, consideration of student satisfaction, often used in higher education as a key indicator of quality (National Survey of Student Engagement, 2000) as related to post-secondary distance education can be measured for strength against that of the learning environment. Below is a discussion of how the scales are consistent with literature related to distance education in terms of Moo's three psychosocial dimensions of Relationship, Personal Development, and System Maintenance and Change.
Relationship Dimension. Over one third of the preliminary scales developed for the DELES, both new scales and those scales adapted from previous instruments, representing characteristics of distance education as related to the learning environment, tend to fall under the Relationship Dimension, whereby individuals are active in their learning environment and engage with one another. Peer collaboration, review, interaction, exchange, contribution, and community development are keywords continually identified as indicators of a quality distance education environment that promotes thinking and learning (Butler, 2001; Chickering & Ehrmann, 1996; Frederick, Cannon, Umble, Steckler, & Shay, 2001; Golbeck & Sinagra, 2000; Hartley, 2000; Ho & Tabata, 2001; Morihara, 2001; Nader, 2001; Owen, 2000; Park-Kim, 2001; Twigg, 2004; Zhu & McKnight, 2001). Likewise, the opportunity for interaction between the student and the instructor was continually noted as a leading indicator of a quality distance education environment (Chickering & Ehrmann, 1996; Ho & Tabata, 2001; Morihara, 2001; Park-Kim, 2001; Zhu & McKnight, 2001). Sinagra and Golbeck (2000) summarized the strength for the need of relationship-oriented interaction in distance education by stating, "from a Piagetian, constructivist perspective, the symmetrical nature of peer relationships presents an ideal context for promoting the development of thinking" (p. 22).
Personal Development. The opportunities offered by the distance-education learning environment for self-enhancement and development of self-worth, personal development, independence, intellectual pursuit, and academic achievement make up the dimension of Personal Development. Fewer references to characteristics of this dimension were found in the literature directly related to distance education. However, key indicators include such aspects of learning as authentic learning, problem solving, active learning, student reflection, and scaffolded activities (Chickering & Ehrmann, 1996; Hartley, 2000; Ho & Tabata, 2001; Markel, 1999; Merrill, 2001; Morihara, 2001; Owen, 2000, Twigg, 2004).
System Maintenance and Change. In terms of System Maintenance & Change, this is the extent to which the distance education environment is orderly, organized, clear in expectation, student control exercisable and the environment is student focused, responsive to change, and orderly. In this dimension, motivational goals are mastery goals where locus of control is student-oriented and effort and outcome are seen as interdependent (Cox, Preston, & Cox, 1999; Chang, & Fisher, 2001; Jegede, Taplin, Fan, Chan & Yum, 1999). Students are provided assignment overviews and advanced organizers (Park-Kim, 2001), there is room for student decision-making, activity initiation, and expectations are clearly set forth by the instructor (Owen, 2000).
Student Satisfaction. Student satisfaction is not a measure of the psychosocial learning environment, yet is used consistently in post-secondary education to measure how effectively a program or institution delivers what students, expect, need, and want, and is associated with student achievement (Kuh, 2001a; 2001b). With the addition of an attitudinal scale to a learning environment instrument, the relationship between learners' attitudes and their perceptions of the psychosocial distinctiveness of the learning environment can be measured (Fraser, 1981). When classified in terms of Kirkpatrick's four level framework of evaluation within a distance environment (Walker, 1998) the researcher gains a subjective measure of the learners' reactions to the materials, instructor, methodology, and environment with measures of satisfaction. Human resources training development programs regularly use any number of approaches to reaction evaluation such as with the following instruments: Reaction Outcomes (Bell System Approach), Reaction Evaluation (The CIRO Approach), Training Satisfaction (Saratoga Institute Approach), and Reaction (The IBM Approach) (Phillips, 1991).
Review of preliminary scales by a panel of experts. Given the literature above, new scales were created to address each of Moos' three psychosocial dimensions. Likewise, previously developed scales from existing instruments were modified, adapted, and considered in a compilation making up a set of 14 preliminary scales.
One of the aims of learning environment studies by means of survey instrument is that the instrument be designed to be economic in size (Fraser, 1986). After the preliminary 14 salient scales were developed through reviews of literature and previous instruments, they then required reduction in order to generate an economic survey instrument. To achieve this goal of reduction of scales and to reduce subjectivity in that reduction, seven new scales developed from literature review and seven scales adapted from previous learning environment surveys, along with their descriptions, were presented to an international panel of experts following precedent in previous scale development (Fish & Dane, 2000; Jegede, Fraser, & Fisher, 1998). A 14-person panel of distance education researchers and practitioners was assembled after being identified through literature review and a call for participation via 10 international distance-education specific e-mail discussion groups. The panel was asked to rank each preliminary scale related to its appropriateness, relevance, and suitability toward post-secondary distance education. The preliminary scales were then reduced based on the panel's rankings and comments. This reduction resulted in six scales assumed to be principal to the study of post-secondary distance education as listed in Table 1.
Table 1. Six scales remaining after face validation by an expert panel.
| Dimension | Scale |
| Relationship - Individuals are involved in the environment and support/help each other; personal relationships between inhabitants; involvement, affiliation, support, assistance, peer cohesion. | Instructor Support |
| Student Interaction & Collaboration | |
| Personal Development - Opportunity afforded by the environment for self-enhancement and development of self-esteem; personal growth and development; autonomy, independence, intellectuality, academic achievement, competition | Personal Relevance |
| Authentic Learning | |
| Active Learning | |
| System Maintenance & Change - the environment is orderly, clear in expectation, control, responses to change; order, organization, clarity, control. | Student Autonomy |
In addition to the six scales designed to measure Moos' three psychosocial dimensions, a scale of satisfaction, adapted from the Test of Science-Related Attitudes (Fraser, 1981) was included for later analysis related to discovering associations between the psychosocial learning environment scales in Table 1 and an affective measure of student satisfaction (see Table 2).
Table 2. Measure of satisfaction.
| Measure | Scale | Scale Description |
| Satisfaction - Program or institution's effectiveness in delivering what students, expect, need, and want. | Enjoyment | Extend to which students enjoy learning in a distance education environment |
Stage Two - Writing Individual Items
Once the salient scales were developed and face validated, the next step in the development of this survey was writing a set of items to measure each scale. In writing items, there must be consideration given to maximizing the relationship between the item and the scale under measure and in avoiding cross-scale measurement.
The draft items used in this survey were distributed to an international panel of distance education practitioners for their comments on the individual item's suitability, face validity, readability, and freedom from ambiguity (Fraser, 1986; Jegede, Fraser, & Fisher, 1998). Items related to scales adapted from previous studies were subsequently adapted and reviewed for this study. Further face validation was conducted by submitting the items to graduate-level distance education students (N=12) for review of readability and ambiguity. All survey items were rendered in their 'actual' form of the three forms (preferred, actual, instructor) and the survey was distributed for the present research.
Table 3 presents the dimensions/measure, scales, and items as they relate to one another. The items were then distributed in the final survey instrument.
Table 3. Final scales and their preliminary items.
| Scale | Items |
| Instructor Support | In this class...
|
| Student Interaction & Collaboration | In this class...
|
| Personal Relevance | In this class...
|
| Authentic Learning | In this class...
|
| Active Learning | In this class...
|
| Student Autonomy | In this class...
|
Response choices are: Always, Often, Sometimes, Seldom, and Never. (c) 2003, 2004
Stage Three - Field Testing and Item Analysis
The final stage of the development of the DELES was a field test in the form of a Web-based survey. The Web-based instrument used the response choices of Always, Often, Sometimes, Seldom, and Never. Data resulting from the field test was then analysed for internal consistency reliability. Likewise, the data for each item was analysed for discriminant validity to identify if each item measures a unique dimension not measured by other items and to be certain items are primarily measuring within their own scales. Analysis also included the identification of associations between six DELES psychosocial learning environment scales and the scale of student satisfaction by means of simple correlations (r) and standardized regression coefficients (ß). The results of these analyses are presented in detail at the South Central RTEC's Insight Instrument Library and Data Repository found at the following URL: http://insight.southcentralrtec.org/ilib/delesa/delesainfo.html.