Design Intro
Research is only as good as its design.
Design Intro
Research is only as good as its design.
The MSL foundation is characterized by high psychometric rigor and clear conceptual and theoretical grounding. Since its inception, the MSL has strived to generate data and results that are accurate, able to be applied with confidence, and add to the collective knowledge-base on leadership development.
Conceptual model
Conceptual Model
Conceptual model
Conceptual Model
An adapted version of Astin’s (1993) “input-environment-outcome” (I-E-O) college impact model provides the conceptual framework for MSL. This involves collecting data about students’ knowledge and experiences prior to college as well as their experiences during college. These can be triangulated to then examine the influences of experiences during college on a variety of educational outcomes.
The model is adapted in two key ways:
- The environment is extended to include variables representing experiences outside the college context (e.g., mentoring from employers, involvement in off-campus organizations).
- The study collects data at a single point with pre-college data collected through retrospective questions. Students are asked to think back to before they started college to capture these data points. This approach is supported by prior research indicating it reduces response-shift-bias and yields accurate indications of student gains (Howard, 1980; Rohs, 2002; Rohs & Langone, 1997).
Theoretical Framework
Theoretical Framework
Theoretical Framework
Theoretical Framework
More than the Social Change Model
The original theoretical framework for the MSL was the Social Change Model of Leadership Development (SCM; HERI, 1996). It was selected given the SCM:
Is consistent with contemporary leadership theory across a wide-array of disciplines;
Was created specifically for use in working with college students; and
Is consistently named as one of the most well-known and applied student leadership models (Kezar, Carducci, & Contreras-McGavin, 2006; Owen, 2012)
The SCM measures socially responsible leadership capacity defined as “a purposeful, collaborative, values-based process that results in positive social change” (Komives, Wagner, & Associates, 2009, p. xii). The SCM’s central principles are assessed through core values that describe students’ levels of self-awareness and abilities to work with others (HERI, 1996).
Framework Evolution-Coparison
As THE MSL HAS EVOLVED, the theoretical framework has been adapted to capture the increasing complexity of the study. The current theoretical framework is still nested in the values of the SCM, but draws from a much wider set of theoretical bases
Framework Evolution-Coparison
As THE MSL HAS EVOLVED, the theoretical framework has been adapted to capture the increasing complexity of the study. The current theoretical framework is still nested in the values of the SCM, but draws from a much wider set of theoretical bases
This includes:
- Contemporary Leadership Theory– Research from MSL has demonstrated that post-industrial theories of leadership have much in common. Thus, the theoretical framework now draws on key dimensions from a wider range of contemporary theories.
- Social Psychology & Human Development– Research from MSL points to the complexity of developmental processes. Contributing to leadership development involves much more than simple skill-building. As such, the theoretical frame relies heavily on insights from psychological and human development theories to inform the study.
- Critical & Justice–Based Perspectives– The MSL design has always attempted to model the values of social justice on which the SCM is predicated. This has led to increasing attention to design elements that draw on critical and justice-based perspectives.
| Model | Constructs | Does MSL Measure? | Direct or Indirect | How? |
| Authentic Leadership | Self-Awareness | ✔ | Directly | ✔ Consciousness of Self ✔ Collective Racial Esteem ✔ Motivation ✔ Self-Directed Learning ✔ Resiliency |
| Internalized Moral Perspective | ✔ | Directly | ✔ Congruence ✔ Commitment ✔ Resiliency ✔ Cognitive Skills | |
| Balanced Processing | ✔ | Directly | ✔ Cognitive Skills ✔ Social Perspective Taking | |
| Relational Transparency | ✔ | Indirectly | ✔ Consciousness of Self ✔ Leadership Efficacy ✔ Resiliency ✔ Congruence | |
| Relational Leadership Model | Purpose | ✔ | Directly | ✔ Commitment ✔ Common Purpose |
| Process | ✔ | Directly | ✔ Common Purpose ✔ Collaboration ✔ Controversy with Civility | |
| Inclusive | ✔ | Indirectly | ✔ Collaboration ✔ Social Perspective Taking | |
| Ethical | ✔ | Indirectly | ✔ Consciousness of Self ✔ Congruence ✔ Citizenship | |
| Empowering | ✔ | Indirectly | ✔ Collaboration | |
| Emotionally Intelligent Leadership** | Consciousness of Context | ✔ | indirectly | ✔ Belonging / Climate ✔ Cognitive Skills ✔ Social Perspective Taking ✔ Directed Learning |
| Consciousness of Self | ✔ | Directly | ✔ Consciousness of Self ✔ Collective Racial Esteem ✔ Motivation ✔ Resiliency ✔ Congruence | |
| Consciousness of Others | ✔ | Directly | ✔ Common Purpose ✔ Collaboration ✔ Social Perspective Taking ✔ Controversy with Civility | |
| Servant Leadership | Conceptualizing | ✔ | Indirectly | ✔ Common Purpose ✔ Cognitive Skills |
| Emotional Healing | ✔ | Indirectly | ✔ Social Perspective Taking | |
| Putting Followers First | ✘ | ✘ | ✘ | |
| Helping Followers Grow & Succeed | ✘ | ✘ | ✘ | |
| Behaving Ethically | ✔ | Indirectly | ✔ Consciousness of Self ✔ Congruence ✔ Citizenship | |
| Empowering | ✔ | Indirectly | ✔ Collaboration | |
| Creating Value for the Community | ✔ | Directly | ✔ Citizenship | |
| Leadership Practices Inventory** | Challenge the Process | ✔ | Indirectly | ✔ Change ✔ Cognitive Skills ✔ Leadership Efficacy ✔ Controversy with Civility |
| Create a Shared Vision | ✔ | Directly | ✔ Common Purpose ✔ Collaboration | |
| Enable Other to Act | ✔ | Indirectly | ✔ Collaboration | |
| Model the Way | ✔ | Indirectly | ✔ Congruence ✔ Commitment ✔ Resiliency | |
| Encourage the Heart | ✔ | Indirectly | ✔ Collaboration | |
| Transformational Leadership | Idealized Influence / Charisma | ✔ | Indirectly | ✔ Congruence ✔ Commitment |
| Inspirational Motivation | ✔ | Indirectly | ✔ Common Purpose ✔ Collaboration | |
| Intellectual Stimulation | ✔ | Indirectly | ✔ Controversy with Civility ✔ Common Purpose ✔ Cognitive Skills | |
| Individualized Consideration | ✔ | Indirectly | ✔ Social Perspective Taking ✔ Collaboration |
** Leadership Practices Inventory and Emotionally Intelligent Leadership are copyrighted by John Wiley & Sons, Inc.
Instrument & Psychometrics
Instrument & Psychometrics
Instrument & Psychometrics
Instrument & Psychometrics
The MSL survey questionnaire is composed of both existing and newly created scales designed to measure elements of the conceptual and theoretical frameworks. The amount of data generated for use in evidence-based practice is significant and transferable across a variety of areas within higher education (e.g., leadership programs, residence life, multicultural affairs, academic units, institutional research, campus recreation, fraternity and sorority life).
The MSL survey is particularly useful because it:
- Captures one of the widest ranges of student demographic data modeling the justice approach embedded in the theoretical framework. This includes capturing data on racial group membership, ethnic sub-group membership, AND racial identity salience; inclusion of identifiers for gender and sexual identity; a wide range of religious affiliations; and multiple dimensions of ability status.
- Examines both collegiate experiences AND educational outcomes including a wide-range of identity and human development constructs (e.g., complex cognitive skills, resilience, social perspective-taking, spiritual development).
The MSL survey instrument generates over 400 items and scales. The information below provides samples of key variables from across three domains of the study:
Input Variables
Capture student demographics as well as knowledge and experiences prior to enrollment in college. Variables include:
- Age
- Gender and sexual identity
- Racial and ethnic group membership
- Military status
- Parental education and income
- Pretest measures for all educational outcomes
- Involvement experiences prior to higher education
Experiences During College
Capture a wide range of experiences that occur during college and are often aligned with high-impact educational practices. Variables include:
- Mentoring relationships (e.g., with faculty, peers, community members, employers, parents)
- Academic-based experiences (e.g., study abroad, first-year seminars, research with faculty, internships)
- Involvement experiences (e.g., breadth and depth of involvement in both on and off-campus organizations)
- Civic engagement involvement (e.g., type and levels of involvement in community service learning experiences)
- Leadership development experiences (e.g., levels of involvement in both curricular and co-curricular leadership programs)
- Interactions about and across difference and perceptions of campus climate
Outcomes
Capture the degree of achievement across a wide-array of educational and leadership-related outcomes. This includes measures of:
- Leadership capacity (i.e., the knowledge, skills, and abilities associated with socially responsible leadership) including self-awareness, values congruence, ability to collaborate effectively, community engagement, and navigating conflict with civility among others;
- Leadership efficacy (i.e., one’s internal belief in their ability to lead effectively);
- Leadership behaviors (i.e., the degree to which on actually enacts leadership capacity to advance social responsibility and change);
- Higher order cognitive abilities including complex cognitive skills and social perspective-taking;
- Developmental outcomes related to resilience, racial identity and spiritual development; and
- Sense of belonging on campus.
Note that MSL also makes use of “sub-studies” as part of the research design. This involves administering some variables to a randomly selected 50% of the participants at each campus. MSL uses sub-studies to explore new areas of interest without dramatically increasing the length of the survey.
The MSL survey includes more than 400 variables, scales, and composite measures. As such, it would be impossible to detail full information related to the validity and reliability of measures. Much of this information can be found in academic articles published using the various scales accessible via this website. Additionally, a brief summary of the psychometrics of the survey can be found here.
The MSL survey instrument is also constantly evolving to include important new variables associated with student inputs, experiences during college, and outcomes. The survey instrument is typically finalized in June of the year prior to administration. If you would like to see a sample of the MSL survey instrument please contact MSLConnection@gmail.com.
Sampling
Sampling
Sampling
Sampling
So, what does it look like for a campus to participate in the MSL?
The MSL is administered online by SoundRocket, LLC. (formerly Survey Sciences Group, LLC). SoundRocket is an independent research organization with specific expertise in multi-campus studies. All data are collected using empirically proven standards for web-based survey research.
MSL’s online format allows students to share their experiences at any time or place that is convenient to them.
They can respond to the MSL when they feel comfortable, secure, and unhurried. We’ve partnered with SoundRocket to also conduct “studies of the study.” This allows us to better target data collection based on empirical evidence of how best to captures students’ information with minimized burden and maximized rates of return.
The data-collection period extends from January though the end of April, and each institution selects a three-week window that best fits their unique academic calendar. Students are invited to participate via personalized e-mails. Each correspondence outlines the study, addresses issues of confidentiality and consent, and supplies a link to the survey instrument. Students may receive up to four total contacts.
As the survey is administered, SoundRocket provides comprehensive respondent support via e-mail or telephone to minimize the inconvenience to your students. The service also allows students to opt out of the survey by replying to any MSL correspondence and requesting to be removed from the dataset. SoundRocket will monitor the survey completion rates and work with you to encourage participation if the response is insufficient.
To increase your response rate, you may host a sweepstakes-style drawing for students who complete the survey. Your institution will decide on the number, type, and value of the prizes; and SoundRocket will conduct the random drawing. Additionally, the MSL offers a number of monetary prizes raffled at the national level to stimulate survey response.
Data Security
Data Security
Data Security
Data Security
SoundRocket fully commits to confidentiality and security of survey data. We approach security in two ways:
Protocols for maintaining confidentiality of survey participants and their data, and
Technical systems that prevent unwanted access of data from outsiders
Confidentiality
Protecting the confidentiality of study participants is the most important concern of the SoundRocket staff. All individuals employed at SoundRocket are bound by confidentiality as condition of their employment. They have signed a pledge of confidentiality and have been trained in procedures for maintaining confidentiality and privacy.
Data are stored on SoundRocket internal servers that are password protected. The survey data captured in the web-based interface do not have any personal identifiers (individual or company name, email address, other contact information) in the data files. Personal identifiers are kept in a separate sample file in a secure space on our local file server accessible only by staff who require this information as part of their work. We never rent, sell, or give your personal information to any third party for the purpose of directly marketing any products or services to you. Any information you provide in the survey is used strictly for research purposes.
Data Security
We establish both logistical and physical barriers to protect respondent data to ensure its secure transmission and storage. Data are received and transmitted via password-protected, 128-bit SSL technology. Survey data submitted are encrypted before transmission via the SoundRocket web site from participants’ PCs/laptops to a secured server at SoundRocket. Data are archived in secure servers, accessible through password-protected networks by appropriate personnel. We use both stand-alone networks and firewalls to safeguard data against outside networks attacks.
The security of the information on our system has never been compromised.
Many websites use “cookies” to store information about a user in order to expedite the completion of forms on future visits to that site. SoundRocket does not use cookies on our website. The only information we collect from you on our website or any web-based survey is the information you provide.
SoundRocket monitors industry alerts and trends to ensure that we abide by the latest laws and regulations in protecting respondent data. We rigorously uphold — and often surpass — industry standards for research ethics, privacy, and confidentiality.
If you have questions about data confidentiality and security, please contact us:
E-mail: privacy@soundrocket.com
Phone: (734) 527-2180 and request to speak with our Privacy, Confidentiality and Ethics Officer.